nnkiirue99-maker GitHub avatar

llm-memory-postmortem

nnkiirue99-maker

A real-world case study of moving a 530M ML stack out of a Flask main process, accidentally discovering a 32x retrieval speedup through profiling and caching.

Stars

8

7-day growth

No data

Forks

1

Open issues

0

License

MIT

Last updated

2026-07-27

AI repository intelligence
FR-AI / ANALYSIS

Why it is worth attention

Demonstrates a systematic debugging approach (measuring instead of guessing), pragmatic risk-aware architecture decisions, and provides fully reproducible code and benchmarks.

Who it is for

  • Python web developers
  • DevOps engineers optimizing memory usage
  • ML engineers deploying models in low-resource environments
  • Developers interested in performance profiling and caching strategies

Use cases

  • Optimizing memory-constrained Flask applications with ML components
  • Separating heavy model inference into standalone microservices
  • Profiling and fixing unexpected performance bottlenecks (BM25 index rebuild)
  • Implementing safe fallback and retry strategies for service degradation

Strengths

  • Reduced Flask RSS from 888M to 68M (stable)
  • Improved semantic search latency from 10.99s to 0.34s (32x faster)
  • Full consistency verification (bit‑identical results)
  • Clear reasoning about tradeoffs (risk vs. benefit, rollback safety)

Considerations

  • Requires systemd for idle process unloading (5s restart window)
  • Cold start takes ~20s (model load 8s + BM25 index rebuild 11.5s)
  • BM25 cache uses ~60M memory, not yet optimized for disk persistence
  • Solution is specific to this application's architecture (Flask + BM25 + sentence‑transformers)

README quick start

把 530M 的 ML 栈请出 Flask 主进程,顺便让检索快了 32 倍

一次真实的排查记录。起因是一个跑在 3.6G 内存小机器上的 Flask 应用,进程内存三天涨到 888M。

结论先放这里,省得你翻到最后:

之前之后
Flask 主进程 RSS888M(三天)68M(稳定)
一次语义检索10.99s0.34s
检索结果逐位一致,没变

两件事其实没什么关系,是排查过程中前后脚撞上的。但它们合起来说明一个道理:动手优化之前先测量,你以为的瓶颈通常不是瓶颈。我在这篇里两次猜错,都写出来了。


一、现象

一个 Flask 应用,功能是笔记 + 对话,带语义检索(bge 中文小模型做 embedding,BM25 做关键词兜底)。机器 3.6G 内存。

现象是主进程 RSS 一路涨:

重启后          79M
17 分钟后      662M
三天后         888M

看起来像内存泄漏。但"看起来像"不等于是。

二、第一步:先证伪,别急着抓泄漏

如果真是泄漏,那一定有某个代码路径反复执行、每次留下一点。最省事的验证方式是按真实流量比例打请求,看 RSS 和对象数怎么走

我没用 py-spy、没上 tracemalloc,就用 Flask 自带的 test_client 在独立进程里跑:

import gc, collections, time

def rss_kb():
    for line in open('/proc/self/status'):
        if line.startswith('VmRSS:'):
            return int(line.split()[1])

def obj_profile():
    c = collections.Counter()
    for o in gc.get_objects():
        c[type(o).__name__] += 1
    return c

import app as A
client = A.app.test_client()

gc.collect()
base_rss, base_obj = rss_kb(), obj_profile()

# 1) 先测后台线程空转:什么都不请求,只等
time.sleep(45)
gc.collect()
print('空转 45 秒:', (rss_kb() - base_rss) / 1024, 'M')

# 2) 再按真实流量比例逐个端点压
for method, url in ENDPOINTS:
    gc.collect(); before = rss_kb()
    for _ in range(150):
        client.get(url)
    gc.collect(); after = rss_kb()
    print(f'{url:42} {(after-before)/1024:+.1f}M')

# 3) 最后看对象类型的净增长
gc.collect()
for name, n in (obj_profile() - base_obj).most_common(12):
    print(f'{name:28} +{n}')

完整脚本在 code/memprobe.py

结果:

空转 45 秒后                    +0.3M
/api/notes_by_date              +0.3M
/api/profile                    -0.3M
/api/feed                       +0.8M
/api/tags/list                  +0.0M
/api/messages                   +2.0M
/api/shelf                      +0.0M
...
1200 个请求总计                 +3M

对象数增长 top:
  function                     +23
  ReferenceType                +19
  builtin_function_or_method   +19
  SplitResult                  +18
  list                          +4

读接口是干净的。 1200 个请求只涨 3M,对象增长几十个而且全是模块导入的正常副产物(functionModuleSpecSourceFileLoader 这些是首次走到某分支时 import 带进来的,一次性)。后台线程空转也干净。

那 888M 从哪来的?

三、分层测:不是泄漏,是一次性加载

既然不是反复累积,那就是某个东西一次性吃掉一大块然后赖着不走。分层测最直接:

def rss_mb():
    for line in open('/proc/self/status'):
        if line.startswith('VmRSS:'):
            retu

Description

把 530M 的 ML 栈请出 Flask 主进程,顺便让语义检索快了 32 倍。一次真实排查的完整记录:怎么证伪内存泄漏、为什么拆进程而不换 ONNX、以及 profile 如何打脸我的直觉。

Related repositories

Similar projects matched by category, topics, and programming language.

slvDev
Featured
slvDev GitHub avatar

esp32-ai

A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.

AI & Machine LearningLarge Language Models
1,960
gavamedia
Featured
gavamedia GitHub avatar

deltafin

Deltafin is a research project that runs the 2.8-trillion-parameter Mixture-of-Experts model Kimi K3 on a single Apple Silicon Mac (e.g., M1 Max with 64 GB) at about 16 seconds per token, using exact, reproducible inference with local or streaming expert loading.

AI & Machine LearningLarge Language Models
304
jamesob
Featured
jamesob GitHub avatar

local-llm

A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.

AI & Machine LearningLarge Language Models
1,660