该仓库提供了异常详细且可复现的基准测试存档，保留了每次运行的模型检查点、量化方式、KV缓存类型、张量并行布局、引擎构建及启动命令，使得仅凭单个令牌/秒数字无法实现的精确交叉比较与分析成为可能。
638 local inference measurements from one 96 GB home rig. The exact launch command is attached to 637 of them.
Most benchmark posts give you a model name and one tok/s number. I kept
losing the rest: which checkpoint, which quant, which KV dtype, which TP layout,
which engine build, and whether the number meant single-stream decode or
aggregate throughput.
This repository is the database I wanted. You can inspect one result in the Markdown catalog, query all of them in SQLite, or take the JSONL and run your own analysis.
| Open | What is there |
|---|---|
| Models | All 14 model families and their runs |
| Benchmark campaigns | Matched experiments and test matrices |
| Engines | vLLM, llama.cpp, and the experimental exllamav3 slice |
| Query guide | Ready-to-run SQLite, JSONL, and Python examples |
| Methodology | Metric definitions, comparison rules, and limitations |
| Hardware | The 4x RTX 3090 rig, topology, power limits, and software versions |
| Dataset files | JSONL, SQLite, snapshot metadata, and schema notes |
The archive currently has 638 measurements from 20 benchmark campaigns across 14 model families. Most runs use vLLM. The llama.cpp set is smaller, and the 38 exllamav3 rows came from a short experiment rather than broad engine coverage.
This is a benchmark archive, not a leaderboard. The harness and the variable I was testing changed between campaigns, so some rows should not be compared directly. Each row keeps its workload, topology, power limit, engine version, source references, and launch command when available. 637 of 638 measurements retain an exact argv or command.
Every result keeps the context needed to understand or reproduce it:
| Recorded with the number | Examples |
|---|---|
| Model artifact | checkpoint, quantization recipe, source revision |
| Runtime | engine, version, attention backend, KV-cache dtype |
| GPU layout | TP, PP, DP, instance count, power limit, P2P state |
| Workload | prompt length, output length, concurrency, context limit |
| Metric meaning | single-stream wall rate, decode rate, aggregate output, TTFT |
| Receipt |
Local LLM speed measurements from a 4x RTX 3090 rig, with exact launch commands and per-run provenance.
根据分类、Topic 和编程语言匹配的相似项目。
Deltafin is a research project that runs the 2.8-trillion-parameter Mixture-of-Experts model Kimi K3 on a single Apple Silicon Mac (e.g., M1 Max with 64 GB) at about 16 seconds per token, using exact, reproducible inference with local or streaming expert loading.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.
