为什么值得关注

无需框架、无需构建步骤、无外部请求；仅需 Python 和 nvidia-smi 即可开箱即用，还提供可选的 Rust 构建版本，用于精确的客户端测量吞吐量和 SSE 推送。

适合谁使用

运行本地 LLM 推理服务器的开发者
监控 GPU 使用率的系统管理员
尝试多个 LLM 后端的研究人员
搭建本地 AI 设备的爱好者

典型使用场景

实时监控 llama.cpp/vLLM 服务器的 GPU 和吞吐量
检测瓶颈并优化模型部署
通过可选的推理 tap 跟踪每个请求的推理/思维链日志
浏览模型库并比较性能指标

项目优势

单文件后端，零依赖（仅使用 Python 标准库）
自动发现工作端口（从监听套接字）
可选的推理 tap，用于实时的 CoT 面板
两个构建版本：Python 快速设置，Rust 精确吞吐量和 SSE 推送

使用前须知

需要 NVIDIA GPU 以运行 nvidia-smi
无身份验证（设计为本地使用，但绑定到 0.0.0.0）
Python 版本使用 predicted_tokens_seconds 指标，模型空闲时可能读数为 0

README 快速开始

LLM Serve Dashboard

A single-file, dependency-free live dashboard for your local LLM serving box — GPU utilization, per-model throughput, KV/context fill, and system stats for llama.cpp and vLLM, in one green terminal-styled page.

No framework, no build step, no external requests. The frontend is one index.html (opens on file://); the backend is one stdlib Python file that reads nvidia-smi and each server's Prometheus /metrics.

(Screenshot rendered with example data — plug in your own box and it goes live.)

What it shows

GPUs — per-card utilization, VRAM, power, temperature, clocks, and the actual compute tenants on each card (pulled from nvidia-smi --query-compute-apps, so cards are labeled from ground truth, not VRAM guesswork).
Primary worker — decode & prefill tokens/sec, request counts, context/KV fill, LoRA adapters. Works with llama.cpp (/metrics + /props) and vLLM (/metrics + /v1/models). The worker port is auto-discovered from listening sockets, so a bench or swap that moves the model to another port still lands on the dashboard.
Secondary servers — an optional row of cards for extra CPU/GPU llama-servers you run (point them at any endpoints — a small CPU model, a second box, etc.). Configure with SECONDARY_SERVERS.
System — CPU, RAM, load, network, disk.
Model library — a browsable inventory of your loadable models with quant, ctx, and measured throughput, driven by a JSON registry you edit.
Reasoning tap (optional) — live per-request reasoning/CoT panels, if you tee a model's reasoning_content to a log (see THOUGHT_LOG below). Off by default.

Run it

Requirements: Python 3.8+ and nvidia-smi (NVIDIA GPUs). No pip installs.

# 1. start the metrics endpoint (serves JSON on :8092)
python3 fleet-metrics.py

# 2. open the dashboard
xdg-open index.html      # or just double-click it / open in a browser

The page polls http://localhost:8092 for live data. That's it.

Point it at your servers

By default it looks for a llama.cpp/vLLM server on :8001 (then :8010, :8123). Override:

# candidate ports for the primary worker (first responder with the largest ctx wins)
WORKER_PORT_CANDIDATES=8000,8001,8080 python3 fleet-metrics.py

# secondary servers shown as extra cards: name:port,name:port
SECONDARY_SERVERS='cpu-a:9093,b

项目描述

Single-file, dependency-free live dashboard for local llama.cpp/vLLM serving — GPU, throughput, KV/ctx, model library. Stdlib Python + one local-first HTML page.

llm-serve-dashboard