LLM Serve Dashboard

A single-file, dependency-free live dashboard for your local LLM serving box — GPU utilization, per-model throughput, KV/context fill, and system stats for llama.cpp and vLLM, in one green terminal-styled page.

No framework, no build step, no external requests. The frontend is one index.html (opens on file://); the backend is one stdlib Python file that reads nvidia-smi and each server's Prometheus /metrics.

(Screenshot rendered with example data — plug in your own box and it goes live.)

What it shows

• GPUs — per-card utilization, VRAM, power, temperature, clocks, and the actual compute tenants on each card (pulled from nvidia-smi --query-compute-apps, so cards are labeled from ground truth, not VRAM guesswork).
• Primary worker — decode & prefill tokens/sec, request counts, context/KV fill, LoRA adapters. Works with llama.cpp (/metrics + /props) and vLLM (/metrics + /v1/models). The worker port is auto-discovered from listening sockets, so a bench or swap that moves the model to another port still lands on the dashboard.
• Secondary servers — an optional row of cards for extra CPU/GPU llama-servers you run (point them at any endpoints — a small CPU model, a second box, etc.). Configure with SECONDARY_SERVERS.
• System — CPU, RAM, load, network, disk.
• Model library — a browsable inventory of your loadable models with quant, ctx, and measured throughput, driven by a JSON registry you edit.
• Reasoning tap (optional) — live per-request reasoning/CoT panels, if you tee a model's reasoning_content to a log (see THOUGHT_LOG below). Off by default.

Run it

Requirements: Python 3.8+ and nvidia-smi (NVIDIA GPUs). No pip installs.

# 1. start the metrics endpoint (serves JSON on :8092)
python3 fleet-metrics.py

# 2. open the dashboard
xdg-open index.html      # or just double-click it / open in a browser

The page polls http://localhost:8092 for live data. That's it.

Point it at your servers

By default it looks for a llama.cpp/vLLM server on :8001 (then :8010, :8123). Override:

# candidate ports for the primary worker (first responder with the largest ctx wins)
WORKER_PORT_CANDIDATES=8000,8001,8080 python3 fleet-metrics.py

# secondary servers shown as extra cards: name:port,name:port
SECONDARY_SERVERS='cpu-a:9093,b