AI repository intelligence

FR-AI / ANALYSIS

Why it is worth attention

It is a comprehensive open-source toolkit from DeepSeek that includes multiple state-of-the-art speculative decoding algorithms, pre-trained checkpoints for several target models, and a complete workflow from data generation to benchmark evaluation, enabling reproducible research and practical deployment.

Who it is for

ML researchers studying speculative decoding and LLM inference acceleration
Engineers deploying large language models with latency requirements
Teams developing draft models for production LLM serving systems

Use cases

Train a draft model to accelerate inference of a target LLM via speculative decoding
Evaluate draft model performance on standard benchmarks like GSM8K, MATH500, and HumanEval
Reproduce results from the associated DSpark paper or compare with DFlash/Eagle3 baselines

Strengths

Covers the entire pipeline from data preparation to evaluation, with clear scripts and configuration files
Supports three cutting-edge draft model algorithms (DSpark, DFlash, Eagle3) with released Hugging Face checkpoints
Comes with a set of evaluation benchmarks (9 datasets) and pre-configured configs for four target models (Qwen3-4B/8B/14B, Gemma-4-12B)
Adapted from well-tested open-source projects (SpecForge, DFlash) and includes a detailed NOTICE file for licensing compliance

Considerations

Data preparation requires an inference engine and a very large target cache (≈38 TB for the default setting), making it costly to run
Training scripts assume a single node with 8 GPUs; using fewer GPUs requires manual adjustment of CUDA_VISIBLE_DEVICES
The repository does not include instructions for fine-tuning in thinking mode; users are advised to retrain for domain-specific or thinking-mode use cases

README quick start

DeepSpec

DeepSpec is a full-stack codebase for training and evaluating draft models for speculative decoding. It contains data preparation utilities, draft model implementations, training code, and evaluation scripts.

Environment

Install the Python dependencies:

python -m pip install -r requirements.txt

Data preparation additionally requires an inference engine to serve the target model when regenerating answers; see scripts/data/README.md for details.

Workflow

Run the stages in order — each stage's output feeds the next:

Data Preparation — download prompts, regenerate target answers, and build the target cache.
Training — train a draft model against the cached target outputs.
Evaluation — measure speculative-decoding acceptance on benchmark tasks.

Data Preparation

See scripts/data/README.md for the step-by-step data pipeline:

download and split training data,
regenerate answers,
prepare the target cache (storage warning: this can be very large — roughly 38 TB for the default Qwen/Qwen3-4B setting).

Training

bash scripts/train/train.sh

train.sh launches train.py, which spawns one worker per visible GPU. Select the algorithm and target model by pointing config_path at one of the configs under config/ (e.g. config/dspark/dspark_qwen3_4b.py); see the script header for the full list of configs, how to override config_path / target_cache_dir, and how to use --opts to override individual config fields. Checkpoints are written to ~/checkpoints///step_*.

Hardware: the default configs and scripts assume a single node with 8 GPUs. For fewer GPUs, reduce CUDA_VISIBLE_DEVICES.

Evaluation

bash scripts/eval/eval.sh

eval.sh runs eval.py against a trained draft checkpoint over the speculative-decoding benchmarks in eval_datasets/ (gsm8k, math500, aime25, humaneval, mbpp, livecodebench, mt-bench, alpaca, arena-hard-v2). Set:

target_name_or_path — the target model the draft was trained against (e.g. Qwen/Qwen3-4B),
draft_name_or_path — the draft checkpoint, e.g. ~/checkpoints/deepspec/dspark_block7_qwen3_4b/step_latest, or one of the Hugging Face repo IDs listed in Released Checkpoints.

Released Checkpoints

The checkpoint

Description

DeepSpec: a full-stack codebase for training and evaluating speculative decoding algorithms

Related repositories

Similar projects matched by category, topics, and programming language.

lopopolo

Featured

harness-engineering

Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.

AI & Machine LearningAI Agents

2,390

slvDev

Featured

esp32-ai

A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.

AI & Machine LearningLarge Language Models

1,960

littledivy

Featured

mimic

mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.

AI & Machine Learning

1,482