为什么值得关注

引入了一个严谨、透明的微调加速基准测试框架，灵感来自modded-nanogpt；任何参数不超过30M的LoRA提交都可在相同硬件上免费独立验证，且技术必须在两个不同模型系列和任务上泛化才能证明其鲁棒性。

适合谁使用

预算有限、微调小模型的机器学习从业者
研究高效微调方法（LoRA变体、量化、内核）的研究人员
喜欢优化训练流程速度的竞赛型开发者
需要可复现基准来比较微调技术的工程师

典型使用场景

在相同条件下比较不同LoRA配置的挂钟时间速度
开发和测试新的训练技巧（数据剪枝、打包、融合内核）并采用明确指标
学习在单GPU上对1–2B参数模型进行最优微调实践
构建公开的、可复现的时间优化模型适配方案作品集

项目优势

固定硬件和任务确保公平比较；所有记录均使用新种子独立重复运行3次
利用Modal的月度免费额度可免费参与和验证；自动化CI和安全审查维护完整性
两个赛道采用不同模型族（Qwen2.5、SmolLM2）和任务（数学问答 vs 阅读理解），防止过拟合
详细的验证报告和写入记录公开提交，作为技术笔记和已拒绝变体的实验记录

使用前须知

官方计时绑定特定云提供商（Modal）和GPU（L40S），可能并非所有人都能使用
仅限于≤1.7B参数的模型和≤30M可训练参数的LoRA；技术可能无法扩展到更大模型
提交过程需要代码审查和自动化安全筛查，并非完全无摩擦的基准测试

README 快速开始

LoRA Speedrun 🏁

How fast can you LoRA-fine-tune Qwen2.5-1.5B to ≥ 57% on GSM8K — on a single L40S?

This is modded-nanogpt for fine-tuning: a frozen task, frozen hardware, and a public leaderboard of wall-clock records. Every record is independently re-run 3× with fresh seeds on identical hardware before it counts.

Attempting and verifying are free: official timing runs on a Modal L40S sandbox, and Modal's free monthly compute credits cover full runs — so anyone can compete, and anyone can re-verify any record with one command.

Leaderboard

Track 1 — GSM8K · Qwen2.5-1.5B · target ≥ 57.0% · 1× L40S

Current record: 0m 44s by @slippylolo — 0.75-epoch six-target BF16 LoRA: shortest-4k pruning with down_proj dropped (13.76M params), deterministic best-fit packing, per-layer fused base+LoRA-A q/k/v & gate/up GEMMs, chunked completion-only CE.

#	Date	Author	Train time	GSM8K/EM	Δ	Technique
0	2026-07-18	@Saivineeth147	11m 57s	59.4%	—	Baseline: plain LoRA r=16 on all linear layers, 3 epochs, cosine LR. No tricks. (report)
1	2026-07-18	@Saivineeth147	6m 05s	61.1%	−49%	Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy. (report)
2	2026-07-20	@stared	1m 53s	59.6%	−69%	One 4e-4 epoch over 3,000 examples with 2x optimizer updates (integrated-LR), 512-token packs. First outside record. (report)
3	2026-07-20	@stared	1m 44s	60.3%	−8%	Shortest-4k data pruning, 1 aggressive-LR epoch, > annotations stripped, custom GPU-resident packed loop, chunked completion-only CE (no full logits). (report)
4	2026-07-25	@slippylolo	0m 44s	59.6%	−58%	0.75-epoch six-target BF16 LoRA: short

项目描述

Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning.

lora-speedrun