makecindy
Featured
cindy
Cindy is an open-source AI agent that runs locally on your machine, integrates multiple AI harnesses and models, and provides memory, skills, and automation to perform real work in your projects and apps.
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
Introduces a rigorous, transparent benchmarking framework for fine-tuning speedups, inspired by modded-nanogpt, where any sub-30M-parameter LoRA submission can be independently verified for free on identical hardware, and techniques must generalize across two different model families and tasks to prove robustness.
Who it is for
- ML practitioners fine-tuning small models on a budget
- Researchers developing efficient fine-tuning methods (LoRA variants, quantization, kernels)
- Competition-minded developers who enjoy optimizing training pipelines for speed
- Engineers seeking reproducible benchmarks to compare fine-tuning techniques
Use cases
- Comparing wall-clock speed of different LoRA configurations under identical conditions
- Developing and testing new training tricks (data pruning, packing, fused kernels) with a clear metric
- Learning optimal fine-tuning practices for 1–2B parameter models on a single GPU
- Building a public portfolio of reproducible, time-optimized model adaptation recipes
Strengths
- Frozen hardware and task ensure apples-to-apples comparison; all records independently re-run 3× with fresh seeds
- Free to compete and verify via Modal's monthly credits; automated CI and security review maintain integrity
- Two tracks with different model families (Qwen2.5, SmolLM2) and tasks (math QA vs. reading comprehension) to prevent overfitting
- Detailed verification reports and write-ups are publicly committed, serving as a lab notebook of techniques and rejected variants
Considerations
- Official timing is tied to a specific cloud provider (Modal) and GPU (L40S), which may not be accessible to all
- Limited to models ≤1.7B parameters and LoRA with ≤30M trainable params; techniques may not scale to larger models
- Submission process requires code review and automated security screening; not a fully frictionless benchmark
README quick start
LoRA Speedrun 🏁
How fast can you LoRA-fine-tune Qwen2.5-1.5B to ≥ 57% on GSM8K — on a single L40S?
This is modded-nanogpt for fine-tuning: a frozen task, frozen hardware, and a public leaderboard of wall-clock records. Every record is independently re-run 3× with fresh seeds on identical hardware before it counts.
Attempting and verifying are free: official timing runs on a Modal L40S sandbox, and Modal's free monthly compute credits cover full runs — so anyone can compete, and anyone can re-verify any record with one command.
Leaderboard
Track 1 — GSM8K · Qwen2.5-1.5B · target ≥ 57.0% · 1× L40S
Current record: 0m 44s by @slippylolo — 0.75-epoch six-target BF16 LoRA: shortest-4k pruning with down_proj dropped (13.76M params), deterministic best-fit packing, per-layer fused base+LoRA-A q/k/v & gate/up GEMMs, chunked completion-only CE.
| # | Date | Author | Train time | GSM8K/EM | Δ | Technique |
|---|---|---|---|---|---|---|
| 0 | 2026-07-18 | @Saivineeth147 | 11m 57s | 59.4% | — | Baseline: plain LoRA r=16 on all linear layers, 3 epochs, cosine LR. No tricks. (report) |
| 1 | 2026-07-18 | @Saivineeth147 | 6m 05s | 61.1% | −49% | Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy. (report) |
| 2 | 2026-07-20 | @stared | 1m 53s | 59.6% | −69% | One 4e-4 epoch over 3,000 examples with 2x optimizer updates (integrated-LR), 512-token packs. First outside record. (report) |
| 3 | 2026-07-20 | @stared | 1m 44s | 60.3% | −8% | Shortest-4k data pruning, 1 aggressive-LR epoch, > annotations stripped, custom GPU-resident packed loop, chunked completion-only CE (no full logits). (report) |
| 4 | 2026-07-25 | @slippylolo | 0m 44s | 59.6% | −58% | 0.75-epoch six-target BF16 LoRA: short |
Description
Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning.
Related repositories
Similar projects matched by category, topics, and programming language.
uzairansaruzi
Featured
hermex
Hermex is a native SwiftUI iPhone app that lets you control a self-hosted Hermes AI agent directly from your phone, with no subscriptions, tracking, or third-party relay.
hahhforest
Featured
pi-textbook
A hands-on course that walks through building a Pi-style coding agent from scratch across 15 checkpoints, starting from an offline agent trajectory.