MoonshotAI
Featured
Kimi-K3
Kimi K3 is an open-weight, 2.8T-parameter native multimodal agentic model with a 1M-token context window, designed for frontier coding, knowledge work, and reasoning tasks.
a6b-k-expansion
This repository documents an ongoing research campaign to artificially expand the active parameters of Qwen3.6-35B-A3B from 3B to 6.6B by raising its MoE router top-k from 8 to 32 without adding any new weights, then repairing the resulting accuracy loss via selective expert fine-tuning.
Stars
11
7-day growth
No data
Forks
0
Open issues
0
License
MIT
Last updated
2026-07-10
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It presents a rigorous, transparent, and self-critical empirical investigation of a novel weight-free capacity expansion technique, including detailed A/B tests, negative results, and interactive HTML reports that reveal both successes and persistent limitations.
Who it is for
- Researchers studying mixture-of-experts architectures
- Practitioners seeking to increase model capacity without retraining
- Developers interested in selective fine-tuning and router manipulation
- Open-source model hackers exploring frontier model internals
Use cases
- Boosting inference-time compute budget on existing MoE models without adding parameters
- Understanding the relationship between router top-k, expert activation, and task performance
- Evaluating the feasibility of post-hoc capacity expansion for fine-tuning and RL pipelines
- Benchmarking the transfer of style vs knowledge in SFT for coding and reasoning tasks
Strengths
- All measurements are paired, same-condition A/B tests with exact McNemar significance
- Includes full framework for selective expert fine-tuning (ESFT) with toggleable residual deltas
- Provides publicly viewable interactive HTML reports with inline SVG charts
- Transparently documents failure modes such as the capacity ceiling and generation-length hazard
Considerations
- Healing is demonstrated but not complete – point estimates remain slightly negative vs the native top-8 baseline
- Selective fine-tuning saturates at ~38% of training, indicating a capacity ceiling that may require full-FFN or router unfreezing
- The method is currently research-stage and not released as a finished model; ongoing rejection FT and RL may change results
README quick start
Qwen3.6-35B-A3B → A6B: router k-expansion and selective healing
Force a top-8 Mixture-of-Experts router to fire 32 experts per token, then heal the resulting degradation with a router-frozen expert fine-tune — measured, honestly, with paired A/B tests.
The whole idea in one picture: identical weights, one routing constant changed. The added experts (orange) are load-bearing — 54% of the gate mass — but were never trained to collaborate. Interactive version →
This repository is a research log. It documents an ongoing campaign to turn Qwen3.6-35B-A3B (35B total / ~3B active, 256 experts per layer, native top-8 routing) into a "35B-A6B" regime by raising the inference-time router top-k to 32 (~6.6B active params/token, at zero new weights), and then repairing the accuracy it costs. Everything here is a measured account — including the losses and the failure modes, which are the most useful part.
Status: ongoing. Supervised mixture-healing is demonstrated but not complete (point estimates are still slightly negative). Rejection fine-tuning, GRPO reinforcement learning, and a Terminal-Bench evaluation are in progress — see Status below.
The reports
Two self-contained, theme-aware HTML reports (no external assets, inline SVG charts):
| Report | What it covers |
|---|---|
docs/arch_a3b_a6b.html | Architecture & tuning. What the top-k edit changes inside the MoE block, why the added ranks 9–32 carry 54% of the gate mass, and the residual-delta ESFT recipe step by step. |
docs/healing_report.html | Measurements. The monotonic damage curve across k, three healing generations vs base@k8 (with p-values), and the generation-length hazard where SFT transfers style before knowledge. |
docs/workflow_timeline.html | Workflow. A three-node timeline of the healing → probe → branch → RL pipeline: how the diagnosis re-ordered around making the router mobile (joint expert+router training) before deciding whether full-FFN capacity is needed. |
There is also a small landing page at docs/index.html.
The pipeline at a glance
*Left→right is time; the three lanes are the compute nodes. The
Related repositories
Similar projects matched by category, topics, and programming language.
xuchonglang
Featured
investing-for-beginners
A structured investing guide for Chinese beginners covering US stocks, options, and cryptocurrency, with focus on foundational concepts and risk awareness.
Krishnagangwal
Featured
CS-Fundamentals
A curated collection of Computer Science fundamentals (PDFs, notes, cheatsheets, interview question banks) for placement preparation, covering seven core subjects plus general resources.