为什么值得关注

该工作以严格、透明和自我批评的方式对一种新颖的无权重重能力扩展技术进行了实证研究，包含详细的 A/B 测试、负面结果以及交互式 HTML 报告，既展示了成功也揭示了持续存在的局限。

适合谁使用

研究混合专家架构的研究人员
希望在不重新训练的情况下增加模型容量的实践者
对选择性微调和路由器操作感兴趣的开发者
探索前沿模型内部机制的开源模型黑客

典型使用场景

在不增加参数的情况下提升已有 MoE 模型的推理计算预算
理解路由器 top-k、专家激活与任务性能之间的关系
评估事后能力扩展在微调和强化学习流程中的可行性
基准测试 SFT 在编码和推理任务中风格转移与知识转移的区别

项目优势

所有测量均为配对、同条件下的 A/B 测试，并使用了精确 McNemar 显著性检验
提供了完整的选择性专家微调 (ESFT) 框架，支持可切换的残差增量
公开发布了包含内联 SVG 图表的交互式 HTML 报告
透明地记录了失败模式，如容量上限和生成长度风险

使用前须知

修复效果已得到验证但未完成，点估计值仍略低于原生 top-8 基线
选择性微调在训练约 38% 后饱和，表明存在容量上限，可能需要全 FFN 或解冻路由器
该方法目前处于研究阶段，尚未发布为成品模型；正在进行的拒绝微调和强化学习可能改变结果

README 快速开始

Qwen3.6-35B-A3B → A6B: router k-expansion and selective healing

Force a top-8 Mixture-of-Experts router to fire 32 experts per token, then heal the resulting degradation with a router-frozen expert fine-tune — measured, honestly, with paired A/B tests.

The whole idea in one picture: identical weights, one routing constant changed. The added experts (orange) are load-bearing — 54% of the gate mass — but were never trained to collaborate. Interactive version →

This repository is a research log. It documents an ongoing campaign to turn Qwen3.6-35B-A3B (35B total / ~3B active, 256 experts per layer, native top-8 routing) into a "35B-A6B" regime by raising the inference-time router top-k to 32 (~6.6B active params/token, at zero new weights), and then repairing the accuracy it costs. Everything here is a measured account — including the losses and the failure modes, which are the most useful part.

Status: ongoing. Supervised mixture-healing is demonstrated but not complete (point estimates are still slightly negative). Rejection fine-tuning, GRPO reinforcement learning, and a Terminal-Bench evaluation are in progress — see Status below.

The reports

Two self-contained, theme-aware HTML reports (no external assets, inline SVG charts):

Report	What it covers
`docs/arch_a3b_a6b.html`	Architecture & tuning. What the top-k edit changes inside the MoE block, why the added ranks 9–32 carry 54% of the gate mass, and the residual-delta ESFT recipe step by step.
`docs/healing_report.html`	Measurements. The monotonic damage curve across k, three healing generations vs base@k8 (with p-values), and the generation-length hazard where SFT transfers style before knowledge.
`docs/workflow_timeline.html`	Workflow. A three-node timeline of the healing → probe → branch → RL pipeline: how the diagnosis re-ordered around making the router mobile (joint expert+router training) before deciding whether full-FFN capacity is needed.

There is also a small landing page at docs/index.html.

The pipeline at a glance

*Left→right is time; the three lanes are the compute nodes. The

a6b-k-expansion