MoonshotAI GitHub avatar

MoonEP

MoonshotAI

MoonEP is an Expert Parallelism communication library that achieves perfect token load balance across ranks using dynamic redundant experts and online planning.

Stars

718

7-day growth

No data

Forks

70

Open issues

9

License

MIT

Last updated

2026-07-28

AI repository intelligence
FR-AI / ANALYSIS

Why it is worth attention

It solves the severe imbalance problem in routed MoE by ensuring every rank always receives exactly S×K tokens, making iteration time and memory usage immune to routing skew, with zero-copy communication and static shapes.

Who it is for

  • Machine learning engineers training large-scale MoE models
  • Researchers working on distributed training and expert parallelism
  • Developers deploying MoE inference systems with unpredictable token distributions
  • Teams using NVIDIA GPUs (and future Zhenwu PPU) for MoE workloads

Use cases

  • Training MoE models with highly skewed routing without performance degradation or OOM
  • Inference serving for MoE models where token distribution varies per request
  • Scaling expert parallelism to many ranks while maintaining constant per-rank compute and memory
  • Replacing DeepEP in existing MoE training pipelines to gain imbalance immunity and faster communication

Strengths

  • Perfect token balance: every rank receives exactly S×K tokens regardless of routing skew
  • Zero-copy fused permute/unpermute eliminates comm-buffer copies and reduces latency
  • Static memory shapes prevent GPU memory fragmentation and eliminate per-layer host synchronization
  • Online planning kernel with negligible overhead selects redundant experts near-optimally

Considerations

  • Requires contiguous symmetric-memory weight tensors and a specific buffer layout, which may not integrate with all frameworks
  • Currently only supports NVIDIA GPU; Zhenwu PPU support is under review
  • Dynamic redundant experts increase memory footprint (B prefetch slots per projection) and add planning/prefetch kernels not needed in simpler libraries

README quick start

Usage

Description

MoonEP: A Perfectly Balanced Expert Parallelism Library via Dynamic Redundant Experts

Related repositories

Similar projects matched by category, topics, and programming language.

lopopolo
Featured
lopopolo GitHub avatar

harness-engineering

Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.

AI & Machine LearningAI Agents
2,390
slvDev
Featured
slvDev GitHub avatar

esp32-ai

A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.

AI & Machine LearningLarge Language Models
1,960
littledivy
Featured
littledivy GitHub avatar

mimic

mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.

AI & Machine Learning
1,482