slvDev
Featured
esp32-ai
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
online-kl-shampoo-release
Online KL Shampoo (OKLS) is a zero-staleness Kronecker-factored optimizer that approximates full-matrix AdaGrad for large-scale language model training, achieving 1.45× parameter efficiency over Muon with comparable throughput.
Stars
26
7-day growth
No data
Forks
2
Open issues
0
License
No data
Last updated
2026-07-28
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It combines KL-optimal Kronecker preconditioning with a novel scaled Chebyshev inverse-root method and zero-staleness covariance updates, demonstrating significant parameter efficiency gains in scaling experiments.
Who it is for
- Researchers working on large language model optimization
- Practitioners seeking memory-efficient training methods
- Developers interested in advanced Kronecker-factored optimizers
- Machine learning engineers focused on training throughput
Use cases
- Training transformer-based language models
- Optimizing matrix parameters in deep neural networks
- Benchmarking parameter efficiency against Muon and AdamW
- Research on second-order optimization methods at scale
Strengths
- Zero-staleness updates ensure information from the current gradient is used immediately
- Scaled CANS Coupled Newton–Schulz provides efficient inverse-root computation with 27 GEMMs
- Achieves 1.45× parameter efficiency over Muon while retaining 98% throughput
- Integrated muP shape scaling and AdamC decoupled weight decay for stability
Considerations
- Requires CUDA GPU and recent PyTorch with specific FP16/FP32 support
- Only applicable to matrix parameters; embeddings, norms, biases need separate optimizer
- Pure PyTorch implementation without distributed offload or fused kernels as in production system
- Higher memory usage due to storing FP32 momentum and two copies of covariance factors
README quick start
Usage
Related repositories
Similar projects matched by category, topics, and programming language.
gavamedia
Featured
deltafin
Deltafin is a research project that runs the 2.8-trillion-parameter Mixture-of-Experts model Kimi K3 on a single Apple Silicon Mac (e.g., M1 Max with 64 GB) at about 16 seconds per token, using exact, reproducible inference with local or streaming expert loading.
jamesob
Featured
local-llm
A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.