uzairansaruzi
Featured
hermex
Hermex is a native SwiftUI iPhone app that lets you control a self-hosted Hermes AI agent directly from your phone, with no subscriptions, tracking, or third-party relay.
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It achieves large-model inference on memory-constrained hardware through SSD-backed expert streaming and custom Metal kernels, with a complete native Mac app, CLI, and OpenAI-compatible server—all built from scratch without depending on MLX or llama.cpp.
Who it is for
- Mac users with 8 GB or limited RAM wanting to run a 26B-parameter LLM locally
- AI researchers and engineers interested in efficient MoE inference techniques
- Swift and Metal developers seeking a reference implementation for on-device AI
- Hobbyists and tinkerers exploring local LLM deployment on Apple Silicon
Use cases
- Running text-based instruction chat and completion on a MacBook Air with only 8 GB RAM
- Setting up a local, loopback OpenAI-compatible server for tool-augmented chat applications
- Experimenting with chunked prefill and SSD-backed expert caching for MoE models
- Benchmarking the performance of Gemma 4 26B-A4B on various Apple Silicon Macs
Strengths
- Runs a 26B-parameter model with only ~2 GB weight memory via expert streaming and 4-bit quantization
- Custom Metal kernels for quantized GEMV, attention, MoE, and other operations, optimized for Apple Silicon
- Comprehensive documentation including system design, benchmark results, and 102 audited experiments
- Streaming installer avoids downloading the full checkpoint, reducing disk and memory usage
Considerations
- Only supports Apple Silicon Macs with macOS 26+ and Metal 4; no support for Intel or older macOS
- Text-only inference; images, audio, and video are not supported
- Model-specific to Gemma 4 26B-A4B; not a general framework like llama.cpp or MLX
README quick start
TurboFieldfare
Gemma 4 26B-A4B inference in about 2 GB of RAM A custom Swift + Metal runtime for any Apple Silicon Mac, even the 8 GB ones.
Quick start · Local server · Benchmarks · Contribute results · How it works · Experiments · References
Memory got expensive. So I gave a 26-billion-parameter model a ~2 GB budget.
TurboFieldfare runs the instruction-tuned Gemma 4 26B-A4B without loading the entire 14.3 GB model into memory. It keeps the shared 1.35 GB core and FP16 KV cache in memory, then streams only the experts needed for each token from SSD. This is what lets the model run on Macs with 8 GB of RAM.
The runtime, streaming installer, CLI, and native Mac app are written in Swift and Metal. TurboFieldfare is model-specific rather than a wrapper around MLX or llama.cpp. The curated experiment record summarizes 102 measured results across kernels, caching, I/O, prefill, and decode.
Try it
git clone https://github.com/drumih/turbo-fieldfare.git
cd turbo-fieldfare
swift build -c release
.build/release/TurboFieldfareMac
On the first run, Swift Package Manager downloads and builds the Swift packages required by the tokenizer. The complete release build includes the foreground Mac app and its sibling decode-service executable.
When the app opens, choose Install and let TurboFieldfare fetch and repack the pinned model (about 15 GB). Once it is ready, choose Load Model, type your prompt, and press Generate.
At a glance
| Metric | Value |
|---|---|
| Model | Gemma 4 26B-A4B IT, 26B total parameters, about 3.88B active per token |
| Weights | MLX affine 4-bit, group 64; 8-bit router; 4-bit shared and routed experts |
| Memory | ~2 GB of weights and 4K KV cache |
| Storage | About 14.3 GB for the installed text-only mod |
Description
Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook
Related repositories
Similar projects matched by category, topics, and programming language.
didriksg
Crisp
Crisp is a free, open-source macOS menu bar app that adds HiDPI scaling, brightness control, and other display management features for external monitors, serving as an alternative to paid tools like BetterDisplay and Lunar.
makecindy
Featured
cindy
Cindy is an open-source AI agent that runs locally on your machine, integrates multiple AI harnesses and models, and provides memory, skills, and automation to perform real work in your projects and apps.