Offers a thorough, reproducible comparison of many quantized Laguna S 2.1 variants on Apple Silicon, with detailed metrics including token throughput, peak MLX memory, and task scores, plus long-context retrieval tests and a curated quant ladder for different memory budgets.
This repository contains a reproducible local harness for comparing Laguna S 2.1 MLX quantizations on Apple Silicon. Each run records the raw output and task score along with token and prefill throughput, peak MLX memory, load time, package versions, model revision, and machine metadata.
This work was sponsored by DWS LLC.
The new leader on the M5 Max is pipenetwork/Laguna-S-2.1-MLX-2bit through MLX-LM. It passed all 38 assertions, averaged 63.86 generation tok/s across the three generation tasks, and reached 68.49 tok/s in the fixed 256-token decode. It needs the conversion's bundled Laguna loader, so the harness loads that file from a pinned revision without modifying the installed MLX-LM package. The measurements are in BENCHMARK_RESULTS.md.
These results were measured on a 128 GB Apple M5 Max using macOS 27.0, Python 3.13.12, MLX 0.32.0, mlx-vlm 0.6.6, and MLX-LM 0.31.3. The score combines three generation tasks and three agentic coding tasks. Throughput is weighted across the generation tasks.
The left panel reproduces the Terminal-Bench 2.1 figures in Poolside's model card. The right panel comes from this repository's committed CSV and selects the frontier plus one useful representative from each quant/runtime family; the table and CSV below contain every measured variant. The panels use separate scales and are not the same evaluation. Regenerate the SVG with:
uv run --frozen laguna-bench chart
Rows are sorted by overall score, then weighted generation throughput.
| Quant | Overall score | Generation | Agentic | Generation tok/s | Peak GB | Suite time |
|---|---|---|---|---|---|---|
pipenetwork/Laguna-S-2.1-MLX-2bit | 1.000 | 1.000 | 1.000 | 63.86 | 39.31 | 87.47s |
mlx-community/Laguna-S-2.1-oQ2e | 1.000 | 1.000 | 1.000 | 40.85 | 37.77 | 87.38s |
unsloth/Laguna-S-2.1-GGUF UD-Q2_K_XL | 0.875 | 0.750 | 1.000 | 60.37 | 37.99 RSS | 40.17s |
pipenetwork/Laguna-S-2.1-MLX-3bit | 0.875 | 0.750 | 1.000 | 58.63 | 52.88 | 65.82s |
unsloth/Laguna-S-2.1-GGUF UD-Q3_K_M | 0.875 | 0.750 | 1.000 | 51.27 | 51.34 RSS | 65.19s |
unsloth/Laguna-S-2.1-GGUF UD-IQ4_XS | 0.875 | 0.750 | 1.000 | 51.04 | 54.64 RSS | 54.23s |
unsloth/Laguna-S-2.1-GGUF UD-IQ4_NL | 0.875 | 0.750 |
Community benchmarks and scripts for running Poolside Laguna S 2.1
Similar projects matched by category, topics, and programming language.
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.