提供了对 Apple Silicon 上多种量化 Laguna S 2.1 变体的全面、可复现的比较，包含 token 吞吐量、峰值 MLX 内存、任务得分等详细指标，以及长上下文检索测试和针对不同内存预算的量化阶梯。
This repository contains a reproducible local harness for comparing Laguna S 2.1 MLX quantizations on Apple Silicon. Each run records the raw output and task score along with token and prefill throughput, peak MLX memory, load time, package versions, model revision, and machine metadata.
This work was sponsored by DWS LLC.
The new leader on the M5 Max is pipenetwork/Laguna-S-2.1-MLX-2bit through MLX-LM. It passed all 38 assertions, averaged 63.86 generation tok/s across the three generation tasks, and reached 68.49 tok/s in the fixed 256-token decode. It needs the conversion's bundled Laguna loader, so the harness loads that file from a pinned revision without modifying the installed MLX-LM package. The measurements are in BENCHMARK_RESULTS.md.
These results were measured on a 128 GB Apple M5 Max using macOS 27.0, Python 3.13.12, MLX 0.32.0, mlx-vlm 0.6.6, and MLX-LM 0.31.3. The score combines three generation tasks and three agentic coding tasks. Throughput is weighted across the generation tasks.
The left panel reproduces the Terminal-Bench 2.1 figures in Poolside's model card. The right panel comes from this repository's committed CSV and selects the frontier plus one useful representative from each quant/runtime family; the table and CSV below contain every measured variant. The panels use separate scales and are not the same evaluation. Regenerate the SVG with:
uv run --frozen laguna-bench chart
Rows are sorted by overall score, then weighted generation throughput.
| Quant | Overall score | Generation | Agentic | Generation tok/s | Peak GB | Suite time |
|---|---|---|---|---|---|---|
pipenetwork/Laguna-S-2.1-MLX-2bit | 1.000 | 1.000 | 1.000 | 63.86 | 39.31 | 87.47s |
mlx-community/Laguna-S-2.1-oQ2e | 1.000 | 1.000 | 1.000 | 40.85 | 37.77 | 87.38s |
unsloth/Laguna-S-2.1-GGUF UD-Q2_K_XL | 0.875 | 0.750 | 1.000 | 60.37 | 37.99 RSS | 40.17s |
pipenetwork/Laguna-S-2.1-MLX-3bit | 0.875 | 0.750 | 1.000 | 58.63 | 52.88 | 65.82s |
unsloth/Laguna-S-2.1-GGUF UD-Q3_K_M | 0.875 | 0.750 | 1.000 | 51.27 | 51.34 RSS | 65.19s |
unsloth/Laguna-S-2.1-GGUF UD-IQ4_XS | 0.875 | 0.750 | 1.000 | 51.04 | 54.64 RSS | 54.23s |
unsloth/Laguna-S-2.1-GGUF UD-IQ4_NL | 0.875 | 0.750 |
Community benchmarks and scripts for running Poolside Laguna S 2.1
根据分类、Topic 和编程语言匹配的相似项目。
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.