cactus-hybrid

cactus-compute

Cactus Hybrid 是一个小型设备端模型，它在检查点内嵌入了置信度探针，能够将低置信度查询路由到更大的模型，同时返回结构化的置信度分数。

Stars

180

7 天增长

暂无数据

Fork 数

开放 Issue

开源协议

MIT

为什么值得关注

它提供了一种实用的混合推理系统：小型模型仅需将 15–55% 的查询转发即可匹配更大的模型（Gemini 3.1 Flash-Lite），并且在文本、视觉和音频基准上达到 0.814 的平均 AUROC，即使未在音频数据上训练也能良好泛化。

适合谁使用

开发边缘或设备端 AI 应用的开发者
优化生产环境中成本和延迟的机器学习工程师
研究置信度估计和模型路由的研究人员
需要私有本地推理并自带自动回退方案的用户

典型使用场景

设备端问答，当不确定时自动切换到云端大模型
成本敏感的推理流水线，最大限度减少对大模型的调用
私有本地 AI 助手，仅在必要时升级查询
需要校准置信度的多模态任务（文本、视觉、音频）

项目优势

内建结构化的置信度分数（0–1），无需从回答文本中解析
在文本、视觉和音频基准上平均 AUROC 达 0.814，包括零样本音频
小型模型（Gemma 4 E2B）仅需 15–55% 的转发即可匹配 Gemini 3.1 Flash-Lite
开源（MIT），并提供多种集成方式（Python、MLX、Transformers、llama.cpp）

使用前须知

量化会降低转发率，需要针对不同量化方法进行独立基准测试
Transformers 集成需要特定版本范围（5.5.4–5.6）并显式指定设备
llama.cpp 支持需手动应用补丁

README 快速开始

Cactus Hybrid

A small, on-device model is fast and private, but sometimes wrong. At Cactus we post-train models to know when they are wrong: we ship probes inside the checkpoint that score every answer with a confidence between 0 and 1, returned as structured data (never parsed out of the answer text). Answer on-device when confidence is high; you can re-route to a bigger model when it's low:

if confidence ", answer)[-1]
answer = re.sub(r"^(thought|final)\b\s*", "", answer).strip()
print(answer)
print("confidence:", model.last_confidence)

Transformers

# pip install "transformers>=5.5.4,<5.6" torch   (5.14+ segfaults on this checkpoint)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Cactus-Compute/gemma-4-e2b-it-hybrid"
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype="auto").to(device)

messages = [{"role": "user", "content": "What is the capital of France?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(device)
out = model.generate(**inputs, return_confidence=True, max_new_tokens=512)

print(tokenizer.decode(out.sequences[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
print("confidence:", out.confidence)

Load the model with an explicit .to(device), not device_map="auto": the probe scores generations outside the module forward() path, so weights that accelerate offloads (left on the meta device) crash the confidence read.

llama.cpp

llama.cpp is C++, so the probe is a patch you compile into the engine (see patches/llama.cpp/). Build the patched server once:

git clone https://github.com/cactus-compute/cactus-hybrid && cd cactus-hybrid
./patches/llama.cpp/install.sh && rehash

Then serve and query it like any llama-server — the response carries a top-level confidence field:

llama-server -hf Cactus-Compute/gemma-4-e2b-it-hybrid-GGUF:Q4_K_M --jinja

curl -s http://localhost:8080/v1/chat/completions \
  -d '{"messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":512}' \
  |

项目描述

On-device models that know when they're wrong: every answer carries a confidence score for cloud handoff.