slvDev
Featured
esp32-ai
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
CAPEval
CAPEval is a checklist-based benchmark that decouples image caption quality into Coverage (C) and Precision (P) scores across 300 images with 14,965 atomic checklist items, and links these profiles to VLM understanding and text-to-image generation.
Stars
2
7-day growth
No data
Forks
0
Open issues
0
License
Apache-2.0
Last updated
2026-07-30
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It separates coverage from precision instead of using a single captioning score, and provides a public dataset, judge code, leaderboard, and downstream evaluation integration, making it a practical resource for caption-aware model assessment.
Who it is for
- researchers working on vision-language model evaluation
- developers of image captioning or text-to-image systems
- benchmark maintainers seeking checklist-based metrics
- practitioners comparing VLM understanding and T2I generation quality
Use cases
- score custom captioning outputs for Coverage and Precision
- run the built-in captioner and judge pipeline to benchmark VLM/LLM captioners
- evaluate text-to-image generation models using downstream suites listed in the README
- validate submission JSON format before running full checklist judgment
Strengths
- clear decoupled metrics: C=100*(yes+no)/total and P=100*yes/(yes+no)
- large human-curated checklist dataset: 300 images and 14,965 atomic items across four categories
- provides built-in captioning, scoring, and pipeline scripts plus a Python API
- integrates with VLMEvalKit, GenEval, DPG-Bench, and T2I-CompBench for downstream analysis
Considerations
- paper/arXiv is still TBD, so formal methodology details are not yet peer-reviewed
- default judge is Qwen2.5-72B-Instruct, which may require significant GPU resources
- leaderboard numbers are described as preliminary and public data may change
README quick start
A Decoupled Caption Evaluation across Understanding and Generation
🏆 Leaderboard · 🤗 Hugging Face · 🔗 Downstream · 📄 Paper
A checklist-based benchmark that decouples caption quality into Coverage (C) and Precision (P) (0–100), and studies how each profile transfers to VLM understanding and T2I generation.
🔥 News
- [2026-07] 🏆 Project page & leaderboard: https://liuzhipenggg.github.io/CAPEval/
- [TBD] 📄 Paper release (arXiv).
- [2026-07] 💻 Code released: captioning, checklist judging, and C / P evaluation.
📎 Resources
- 📄 Paper: [Coming soon]
- 🌐 Project page: https://liuzhipenggg.github.io/CAPEval/
- 🤗 Hugging Face Dataset: LiuzhipengUCAS/CAPEval
- 🏆 Leaderboard: https://liuzhipenggg.github.io/CAPEval/leaderboard/
🚀 Installation
git clone https://github.com/liuzhipenggg/CAPEval.git
cd CAPEval
pip install -r requirements.txt
requirements.txt pins light deps and keeps transformers 4.x with `huggingface-hub/
caption/prompt.json
metrics/metrics.json
metrics/results_summary.json
```python
from capeval import CAPEval
score = CAPEval().evaluate_map({"SO001.jpg": "A detailed caption..."})
print(score["C"], score["P"])
print(score["per_category"])
📦 Dataset
300 images · 14,965 atomic checklist items · data/ (image/, gt_caption.jsonl, checklist.jsonl; join key img_path) · Hugging Face
Per-category C / P use the filename prefix (SO / PA / TI / DK). Optional tables in data/meta/ list category / subcategory labels (not required by the scorer).
Super-category Prefix Images Checklist items Avg. / image
Scene & ObjectSO1487,03947.6 People & ActivityPA603,16452.7 Text & InterfaceTI402,14653.6 Design & KnowledgeDK522,61650.3 Total30014,96549.9
🔗 Downstream evaluation
Paper downstream numbers use official stacks:
- Understanding: VLMEvalKit — MME, MMBench (EN/CN), ScienceQA, OCRBench, RealWorldQA, MMStar, MMVP, POPE, HallusionBench, AMBER, CV-Bench (2D/3D), HR-Bench (4K/8K), BLINK, AI2D, SEED-Bench, MMMU
- Generation: GenEval · [DPG-Bench (ELLA)](https://github.com/Tencent
Description
CAPEVAL: A DECOUPLED CAPTION EVALUATION ACROSS UNDERSTANDING AND GENERATION
Related repositories
Similar projects matched by category, topics, and programming language.
gavamedia
Featured
deltafin
Deltafin is a research project that runs the 2.8-trillion-parameter Mixture-of-Experts model Kimi K3 on a single Apple Silicon Mac (e.g., M1 Max with 64 GB) at about 16 seconds per token, using exact, reproducible inference with local or streaming expert loading.
jamesob
Featured
local-llm
A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.