0xwilliamortiz
精选
ponytail-improved
Ponytail is a plugin for AI coding agents that enforces a disciplined ladder of reuse before writing code, reducing code volume by roughly 54% while preserving safety.
AI 仓库情报摘要
FR-AI / ANALYSIS为什么值得关注
它通过提供最新且经过验证的列表（自动刷新星标）脱颖而出，并且超越静态目录，提供了可执行的代理技能，一条命令即可安装，使开发者能够直接实现可观测性工作流。
适合谁使用
- LLM 应用开发者
- MLOps / AIOps 工程师
- 评估可观测性平台的团队
- 对 LLM 监控感兴趣的开源贡献者
典型使用场景
- 根据具体约束条件找到合适的追踪、评估或护栏工具
- 安装代理技能以添加追踪、评估或降低 LLM 成本
- 比较可自托管与商业化的可观测性平台
- 使用内含的指南为受监管行业实现合规追踪
项目优势
- 全面且积极维护的列表，自动从 GitHub API 刷新星标数
- 包含 26 个经过验证的代理技能，平均触发 F1 达 0.99，可自动化 LLMOps 任务
- 除整理外，还提供原创工具（genai_trace.py）和指南（金融可观测性）
- CI 验证确保技能保持可用，星标数保持最新
使用前须知
- 列表可能需要针对特定平台功能做二次核查
- 代理技能主要面向 Claude Code 环境设计
- 对比表格可能未涵盖所有边界情况或最新特性
README 快速开始
Awesome LLM Observability
A current, hand-checked list of 60+ LLM observability tools, plus 26 agent skills you install in one command.
Star counts refresh themselves. The skills get validated in CI. More on both further down.
The tooling for watching LLM apps in production is a fast-moving mess of overlapping projects, and every list I found had stars from 2023 on it. So I kept my own, verified the entries, and wired up a job to keep the numbers honest. Use it to find the right tool without re-researching the whole space.
Legend: 🟢 open-source · 🔵 open-core / hybrid · 🟠 commercial (public repo is an SDK/client only - low star counts don't reflect the product). Star counts are pulled live from the GitHub API and auto-refreshed weekly by CI (tools/refresh_stars.py), so they stay current instead of rotting.
Contents
- What is LLM Observability?
- 🧰 Skills (Batteries Included)
- Platform Comparison (At a Glance)
- Tracing & Observability Platforms
- Evaluation Frameworks
- Prompt Management & Experimentation
- LLM Gateways & Proxies
- Instrumentation & Standards
- Guardrails & Safety Monitoring
- Self-Hosted / Open-Source First
- Research & Benchmarks
- Learning Resources
- Guides & Tools (Original)
- How to Choose
- Contributing
What is LLM Observability?
Regular observability assumes your system is deterministic. LLM apps aren't. They make things up, drift as inputs change, quietly burn tokens, and fail without ever throwing an error. So you end up watching diff
项目描述
50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.
相关仓库与替代方案
根据分类、Topic 和编程语言匹配的相似项目。
deerwork-ai
精选
deer-workflow
An open-source Dynamic Workflow runtime that combines deterministic TypeScript orchestration with replaceable Agent runtimes.
makecindy
精选
cindy
Cindy is an open-source AI agent that runs locally on your machine, integrates multiple AI harnesses and models, and provides memory, skills, and automation to perform real work in your projects and apps.