0xwilliamortiz
Featured
ponytail-improved
Ponytail is a plugin for AI coding agents that enforces a disciplined ladder of reuse before writing code, reducing code volume by roughly 54% while preserving safety.
awesome-llm-observability
A curated, hand-checked list of 60+ LLM observability tools, along with 26 one-command-installable agent skills for common LLMOps tasks, with auto-refreshing star counts and CI-validated skills.
Stars
26
7-day growth
No data
Forks
8
Open issues
0
License
NOASSERTION
Last updated
2026-07-27
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It stands out by providing a current and verified list (auto-refreshed stars) and goes beyond a static directory by including executable agent skills that can be installed in one command, enabling developers to directly implement observability workflows.
Who it is for
- LLM application developers
- MLOps / AIOps engineers
- Teams evaluating observability platforms
- Open-source contributors interested in LLM monitoring
Use cases
- Finding the right tracing, evaluation, or guardrail tool for given constraints
- Installing agent skills to add tracing, evals, or cost reduction to an LLM app
- Comparing self-hostable vs commercial observability platforms
- Using the included guide for compliant tracing in regulated industries
Strengths
- Comprehensive and actively maintained list with auto-refreshed star counts from the GitHub API
- Includes 26 validated agent skills with high trigger F1 (0.99 average) for automating LLMOps tasks
- Provides original tools (genai_trace.py) and guides (finance observability) beyond curation
- CI validation ensures skills remain functional and star counts are current
Considerations
- The list may still require double-checking for specific platform features
- Agent skills are designed primarily for the Claude Code environment
- The comparison table may not capture all edge cases or latest features
README quick start
Awesome LLM Observability
A current, hand-checked list of 60+ LLM observability tools, plus 26 agent skills you install in one command.
Star counts refresh themselves. The skills get validated in CI. More on both further down.
The tooling for watching LLM apps in production is a fast-moving mess of overlapping projects, and every list I found had stars from 2023 on it. So I kept my own, verified the entries, and wired up a job to keep the numbers honest. Use it to find the right tool without re-researching the whole space.
Legend: 🟢 open-source · 🔵 open-core / hybrid · 🟠 commercial (public repo is an SDK/client only - low star counts don't reflect the product). Star counts are pulled live from the GitHub API and auto-refreshed weekly by CI (tools/refresh_stars.py), so they stay current instead of rotting.
Contents
- What is LLM Observability?
- 🧰 Skills (Batteries Included)
- Platform Comparison (At a Glance)
- Tracing & Observability Platforms
- Evaluation Frameworks
- Prompt Management & Experimentation
- LLM Gateways & Proxies
- Instrumentation & Standards
- Guardrails & Safety Monitoring
- Self-Hosted / Open-Source First
- Research & Benchmarks
- Learning Resources
- Guides & Tools (Original)
- How to Choose
- Contributing
What is LLM Observability?
Regular observability assumes your system is deterministic. LLM apps aren't. They make things up, drift as inputs change, quietly burn tokens, and fail without ever throwing an error. So you end up watching diff
Description
50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.
Related repositories
Similar projects matched by category, topics, and programming language.
deerwork-ai
Featured
deer-workflow
An open-source Dynamic Workflow runtime that combines deterministic TypeScript orchestration with replaceable Agent runtimes.
makecindy
Featured
cindy
Cindy is an open-source AI agent that runs locally on your machine, integrates multiple AI harnesses and models, and provides memory, skills, and automation to perform real work in your projects and apps.