A zero-dependency Python tool that audits AI agent skill catalogs to reveal which skills are actually loadable and findable, then generates fix plans for poorly described ones.

Stars

5

7-day growth

No data

Forks

0

Open issues

0

License

MIT

Last updated

2026-07-30

AI repository intelligence
FR-AI / ANALYSIS

Why it is worth attention

It exposes a hidden reality of agent skill management—most skills on disk are dead copies or unfindable—and provides measurable before/after metrics, with no API keys or model calls required.

Who it is for

  • Developers building or maintaining Claude Code, OpenClaw, or Hermes skill collections
  • AI agent engineers debugging why certain skills never get invoked
  • Teams that manage large internal tool catalogs for LLM agents
  • LLM tooling evaluators who need a transparent routing benchmark

Use cases

  • Auditing a skills folder to count actually loadable skills and flag duplicates or name shadows
  • Identifying unwritable or ambiguous skill descriptions that hurt retrieval
  • Generating an LLM-ready rewrite plan (rewrites.md) for weak skill labels
  • Measuring router accuracy before and after description fixes to confirm improvement

Strengths

  • Zero runtime dependencies—requires only Python 3.10+ and runs offline in milliseconds
  • Runtime-agnostic, working with any system that loads SKILL.md files
  • Provides actionable, evidence-based output like self-retrieval failures and collision scores
  • Includes example fixture skills and a documented real-world case study with concrete metrics

Considerations

  • Lexical BM25 retrieval only; it cannot handle requests that share no vocabulary with the correct skill
  • Eval sets do not transfer between machines because every expectation must name a skill installed locally
  • Usage tracking misses skills fired by cron, hooks, or plugin auto-loads, so 'never used' is not proof of uselessness
  • Rewrites of plugin-owned skills will be overwritten by the next plugin upgrade

README quick start

skill-audit-router

Your AI agent has hundreds of skills. Most of them can't be found.

This tells you which ones — and fixes them.

Works with Claude Code, OpenClaw, Hermes, or anything that loads SKILL.md files.


The idea, in plain words

Imagine your agent has a toolbox with 200 tools. Every tool has a label on it.

When you ask the agent to do something, it doesn't look inside the tools. It reads the labels and picks one.

Two things go wrong, and you can't see either of them:

🧰 Half the tools aren't actually in the box. They're in a drawer, a spare box in the garage, or they're copies of the same hammer. You think you have 200. You might have 80.

🏷️ Half the labels are unreadable. A tool labelled "Helper mode" will never be picked for "fix my leaky tap" — because nothing in the label matches what you asked for.

This tool opens the box, counts what's really there, and tells you which labels nobody can read. Then it writes you a to-do list for fixing them.



Try it in 30 seconds

No setup. No install. No API key. It ships with four example skills so you can see exactly what it does before pointing it at anything of yours.

git clone https://github.com/rushindrasinha/skill-audit-router.git
cd skill-audit-router
python3 skill_audit.py --roots examples/fixture-skills --include-workspace

That's the screenshot above. Four example skills, each teaching one lesson:

Example skillScoreWhat it shows you
good-example86 🟢Says what it makes, lists the exact words a user would type, and says what it's not for
confusable-a60 🟡Nearly identical to its twin — the tool catches the collision
confusable-b60 🟡…the other half of that pair, flagged at margin 0.038
weak-example21 🔴Labelled "Helper mode." — unfindable, and the report says exactly why

Why --include-workspace? The examples live in this repo, not in your agent's real skills folder. By default the audit ignores anything your agent can't actually load — which is the whole point. This flag says "look at them anyway."


The problem, measured

Run it on a

Description

Find out which of your agent's skills can actually be found, then route a query to the right one. Claude Code / OpenClaw / Hermes. Pure stdlib.

Related repositories

Similar projects matched by category, topics, and programming language.

lopopolo
Featured
lopopolo GitHub avatar

harness-engineering

Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.

AI & Machine LearningAI Agents
2,390
MoonshotAI
Featured
MoonshotAI GitHub avatar

Kimi-K3

Kimi K3 is an open-weight, 2.8T-parameter native multimodal agentic model with a 1M-token context window, designed for frontier coding, knowledge work, and reasoning tasks.

AI & Machine LearningAI Agents
3,348
mshumer
Featured
mshumer GitHub avatar

Claude-of-Duty

A browser-based first-person shooter built entirely with procedural generation and orchestrated AI agents, featuring 55k lines of Three.js/WebGL2 code and no art assets.

AI & Machine LearningAI Agents
1,234