|
| 1 | +# Skills |
| 2 | + |
| 3 | +This directory holds Agent Skills (Anthropic Agent Skill protocol — YAML |
| 4 | +frontmatter + Markdown body) for OpenJudge. Every `SKILL.md`, wherever it |
| 5 | +lives, is **independently installable**: it carries everything it needs |
| 6 | +inline, so a single `<name>/SKILL.md` folder can be pulled out and installed |
| 7 | +on its own in Claude Code, Cursor, Codex, Hermes, OpenClaw, or any other |
| 8 | +client that speaks the same protocol — you never need this README or a |
| 9 | +suite's own README to make one skill work. |
| 10 | + |
| 11 | +Grouping below **is folder structure only** — there is no code-level registry |
| 12 | +(no `SUITE_REGISTRY`, no `DomainSuite` class). A "domain suite" is just a |
| 13 | +directory of related `<NN-name>/SKILL.md` workflows plus a `README.md` that |
| 14 | +explains how they relate. |
| 15 | + |
| 16 | +## Domain suites (multi-skill, folder-grouped) |
| 17 | + |
| 18 | +| Suite | Skills | Dimension | What it covers | |
| 19 | +|---|---|---|---| |
| 20 | +| [`eval_pipeline/`](eval_pipeline/) | 9 (`00-meta-eval` … `08-bootstrap`) | Horizontal methodology | How to build an evaluation system from scratch — dataset design, metric/grader selection, human-alignment calibration, reporting, plus RAG/prompt-regression/redteam/bootstrap scenarios. Router at `00-meta-eval`. | |
| 21 | +| [`academic-eval/`](academic-eval/) | 4 (`00-academic-router`, `01-paper-review`, `02-bib-verify`, `03-ref-hallucination-arena`) | Vertical domain | Academic paper review, BibTeX verification, and citation-hallucination benchmarking. | |
| 22 | +| [`arena-eval/`](arena-eval/) | 3 (`00-arena-router`, `01-auto-arena`, `02-ref-hallucination-arena`) | Vertical domain | Arena-style model/agent comparison — generic win-rate ranking, and citation-hallucination-specific ranking. | |
| 23 | +| [`openjudge-core/`](openjudge-core/) | 2 (`01-graders-and-pipeline`, `02-rl-reward`) | Horizontal methodology | Direct OpenJudge core-library API: graders/`GradingRunner`/aggregators, and RL reward-signal construction. No router — see the suite's own README for why. | |
| 24 | + |
| 25 | +`eval_pipeline` is the range-of-motion paradigm all the other suites above |
| 26 | +were reshaped to match (three-layer structure, directory-as-grouping, router |
| 27 | +only where sub-skills are genuinely ambiguous, every `<NN-name>/SKILL.md` |
| 28 | +independently installable). See |
| 29 | +[`docs/superpowers/specs/2026-07-09-skills-domain-suite-proposal.md`](../docs/superpowers/specs/2026-07-09-skills-domain-suite-proposal.md) |
| 30 | +for the full design rationale. |
| 31 | + |
| 32 | +### Functional test coverage |
| 33 | + |
| 34 | +`eval_pipeline`, `academic-eval`, and `arena-eval` each have an actor+judge |
| 35 | +functional test harness under their own `tests/` folder — a *runner* |
| 36 | +(shared, parametrized: `eval_pipeline/tests/run_eval_pipeline_skill_tests.py |
| 37 | +--skill-root <suite> --cases <suite>/tests/<suite>_test_cases.jsonl`), a |
| 38 | +suite-specific JSONL fixture, and a testing guide. `openjudge-core` and the |
| 39 | +isolated skills don't have this yet (see each suite's `README.md` under |
| 40 | +"Validating the skills themselves" for the ones that do). |
| 41 | + |
| 42 | +### `ref-hallucination-arena` — intentional duplication, not a bug |
| 43 | + |
| 44 | +`academic-eval/03-ref-hallucination-arena/SKILL.md` and |
| 45 | +`arena-eval/02-ref-hallucination-arena/SKILL.md` start as full copies of the |
| 46 | +same content and are **independently maintained** — no shared `references/` |
| 47 | +directory, no symlink, no sync script. This is a direct consequence of the |
| 48 | +independent-installability constraint: a skill can't rely on content that |
| 49 | +lives outside its own folder. The two copies are free to diverge (the |
| 50 | +academic copy can lean into "how trustworthy are this paper's citations", |
| 51 | +the arena copy into "how does this model compare to others"); duplicated |
| 52 | +content is an accepted cost, not a problem to solve. |
| 53 | + |
| 54 | +## Isolated skills (single-skill, no suite directory) |
| 55 | + |
| 56 | +These have no other skill in this repo referencing them, so they stay at |
| 57 | +`skills/<name>/SKILL.md` with no suite folder — the simplest possible |
| 58 | +grouping (1 skill = 1 domain). |
| 59 | + |
| 60 | +| Skill | What it covers | |
| 61 | +|---|---| |
| 62 | +| [`claude-authenticity/`](claude-authenticity/) | Detect whether an API endpoint is genuinely backed by Claude (vs. a wrapper/proxy/impersonator); extract injected system prompts. Zero OpenJudge dependency. | |
| 63 | +| [`mmx-cli/`](mmx-cli/) | Generate text/image/video/speech/music via the MiniMax AI platform. Third-party CLI wrapper, zero OpenJudge dependency. | |
| 64 | +| [`find-skills-combo/`](find-skills-combo/) | Discover and recommend combinations of skills from the **external** open agent-skills ecosystem to cover a multi-part task. Unrelated to this repo's own suite grouping. | |
| 65 | + |
| 66 | +**Future rule**: if an isolated skill above ever becomes something a domain |
| 67 | +suite depends on or wraps, move the whole folder into that suite's directory |
| 68 | +(same copy-don't-link approach as `ref-hallucination-arena` above, or a |
| 69 | +rename+move if a single owner is clear) rather than keeping it independent |
| 70 | +and cross-linking from two places. |
| 71 | + |
| 72 | +## Breaking change: path migration (this reorg) |
| 73 | + |
| 74 | +This reorg moved 6 previously-published skills to new install paths. There |
| 75 | +are no redirect stubs — installing the old path will 404; use the new path. |
| 76 | + |
| 77 | +| Old path | New path | |
| 78 | +|---|---| |
| 79 | +| `skills/paper-review/` | `skills/academic-eval/01-paper-review/` | |
| 80 | +| `skills/bib-verify/` | `skills/academic-eval/02-bib-verify/` | |
| 81 | +| `skills/ref-hallucination-arena/` | `skills/academic-eval/03-ref-hallucination-arena/` **and** `skills/arena-eval/02-ref-hallucination-arena/` (two independent copies — see above) | |
| 82 | +| `skills/auto-arena/` | `skills/arena-eval/01-auto-arena/` | |
| 83 | +| `skills/openjudge/` | `skills/openjudge-core/01-graders-and-pipeline/` | |
| 84 | +| `skills/rl-reward/` | `skills/openjudge-core/02-rl-reward/` | |
| 85 | + |
| 86 | +The migrated skills' frontmatter `name` values also change to match their |
| 87 | +new directory names. Update explicit skill invocations to use the final |
| 88 | +component of each new path (for example, `paper-review` becomes |
| 89 | +`01-paper-review`, and `openjudge` becomes `01-graders-and-pipeline`). The |
| 90 | +two reference-arena copies use distinct names: `03-ref-hallucination-arena` |
| 91 | +in `academic-eval` and `02-ref-hallucination-arena` in `arena-eval`. |
| 92 | + |
| 93 | +`SkillLoader.load_from_directory("skills")` discovers skills recursively |
| 94 | +through suite directories. A directory containing `SKILL.md` is treated as |
| 95 | +one package, including its bundled references and examples. |
| 96 | + |
| 97 | +`skills/eval_pipeline/`, `skills/claude-authenticity/`, `skills/mmx-cli/`, |
| 98 | +and `skills/find-skills-combo/` are unaffected. |
| 99 | + |
| 100 | +## Adding a new skill |
| 101 | + |
| 102 | +- **Fits an existing suite's methodology or domain?** Add it as |
| 103 | + `<suite>/<NN-name>/SKILL.md` with `name: NN-name` matching its directory, |
| 104 | + choose a name unique across the repository, update that suite's `README.md`, |
| 105 | + and add a router entry only if it creates real selection ambiguity with a |
| 106 | + sibling (see each suite's README for its router-inclusion rationale). |
| 107 | +- **Genuinely stands alone?** Add it as `skills/<name>/SKILL.md` — no suite |
| 108 | + folder needed. |
| 109 | +- **Either way**, the skill's `SKILL.md` must be installable on its own: only |
| 110 | + `name` + `description` in frontmatter, no links outside its own folder (or |
| 111 | + its own suite folder), and any non-`openjudge`/`rl-reward` pip dependency |
| 112 | + listed explicitly under Prerequisites. |
0 commit comments