Skip to content

Commit d1e0642

Browse files
chr6192cursoragentployts
authored
feat(skills): reorganize skills/ into domain suites (#189)
* feat(skills): reorganize skills/ into domain suites (Option D) Regroup the 9 flat non-eval_pipeline skills into three folder-based domain suites, matching the paradigm eval_pipeline already established (three-layer structure, directory-as-grouping, every <NN-name>/SKILL.md independently installable, no code-level registry): - academic-eval/ <- paper-review, bib-verify, + a copy of ref-hallucination-arena, plus a new 00-academic-router - arena-eval/ <- auto-arena, + a second independent copy of ref-hallucination-arena, plus a new 00-arena-router - openjudge-core/ <- openjudge, rl-reward (no router: sub-skills are mutually exclusive by description, see openjudge-core/README.md) ref-hallucination-arena is intentionally duplicated (not shared/symlinked) per the Agent Skill protocol's independent-installability constraint; the two copies are free to diverge. claude-authenticity/mmx-cli/find-skills-combo stay as standalone skills (no other skill references them). Also adds actor+judge functional-test coverage for the two new suites, reusing eval_pipeline's runner (parametrized via --skill-root/--cases/ --report-prefix, not duplicated) rather than forking it per suite. Initial audit: 21/21 test cases pass, including 3x-repeat checks on the trickiest routing-ambiguity cases (see each suite's tests/*_skill_audit.md). See docs/superpowers/specs/2026-07-09-skills-domain-suite-proposal.md for the full design rationale and migration plan (Option D, section 7). Co-authored-by: Cursor <cursoragent@cursor.com> * fix: satisfy black/isort/pylint for eval_pipeline scripts These 6 files under skills/eval_pipeline/ fail pre-commit's --all-files run in CI on plain upstream/main (pre-existing, unrelated to this PR's reorg work) — this branch was cut from main, not from cl_agentic_grader, so it doesn't carry that branch's earlier pre-commit fix for the same files. Reformats with black/isort and adds the same pylint-disable/f-string fixes already applied on cl_agentic_grader (PR #188) so this PR's CI passes too. Verified locally: pre-commit run --files <these 6 files> now passes black/isort/flake8/pylint/pyroma. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(skills): repair domain suite discovery and routing Discover nested skill suites while preserving package boundaries and handling symlink cycles. Align the nine domain skill names with their install directories, and let the arena router preserve combined evaluation goals. Add offline discovery and packaging regressions plus a router-only combined-request fixture. Validation: 24 related tests passed; Black, isort and Flake8 passed. Live actor/judge tests were not run because API credentials are unavailable. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
1 parent 0c2616c commit d1e0642

32 files changed

Lines changed: 1364 additions & 34 deletions

File tree

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,11 @@ local/
4040
tests/plt_*
4141
tests/graders/skills/*.json
4242

43+
# Skill functional-test fixtures — tracked despite the *.jsonl rule above;
44+
# only the fixtures themselves, not generated results/ (ignored below).
45+
!skills/**/tests/*_test_cases.jsonl
46+
skills/**/tests/results/
47+
4348
# Security Files
4449
*.pem
4550
*.key

‎cookbooks/skills_evaluation/skill_models.py‎

Lines changed: 32 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -209,7 +209,7 @@ def _read_text(path: Path) -> str:
209209
class SkillLoader:
210210
"""Loads Agent Skill packages from a directory.
211211
212-
Supports two directory layouts:
212+
Supports individual skills, flat collections, and nested domain suites:
213213
214214
**Single skill**::
215215
@@ -225,6 +225,15 @@ class SkillLoader:
225225
scripts/review.py
226226
paper-review/
227227
SKILL.md
228+
229+
**Domain suites** (grouping directories may be nested)::
230+
231+
skills_dir/
232+
academic-eval/
233+
01-paper-review/
234+
SKILL.md
235+
standalone/
236+
SKILL.md
228237
"""
229238

230239
@classmethod
@@ -330,7 +339,10 @@ def load_from_directory(cls, skills_dir: Union[str, Path]) -> List[SkillPackage]
330339
Args:
331340
skills_dir: Path to a directory. If the directory itself contains
332341
``SKILL.md`` it is treated as a single-skill directory; otherwise
333-
each immediate subdirectory is checked for a ``SKILL.md``.
342+
subdirectories are searched recursively. Discovery stops at
343+
any directory containing ``SKILL.md`` so bundled examples are
344+
not treated as separate packages. Each resolved directory is
345+
visited once, including when reached through a symbolic link.
334346
335347
Returns:
336348
List of successfully loaded :class:`SkillPackage` objects (may be empty).
@@ -342,20 +354,26 @@ def load_from_directory(cls, skills_dir: Union[str, Path]) -> List[SkillPackage]
342354
if not skills_dir.is_dir():
343355
raise ValueError(f"Not a directory: {skills_dir}")
344356

345-
if (skills_dir / SKILL_MD_NAME).is_file():
346-
skill = cls.load_skill(skills_dir)
347-
return [skill] if skill else []
348-
349357
skills: List[SkillPackage] = []
350-
for subdir in sorted(skills_dir.iterdir()):
351-
if not subdir.is_dir():
352-
continue
353-
if any(p in _IGNORE_DIRS for p in subdir.parts):
354-
continue
355-
skill = cls.load_skill(subdir)
356-
if skill:
357-
skills.append(skill)
358+
visited: set[Path] = set()
359+
360+
def collect(directory: Path) -> None:
361+
resolved = directory.resolve()
362+
if resolved in visited:
363+
return
364+
visited.add(resolved)
365+
366+
if (directory / SKILL_MD_NAME).is_file():
367+
skill = cls.load_skill(directory)
368+
if skill:
369+
skills.append(skill)
370+
return
371+
372+
for subdir in sorted(directory.iterdir()):
373+
if subdir.is_dir() and subdir.name not in _IGNORE_DIRS:
374+
collect(subdir)
358375

376+
collect(skills_dir)
359377
return skills
360378

361379

‎skills/README.md‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# Skills
2+
3+
This directory holds Agent Skills (Anthropic Agent Skill protocol — YAML
4+
frontmatter + Markdown body) for OpenJudge. Every `SKILL.md`, wherever it
5+
lives, is **independently installable**: it carries everything it needs
6+
inline, so a single `<name>/SKILL.md` folder can be pulled out and installed
7+
on its own in Claude Code, Cursor, Codex, Hermes, OpenClaw, or any other
8+
client that speaks the same protocol — you never need this README or a
9+
suite's own README to make one skill work.
10+
11+
Grouping below **is folder structure only** — there is no code-level registry
12+
(no `SUITE_REGISTRY`, no `DomainSuite` class). A "domain suite" is just a
13+
directory of related `<NN-name>/SKILL.md` workflows plus a `README.md` that
14+
explains how they relate.
15+
16+
## Domain suites (multi-skill, folder-grouped)
17+
18+
| Suite | Skills | Dimension | What it covers |
19+
|---|---|---|---|
20+
| [`eval_pipeline/`](eval_pipeline/) | 9 (`00-meta-eval` … `08-bootstrap`) | Horizontal methodology | How to build an evaluation system from scratch — dataset design, metric/grader selection, human-alignment calibration, reporting, plus RAG/prompt-regression/redteam/bootstrap scenarios. Router at `00-meta-eval`. |
21+
| [`academic-eval/`](academic-eval/) | 4 (`00-academic-router`, `01-paper-review`, `02-bib-verify`, `03-ref-hallucination-arena`) | Vertical domain | Academic paper review, BibTeX verification, and citation-hallucination benchmarking. |
22+
| [`arena-eval/`](arena-eval/) | 3 (`00-arena-router`, `01-auto-arena`, `02-ref-hallucination-arena`) | Vertical domain | Arena-style model/agent comparison — generic win-rate ranking, and citation-hallucination-specific ranking. |
23+
| [`openjudge-core/`](openjudge-core/) | 2 (`01-graders-and-pipeline`, `02-rl-reward`) | Horizontal methodology | Direct OpenJudge core-library API: graders/`GradingRunner`/aggregators, and RL reward-signal construction. No router — see the suite's own README for why. |
24+
25+
`eval_pipeline` is the range-of-motion paradigm all the other suites above
26+
were reshaped to match (three-layer structure, directory-as-grouping, router
27+
only where sub-skills are genuinely ambiguous, every `<NN-name>/SKILL.md`
28+
independently installable). See
29+
[`docs/superpowers/specs/2026-07-09-skills-domain-suite-proposal.md`](../docs/superpowers/specs/2026-07-09-skills-domain-suite-proposal.md)
30+
for the full design rationale.
31+
32+
### Functional test coverage
33+
34+
`eval_pipeline`, `academic-eval`, and `arena-eval` each have an actor+judge
35+
functional test harness under their own `tests/` folder — a *runner*
36+
(shared, parametrized: `eval_pipeline/tests/run_eval_pipeline_skill_tests.py
37+
--skill-root <suite> --cases <suite>/tests/<suite>_test_cases.jsonl`), a
38+
suite-specific JSONL fixture, and a testing guide. `openjudge-core` and the
39+
isolated skills don't have this yet (see each suite's `README.md` under
40+
"Validating the skills themselves" for the ones that do).
41+
42+
### `ref-hallucination-arena` — intentional duplication, not a bug
43+
44+
`academic-eval/03-ref-hallucination-arena/SKILL.md` and
45+
`arena-eval/02-ref-hallucination-arena/SKILL.md` start as full copies of the
46+
same content and are **independently maintained** — no shared `references/`
47+
directory, no symlink, no sync script. This is a direct consequence of the
48+
independent-installability constraint: a skill can't rely on content that
49+
lives outside its own folder. The two copies are free to diverge (the
50+
academic copy can lean into "how trustworthy are this paper's citations",
51+
the arena copy into "how does this model compare to others"); duplicated
52+
content is an accepted cost, not a problem to solve.
53+
54+
## Isolated skills (single-skill, no suite directory)
55+
56+
These have no other skill in this repo referencing them, so they stay at
57+
`skills/<name>/SKILL.md` with no suite folder — the simplest possible
58+
grouping (1 skill = 1 domain).
59+
60+
| Skill | What it covers |
61+
|---|---|
62+
| [`claude-authenticity/`](claude-authenticity/) | Detect whether an API endpoint is genuinely backed by Claude (vs. a wrapper/proxy/impersonator); extract injected system prompts. Zero OpenJudge dependency. |
63+
| [`mmx-cli/`](mmx-cli/) | Generate text/image/video/speech/music via the MiniMax AI platform. Third-party CLI wrapper, zero OpenJudge dependency. |
64+
| [`find-skills-combo/`](find-skills-combo/) | Discover and recommend combinations of skills from the **external** open agent-skills ecosystem to cover a multi-part task. Unrelated to this repo's own suite grouping. |
65+
66+
**Future rule**: if an isolated skill above ever becomes something a domain
67+
suite depends on or wraps, move the whole folder into that suite's directory
68+
(same copy-don't-link approach as `ref-hallucination-arena` above, or a
69+
rename+move if a single owner is clear) rather than keeping it independent
70+
and cross-linking from two places.
71+
72+
## Breaking change: path migration (this reorg)
73+
74+
This reorg moved 6 previously-published skills to new install paths. There
75+
are no redirect stubs — installing the old path will 404; use the new path.
76+
77+
| Old path | New path |
78+
|---|---|
79+
| `skills/paper-review/` | `skills/academic-eval/01-paper-review/` |
80+
| `skills/bib-verify/` | `skills/academic-eval/02-bib-verify/` |
81+
| `skills/ref-hallucination-arena/` | `skills/academic-eval/03-ref-hallucination-arena/` **and** `skills/arena-eval/02-ref-hallucination-arena/` (two independent copies — see above) |
82+
| `skills/auto-arena/` | `skills/arena-eval/01-auto-arena/` |
83+
| `skills/openjudge/` | `skills/openjudge-core/01-graders-and-pipeline/` |
84+
| `skills/rl-reward/` | `skills/openjudge-core/02-rl-reward/` |
85+
86+
The migrated skills' frontmatter `name` values also change to match their
87+
new directory names. Update explicit skill invocations to use the final
88+
component of each new path (for example, `paper-review` becomes
89+
`01-paper-review`, and `openjudge` becomes `01-graders-and-pipeline`). The
90+
two reference-arena copies use distinct names: `03-ref-hallucination-arena`
91+
in `academic-eval` and `02-ref-hallucination-arena` in `arena-eval`.
92+
93+
`SkillLoader.load_from_directory("skills")` discovers skills recursively
94+
through suite directories. A directory containing `SKILL.md` is treated as
95+
one package, including its bundled references and examples.
96+
97+
`skills/eval_pipeline/`, `skills/claude-authenticity/`, `skills/mmx-cli/`,
98+
and `skills/find-skills-combo/` are unaffected.
99+
100+
## Adding a new skill
101+
102+
- **Fits an existing suite's methodology or domain?** Add it as
103+
`<suite>/<NN-name>/SKILL.md` with `name: NN-name` matching its directory,
104+
choose a name unique across the repository, update that suite's `README.md`,
105+
and add a router entry only if it creates real selection ambiguity with a
106+
sibling (see each suite's README for its router-inclusion rationale).
107+
- **Genuinely stands alone?** Add it as `skills/<name>/SKILL.md` — no suite
108+
folder needed.
109+
- **Either way**, the skill's `SKILL.md` must be installable on its own: only
110+
`name` + `description` in frontmatter, no links outside its own folder (or
111+
its own suite folder), and any non-`openjudge`/`rl-reward` pip dependency
112+
listed explicitly under Prerequisites.
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
---
2+
name: 00-academic-router
3+
description: >
4+
Use when the user wants help with academic papers or citations but it's unclear
5+
which specific workflow fits — reviewing a paper, checking a BibTeX file for fake
6+
references, or benchmarking multiple LLMs on reference-recommendation accuracy.
7+
Also use when the user mentions paper review, peer review, BibTeX verification,
8+
citation checking, reference hallucination, or academic literature accuracy and
9+
hasn't specified which of those three tasks they mean. This skill is the entry
10+
router for the academic-eval suite: it asks one diagnostic question then routes
11+
to the right sub-skill.
12+
---
13+
14+
# Academic Eval Router
15+
16+
Entry router for the `academic-eval` suite. You diagnose what the user actually
17+
wants and route them to one of three sub-skills. You don't review papers, verify
18+
BibTeX files, or run arena benchmarks yourself — you're the triage desk.
19+
20+
Each sub-skill is self-contained: it carries inline everything it needs, so it can
21+
be installed and used on its own.
22+
23+
## Diagnostic Question
24+
25+
Ask (unless the user's request already makes the answer obvious):
26+
27+
```
28+
To route you correctly, which of these matches what you want?
29+
30+
a) Review a single paper (PDF or LaTeX source) for correctness/quality/novelty
31+
— optionally also check its bibliography
32+
b) Check a standalone .bib file for fabricated or mismatched references
33+
(no paper review needed)
34+
c) Benchmark/compare multiple LLMs on how often they hallucinate references
35+
when asked to recommend citations (arena-style, many queries)
36+
```
37+
38+
**Shortcut rule**: if the user already said "review my paper", "check this PDF",
39+
"verify this .bib file", or "compare models on reference hallucination", skip the
40+
question — the routing is already clear from their phrasing.
41+
42+
## Triage Table
43+
44+
| User says / has | Use workflow | What it does |
45+
|---|---|---|
46+
| "Review this paper" (PDF or `.tar.gz`/`.zip` TeX source) | `01-paper-review` | Multi-stage review: safety, correctness, quality/novelty score, criticality — optionally + BibTeX check |
47+
| "Review this paper AND check its references" | `01-paper-review` | Same pipeline with `--bib` set — one run covers both |
48+
| "Just check this .bib file, no paper" | `02-bib-verify` | Cross-checks every entry against CrossRef/arXiv/DBLP, flags `verified`/`suspect`/`not_found` |
49+
| "Compare N models on how often they cite fake papers" / "benchmark reference hallucination rate" | `03-ref-hallucination-arena` | Runs many recommendation queries per model, verifies every returned reference, ranks models by hallucination rate |
50+
| "Compare models on general quality/response, not specifically citations" | — | Not this suite — see the `arena-eval` suite's `01-auto-arena` instead |
51+
52+
## Key distinctions
53+
54+
- **`01-paper-review` vs `02-bib-verify`**: both use the same underlying
55+
`cookbooks.paper_review` pipeline. Use `01-paper-review` whenever a paper file
56+
exists (even if the *only* thing the user cares about is the bibliography —
57+
`--bib_only` mode is documented there). Use `02-bib-verify` only when there is
58+
**no paper**, just a loose `.bib` file to sanity-check.
59+
- **`01-paper-review`/`02-bib-verify` vs `03-ref-hallucination-arena`**: the first
60+
two evaluate *one document's* existing references after the fact. The third
61+
evaluates *model behavior* — how often a model invents fake citations when
62+
asked to recommend some, across a benchmark of queries and models. If the user
63+
wants a leaderboard/ranking of models, not a report on one document, route to
64+
`03-ref-hallucination-arena`.
65+
66+
## Output
67+
68+
```
69+
Recommended workflow: `[skill-name]`
70+
71+
Why: [one sentence tying the user's request to the triage table row]
72+
```
73+
74+
Recommend exactly one workflow. If the request spans two (e.g., "review this
75+
paper, and separately benchmark 3 models on citation accuracy"), say so
76+
explicitly and give both, in the order the user would naturally do them.

skills/paper-review/SKILL.md renamed to skills/academic-eval/01-paper-review/SKILL.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
name: paper-review
2+
name: 01-paper-review
33
description: >
44
Review academic papers for correctness, quality, and novelty using OpenJudge's
55
multi-stage pipeline. Supports PDF files and LaTeX source packages (.tar.gz/.zip).
File renamed without changes.

skills/bib-verify/SKILL.md renamed to skills/academic-eval/02-bib-verify/SKILL.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
name: bib-verify
2+
name: 02-bib-verify
33
description: >
44
Verify a BibTeX file for hallucinated or fabricated references by cross-checking
55
every entry against CrossRef, arXiv, and DBLP. Reports each reference as
@@ -73,5 +73,5 @@ Each reference entry is assigned one of three statuses:
7373

7474
## Additional resources
7575

76-
- Full pipeline options: [../paper-review/reference.md](../paper-review/reference.md)
77-
- Combined PDF review + BibTeX verification: [../paper-review/SKILL.md](../paper-review/SKILL.md)
76+
- Full pipeline options: [../01-paper-review/reference.md](../01-paper-review/reference.md)
77+
- Combined PDF review + BibTeX verification: [../01-paper-review/SKILL.md](../01-paper-review/SKILL.md)

skills/ref-hallucination-arena/SKILL.md renamed to skills/academic-eval/03-ref-hallucination-arena/SKILL.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
name: ref-hallucination-arena
2+
name: 03-ref-hallucination-arena
33
description: >
44
Benchmark LLM reference recommendation capabilities by verifying every cited
55
paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate,
@@ -254,7 +254,7 @@ evaluation_results/ref_hallucination_arena/
254254
255255
## Additional resources
256256
257-
- Full config examples: [cookbooks/ref_hallucination_arena/examples/](../../cookbooks/ref_hallucination_arena/examples/)
258-
- Documentation: [docs/validating_graders/ref_hallucination_arena.md](../../docs/validating_graders/ref_hallucination_arena.md)
257+
- Full config examples: [cookbooks/ref_hallucination_arena/examples/](../../../cookbooks/ref_hallucination_arena/examples/)
258+
- Documentation: [docs/validating_graders/ref_hallucination_arena.md](../../../docs/validating_graders/ref_hallucination_arena.md)
259259
- Official dataset: [HuggingFace](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena)
260260
- Leaderboard: [openjudge.me/leaderboard](https://openjudge.me/leaderboard)

‎skills/academic-eval/README.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Academic Eval — paper review & citation accuracy
2+
3+
A set of skills for academic-paper workflows built on OpenJudge: reviewing a
4+
paper end-to-end, spot-checking a bibliography for fabricated references, or
5+
benchmarking how often LLMs hallucinate citations at scale.
6+
7+
Each skill is a self-contained workflow in `<NN-name>/SKILL.md`. Start at
8+
`00-academic-router` if you're not sure which one you need.
9+
10+
## The workflows
11+
12+
| # | Skill | Use it when |
13+
|---|---|---|
14+
| 00 | `00-academic-router` | You're not sure whether you want a paper review, a BibTeX check, or an arena-style benchmark. |
15+
| 01 | `01-paper-review` | You have a paper (PDF or LaTeX source) and want a multi-stage review — safety, correctness, quality/novelty, criticality, optionally + BibTeX. |
16+
| 02 | `02-bib-verify` | You have a standalone `.bib` file (no paper) and want to check it for fabricated/mismatched references. |
17+
| 03 | `03-ref-hallucination-arena` | You want to benchmark/compare multiple LLMs on how often they invent fake citations, across many queries. |
18+
19+
## Relationship between the workflows
20+
21+
`01-paper-review` and `02-bib-verify` both run on top of the same
22+
`cookbooks.paper_review` pipeline — `01` is the full document review (with an
23+
optional `--bib` flag to also verify references), `02` is the BibTeX-only mode
24+
for when there's no paper to review, just a bibliography to sanity-check.
25+
26+
`03-ref-hallucination-arena` is a different axis entirely: instead of
27+
evaluating one document's existing references, it evaluates *model behavior* —
28+
how often a model fabricates references when asked to recommend citations,
29+
scored across a benchmark of queries and ranked across models. It shares
30+
`ref-hallucination-arena`'s content with the `arena-eval` suite's
31+
`02-ref-hallucination-arena` — the two copies are independently maintained
32+
(see the root [`skills/README.md`](../README.md) for why).
33+
34+
## Dependencies
35+
36+
| Skill | Cookbook |
37+
|---|---|
38+
| `01-paper-review` | `cookbooks/paper_review/` |
39+
| `02-bib-verify` | `cookbooks/paper_review/` (BibTeX-only mode) |
40+
| `03-ref-hallucination-arena` | `cookbooks/ref_hallucination_arena/` |
41+
42+
```bash
43+
pip install py-openjudge litellm
44+
pip install matplotlib # only needed by 03-ref-hallucination-arena (charts)
45+
```
46+
47+
## Self-contained skills
48+
49+
Each `<NN-name>/SKILL.md` is self-contained per the Anthropic Agent Skill
50+
protocol — it can be installed and used on its own without this README or the
51+
rest of the suite. Cross-references between skills in this suite use relative
52+
links scoped to this directory (e.g. `02-bib-verify` links to
53+
`../01-paper-review/`); there are no links out to other suites.
54+
55+
## Validating the skills themselves
56+
57+
This suite has an actor+judge functional test harness, same pattern as
58+
`eval_pipeline`: an *actor* model follows a skill to answer a realistic user
59+
request, and a separate *judge* model grades the answer against the case's
60+
acceptance criteria. See [`tests/academic_eval_testing_guide.md`](tests/academic_eval_testing_guide.md)
61+
for how to run it — it reuses `eval_pipeline`'s runner (parametrized, not
62+
duplicated) against this suite's own 12 test cases in
63+
[`tests/academic_eval_test_cases.jsonl`](tests/academic_eval_test_cases.jsonl).

0 commit comments

Comments
 (0)