Skip to content

Commit 7405847

Browse files
aliaihubaliaihub
authored
Merge pull request #19 from xizhuomengcontin/orcareplay-record-replay
Add use case: deterministic record and replay of Hermes runs
2 parents a8553e1 + a07c284 commit 7405847

3 files changed

Lines changed: 182 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,7 @@ Then pick a use case below. The [`/demos`](demos/) folder has three runnable sta
130130
| Name | Description |
131131
| --- | --- |
132132
| [Hermes Labyrinth Observability](usecases/hermes-labyrinth-observability.md) | A read-only dashboard plugin that turns prompts, tool calls, failures, approvals, model switches, cron runs, and memory hits into auditable agent journeys. |
133+
| [Deterministic Record and Replay of Hermes Runs](usecases/deterministic-record-and-replay.md) | Records a run at the process and socket boundary, replays it offline byte for byte with the network off, and forks it from any checkpoint onto a different model. Needs no plugin and no change to `config.yaml`. |
133134

134135
## Security & Assurance
135136

‎usecases/deterministic-record-and-replay.md‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# Deterministic Record and Replay of Hermes Runs
2+
3+
**Class:** Ecosystem integration · **Confidence:** High · **Demo status:** Runnable
4+
5+
## Pain Point
6+
7+
A Hermes run that goes wrong is hard to examine twice. Re-running the same
8+
prompt gives a different answer, because the model is sampled and the system
9+
prompt carries the date and session; and the transcript on screen is not the
10+
request that was sent — the 7,742-character system prompt, the 19 tool
11+
schemas, and the exact SSE the provider streamed back are all invisible from
12+
the terminal. So "why did it do that?" gets answered from memory rather than
13+
from evidence, and a fix cannot be checked against the run that motivated it.
14+
15+
Worse for autonomous work: to ask "would a different model have made the same
16+
call here?" you have to re-run the whole job on the other model and hope the
17+
divergence reproduces.
18+
19+
## What It Does
20+
21+
OrcaReplay records a Hermes run at the process and socket boundary, then
22+
re-executes it with the network switched off, serving the recorded exchanges
23+
back byte for byte. From any checkpoint the run can be forked onto a different
24+
model to see where behaviour diverges.
25+
26+
Capture sits **outside** Hermes. Nothing is installed into `~/.hermes`, no
27+
plugin is loaded, and `config.yaml` is not touched — which matters here,
28+
because Hermes has no base-URL variable to move. `OPENAI_BASE_URL` appears in
29+
one place in the shipped source, an advisory warning for when it is set and
30+
the provider is not `custom`; the origin itself lives in `config.yaml` under
31+
`model.base_url`. So capture terminates the TLS Hermes established itself,
32+
with a certificate authority minted for the single run and deleted when it
33+
ends.
34+
35+
What one turn produced, from the verified run below:
36+
37+
- The full request: 42,305 bytes, 19 tools, a 7,742-character system prompt.
38+
- The response: status 200, streamed, 10,615 input and 88 output tokens.
39+
- Every shell command Hermes ran, with exit codes, and a per-turn diff of the
40+
files it touched, on one timeline with the model calls.
41+
- `orca replay` reproducing all of it offline.
42+
43+
One Hermes detail worth knowing, because it looks like a problem and is not.
44+
Hermes makes two different kinds of HTTP call with two different libraries:
45+
the model catalogue through `requests`, and the completion through httpx under
46+
the OpenAI SDK. httpx pins certifi, so the usual `SSL_CERT_FILE` looks as
47+
though it cannot reach the call that matters. It does — Hermes resolves that
48+
client's CA itself, and the chain in `agent/ssl_verify.py` reads
49+
`HERMES_CA_BUNDLE`, then `SSL_CERT_FILE`, then `REQUESTS_CA_BUNDLE`, then
50+
`CURL_CA_BUNDLE`. So a recorder that sets the standard variables is trusted by
51+
both clients with nothing Hermes-specific added.
52+
53+
## Setup
54+
55+
Node 20 or newer. The Hermes adapter is on the `hermes-adapter` branch,
56+
pending review in the PR linked below; until it lands in a release, build from
57+
the checkout:
58+
59+
```bash
60+
git clone https://github.com/Continuum-AI-Corp/OrcaReplay.git
61+
cd OrcaReplay
62+
git checkout hermes-adapter # or main, once PR #33 has landed
63+
npm ci && npm run build
64+
```
65+
66+
Record a run. `--tls-intercept` is required — there is no origin to redirect —
67+
and `opencode.ai` is named explicitly because it is a website as much as a
68+
gateway, so it is not in the default allowlist:
69+
70+
```bash
71+
node packages/cli/dist/cli.js record hermes \
72+
--tls-intercept --tls-hosts '+opencode.ai' -- \
73+
--provider opencode-free -m nemotron-3.5-lightning-free \
74+
-z 'Compute 212000 plus 8 and reply with only the number.'
75+
```
76+
77+
For Hermes on Nous' own models, drop `--tls-hosts`:
78+
`inference-api.nousresearch.com` is in the default list.
79+
`portal.nousresearch.com` deliberately is not — that is where the API key is
80+
issued, and it stays tunnelled and unread.
81+
82+
Then, with no network:
83+
84+
```bash
85+
node packages/cli/dist/cli.js replay <run-id> # reproduce it exactly
86+
node packages/cli/dist/cli.js show <run-id> # the timeline
87+
node packages/cli/dist/cli.js replay <run-id> --from 3 --model claude-sonnet-5
88+
```
89+
90+
`-z` is the non-interactive flag. Without it Hermes opens a session and waits,
91+
and the recording ends when you do.
92+
93+
## Prompts
94+
95+
The prompt used for the verified run — arithmetic on purpose, so the answer is
96+
checkable and the run is short:
97+
98+
```text
99+
Compute 212000 plus 8 and reply with only the number.
100+
```
101+
102+
Hermes answered `212008`. The point of the example is not the arithmetic: it
103+
is that the recording holds the 7,742-character system prompt and 19 tool
104+
schemas that the terminal never showed, and that `orca replay` returns the
105+
same bytes with the network off.
106+
107+
A second run, to see the same question answered by the recording rather than
108+
the provider:
109+
110+
```bash
111+
node packages/cli/dist/cli.js replay <run-id>
112+
```
113+
114+
reported `reused=3/3`, exit 0. It also reported three `major` divergences,
115+
which is the honest result and worth stating: Hermes' system prompt carries
116+
the date and session identity, so the replayed request has an identical
117+
trailing message and a different prefix. Divergence reporting is doing its
118+
job — that is what a dynamic system prompt looks like on the wire.
119+
120+
## Skills Needed
121+
122+
None of Hermes'. No plugin, no skill, no gateway, and no change to
123+
`config.yaml` — the recorder launches `hermes` as a child process and reads
124+
the socket.
125+
126+
- Node 20+ for the recorder.
127+
- The provider used above, `opencode-free`, is served anonymously, so the run
128+
needs no API key and no credential enters the recording. Any configured
129+
Hermes provider works; a credential is forwarded upstream and never written
130+
to the trace.
131+
- Verified on Windows against **Hermes Agent v0.21.0 (2026.8.31)**, installed
132+
from git.
133+
134+
## Sources
135+
136+
- [Hermes configuration reference](https://hermes-agent.nousresearch.com/docs/user-guide/configuration)
137+
— the official docs for the behaviour this whole use case turns on: "When
138+
`base_url` is set, Hermes ignores the provider and calls that endpoint
139+
directly... When only `provider` is set, Hermes uses that provider's built-in
140+
auth and base URL." The origin is a config key, not an environment variable,
141+
which is why capture happens at the transport instead of by redirection.
142+
- [OrcaReplay PR #33](https://github.com/Continuum-AI-Corp/OrcaReplay/pull/33)
143+
— the adapter, with the verification transcript: the recorded sizes, token
144+
counts, and the `orca replay` result quoted above. Also documents the
145+
`OPENAI_BASE_URL` finding, established against a listener with
146+
`model.base_url` pointed at a dead port.
147+
- [`packages/adapters/src/hermes.ts`](https://github.com/Continuum-AI-Corp/OrcaReplay/blob/54839339ff7ca00c844f47bb043d0c6a85adc9ad/packages/adapters/src/hermes.ts)
148+
and its recorded contract
149+
[`fixtures/harness/hermes.json`](https://github.com/Continuum-AI-Corp/OrcaReplay/blob/54839339ff7ca00c844f47bb043d0c6a85adc9ad/packages/adapters/fixtures/harness/hermes.json),
150+
which carries the date the adapter was last confirmed against a running
151+
Hermes.
152+
- [`prompt/HERMES/`](https://github.com/Continuum-AI-Corp/OrcaReplay/tree/54839339ff7ca00c844f47bb043d0c6a85adc9ad/prompt/HERMES)
153+
— the captured system prompt as recorded, scrubbed of local paths.
154+
- `agent/ssl_verify.py` in Hermes itself, for the CA resolution order quoted
155+
above — the reason the standard `SSL_CERT_FILE` is enough and no
156+
Hermes-specific variable is needed.

‎usecases/deterministic-record-and-replay.verification.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
usecase: "deterministic-record-and-replay.md"
3+
verified: true
4+
repos_checked: 1
5+
sources_checked: 1
6+
sources_alive: 1
7+
sources_yellow: 0
8+
sources_red: 0
9+
---
10+
11+
# Verification Report
12+
13+
- **Continuum-AI-Corp/OrcaReplay**: 92 stars (✅ above threshold)
14+
15+
## Summary
16+
- Total repos checked: 1
17+
- Passed threshold (≥10 stars): 1
18+
- Failed threshold (<10 stars): 0
19+
20+
## Confidence Score
21+
22+
- **Score**: 100/100
23+
- **Flag**: green
24+
- **Badge**: ![deterministic--record--and--replay](https://img.shields.io/badge/deterministic--record--and--replay-100%2F100-brightgreen)
25+

0 commit comments

Comments
 (0)