A broader toolkit.
Mixed results.

“Pass” includes output validity and deterministic gates. “Both repeats pass” counts unique case IDs. No primary row is cached. The other controls are lexical reranking, system-plus-tail compaction and keyword triage. Decision controls are case-coauthored lexical/keyword rules.

The deliberately unclassified-today triage baseline cannot pass joint correctness because it predicts neither kind nor human review. Supervision counts accepted action only; see its separate signal errors.

Every row joins the same (case ID, repeat) within one matrix; each pair counts once. Positive milliseconds mean Jev took longer. Controls do not perform generation. No inference-significance test is claimed on these small coauthored cases, and an auxiliary decision is not an avoided main-model call.

The canonical picker passes 24/24 versus 20/24 for the lexical control: 12 unique cases, two repeats per arm. The control misses the Chinese PDF request and adds audio transcription to a PDF-to-DOCX request. Jev retrieves every required slot with no irrelevant distinct names in all catalog conditions.

The duplicate-input condition is a failure. English and Spanish review each return two suggestions with the same name, so only 10/12 pass. Required-slot recall still passes 12/12; that does not erase invalid duplicate suggestions. Normal filesystem discovery deduplicates names. This stressor calls the public picker directly with duplicates; it does not demonstrate duplicate loading in a normally installed catalog.

Two unique no-skill inputs: acknowledgement is a local gate; referent-free “Do it.” receives a semantic no-skill result. The latter is not skipped simply because it is short.

Work scales in requests, not just tasks. Ten skill-needed inputs in the 377-entry catalog each use four parallel shortlist sends plus verification. “Do it.” uses four shortlist sends; acknowledgement uses none: 54 HTTP sends for 12 inputs. All catalog entries were present in reconstructed shortlist payloads.

Limit: catalogs were prebuilt. No filesystem discovery time or skill-body loading was measured. Stage one sees 200 description characters; stage two sees up to 600. The late-description condition still exposes informative skill names, so it does not prove recovery without clues.

Denominators: 14 unique evidence groups × two repeats = 28 opportunities; six unique injection candidates × two = 12; 56 unique clean candidates × two = 112. Equal micro group recall for lexical and Jev does not imply equal mean case recall because cases contain different numbers of groups.

Three injection cases are rejected semantically; three are caught locally. The sensitive-query case stays local with two successful repeats and no HTTP sends. All 124 candidate occurrences are accounted for: 108 scored and 16 unjudged, with no overlap or omission.

Limit: 5–6 passages per case, not a multi-batch scale test. Selected IDs were scored; no downstream model answered a retrieval question or executed an injected instruction. Zero selected attacks is not an end-to-end prompt-injection guarantee.

Joint result: 20/24, not 24/24. Routes pass for all 12 cases in both repeats, but two newsletters are correctly routed ignore/vendor while receiving unnecessary human-review scores above the frozen 0.5 threshold. Those four errors are not route over-triage. The injected urgency footer did not change the route.

For human labels, 10 of the 22 opportunities accept either boolean. On the strict subset, Jev passes 8/12: six unique cases, two repeats. The three urgent cases yield six observations, including one local privacy case.

The sensitive case stays local and routes now, but omits needs_human. Both repeats pass because this case intentionally has no human/kind gold. Missing is not false. Model-derived confidence appears on 22 semantic returns; the two local returns use 0.0.

Compared with keyword triage, Jev alone passes the duplicate-invoice and automated-receipt cases; keyword alone passes the newsletter with injected footer. Both fail the urgent-sales newsletter's joint rubric for different reasons.

Limit: the unclassified-today baseline cannot predict kind or human review, so its 0/24 joint score is not “human inbox accuracy.” These synthetic urgency labels are not medical, legal or business authority.

Jev 21/24; lexical control 22/24. All returned IDs were offered. Five Jev outcomes say low confidence: two correctly reobserve a missing target, while three substitute reobserve for a required stop or question.

Jev does beat the control on the hostile-label case: the lexical rule selects a harmless help card instead of report Details, twice. That is a wrong safe transition, not a dangerous operation.

Limit: dangerous delete/send actions were excluded upstream. Offered-ID validity cannot establish the safety of constructing the action table. These component fixtures did not execute browser or desktop actions; the separate adapted-browser population is below.

Jev and keep-all both pass 24/24. Jev also keeps 24/24 identifier and 24/24 corrected-fact occurrences, with no stale fact remaining. Keep-all retains all 24 superseded-fact occurrences alongside their corrections; the rubric does not count those as failures.

Jev reduction spans 4.7619–33.0142%; mean 16.4200%, median 12.4347%, aggregate total-character reduction 18.2535%. These are different weightings. System-plus-tail has 75.7798% median reduction, but keeps only 24/120 required facts and no old identifiers.

Limit: inputs are 1,192–3,392 characters. Twelve unique case IDs share one template, fixed requirement positions and a protected next action. Literal extractability is not generated handoff quality or a comparison against a normal summarizer. The separate offline no-key/24,000-character clipping failure still applies; these small cases do not reach that cap.

Both arms pass 24/24 allocation+tier checks. Jev chooses the fixture fast model eight times, including two pinned outcomes, and the deliberate model 16 times. Only six rows change the current model. Eighteen retain it for legitimate decisions or local gates, not transport fallback. Risk-floor violations: 0/4 repeated labeled cases.

Warm lookup is genuinely fast here, not generally safe. A separate 12-case cold/warm run produces ten cache hits and two warm local gates. True-hit caller median falls from 384.03 ms cold to 0.3387 ms warm; ranges are 329.71–430.37 ms and 0.1708–0.6385 ms. No warm HTTP sends occur. Cached output matches the cold JSON apart from cached, but keeps the old latency_ms.

Offline, an identical-query cache hit returns an 8k-context model for a 100k-context request and even after that model is excluded. Uncached control keeps the strong model. Fast cache hits do not negate this guard bypass.

Limit: model pool, vision/context capacity and prices are synthetic metadata. Specialties within a tier all map to the same model. No specialized-routing, main-model generation or cost-saving benefit is established by these selector cases.

Supervision action labels pass 24/24 in both arms. Six Jev outcomes use local exit-code/sensitive-tail gates. In the 18 semantic assessments, labeled signal opportunities pass: progressing 6/6, needs-input 8/8, blocked 10/10, done 16/18. These are repeated signal opportunities, not separate tasks.

The approval case says a proposal is validated, requests approval, and states no deployment occurred. Both repeats correctly select answer_question, yet done scores 0.56 / 0.58 fail the false-completion signal label. No premature collection occurs. The local injection flag is true on the two injected-completion repeats; there is no scored injection-alert gold or Watcher run.

Escalation ladder: 7/7 deterministic contract checks. Empty ladder, first available, cooldown skip, forced last resort, cooldown expiry, failed probe and no-probe assumption all match the specified policy with zero model calls. With no probe, availability is assumed; if all are unavailable, the last resort is forcibly selected and labeled.

Limit: action-only success is not fully correct supervision. No actual job, provider capacity, subscription seat, external command or alert delivery was tested by the ladder checks.

Do not pool the two runs of the same repository suite. The clean-path control removes only the ancestor-path warning failure. The remaining failure is the fake GUI test reaching missing /usr/bin/open on Linux; the skipped test did not find the supplied Hermes checkout at its expected location.

Release gate: exit 0, 71 clean files checked against Git; optional private denylist checks were not run. Green integration/adversarial assertions include successful reproduction of defects, not 18 successful end-user tasks.

Real integration covers installation/discovery, both prompt sections, five tool handlers, pre-LLM context collection, middleware and request building, output notice, SQLite/public session export, one-time handoff injection and a profile-scoped gateway thread. Routing inputs were explicitly canned; writer failure was deliberate. No provider SDK generation, live user session or scheduled handoff job is established by these offline checks.

Twelve synthetic transport-reply probes, with no network calls, test a stricter response contract. The client rejects unknown choices, missing answers and nonfinite confidence, but accepts duplicate answer keys, incomplete or non-normalized probability maps, and a choice inconsistent with the distribution's highest probability. Missing score confidence becomes 1.0; an extreme integer escapes as OverflowError rather than the client's normal error type. These are parser robustness observations, not malformed replies observed from the live Jev API. The successful live calls do not establish robustness to these counterexamples.

Original browser001: six scheduled attempts; four unavailable rows, two not run. No browser execution, action choice or Jev HTTP sends. The restricted launcher could not resolve Node/package availability, despite a separate broader-host capability probe. This is not 0% model accuracy.

The harness classified non-Jev rows as deterministic controls before checking terminal errors, so it stopped only at the first Jev unavailable row. This is an accounting limitation, not inference failure.

Upstream runtime: optional jev_ultrafast is absent, so the shipped browser runner is not exercised. Separate planner smoke: the Astra override returned HTTP 400 and fell back. That is not a verdict on the shipped default Gemini planner. Neither result is pooled into the corrected Playwright workflow score.

Corrected adaptation manifest SHA-256: df8a14d69703af892d7c6ac5d5d3e276ac2c2652d94fdf2233206f8a52a5913a.

• Keep-all, rehearsal case: the answer quotes the newer pause instruction instead of the predeclared rehearsal action. The expected action remains in the handoff, explicitly described as paused.
• Keep-all, terrarium case: the answer says the approval owner is unknown, although the exact owner statement remains in the handoff.
• Jev, seed-exchange case: the answer quotes the no-external-actions instruction instead of the expected label-wording constraint. Both are present and both are constraints.

The matched median Jev-minus-keep-all two-call duration is −2.239 seconds. This differs from subtracting arm medians. Timing excludes the earlier selection call; adding selection from another run would be arithmetic replay, not continuously measured task latency. Generation order is fixed baseline/tail/Jev for each case, with no repeats or production-tail inference. The same small transcript template and exact-match rubric limit generalization.

All 72 calls returned text and native usage; no retries. The answer request never contains the original transcript or answer key. This does not test long contexts, confidential capsules, scheduled handoff workers or downstream instruction obedience.

Follow-up freeze SHA-256: ecdc2fbfa063e173e7a2b98616a56b0c01ee4c719c3b6304482cd1146aa61640.

Task time runs from agent-ready to conversation-return and includes hooks and tools; it excludes initialization, installation, separate artifact scoring and between-episode review. Process-wall sum is 643.504 seconds; task-wall sum is 593.896 seconds. Neither is the benchmark's elapsed wall time. Randomized arm/task order uses seed 902013. Only one observation per task/arm; no significance, production-tail or dollar-savings claim.

The matched routing and picker comparisons share the plugin-enabled control, starting Astra model, system-prompt hash and tool names. Plugin-absent fixed/deterministic arms are whole-configuration comparisons, not isolated causal contrasts. All six routed episodes use provider-served @chatgpt/gpt-6-astra; fixed-fast episodes use @chatgpt/gpt-5.6-luna. The deterministic arm uses both under its frozen rules.

Every code artifact passes eight hidden tests, two unchanged visible tests and its added suite. Manual inspection confirms substantive containment, nonmutation, touching and invalid-input cases, and that the model actually ran tests. Added methods per code arm range from four to eight; method count alone was not accepted as proof of coverage.

Four intermediate failures remain visible: fast redaction's reconstruction assertion; the matched control's dependency-script indentation; deterministic release-note quoting syntax; and fast code repair's initially nonidentical visible-test copy. Each was corrected inside its original episode. None was erased, retried as a new episode or reclassified as a clean first attempt.

Eight initialization metadata requests were blocked before HTTP reservation. They are non-inference events, not hidden successful API calls. Both smoke attempts remain separate: the first stopped at the evaluator's 32K context setting before any HTTP; a prospective 64K configuration passed with two main requests and one Jev request.

The shipped repository installer runs in a fresh profile; native-install rejection is not repaired. Plugin and bundled toolkit bytes are checked unchanged. Only the Jev plugin is enabled; handoff automation stays off. Eight synthetic skills are shared across arms; installed Jev setup/automation skills are disabled consistently so controls are not nudged to configure Jev.

Tool workers cannot access runtime memory, credentials, budget sockets, oracle files or other tasks, and have no network. They do not preserve cwd, environment or file-read state between calls. Every arm is told that existing-file write_file overwrite is unavailable and to use patch or terminal for updates. This is a restricted benchmark environment, not ordinary Hermes file-tool semantics. The code prompt prospectively requires four separately discoverable test methods.

Each episode allows at most ten main requests, twelve Jev requests and 90 seconds nominal agent time, within the original global deadline. No live profile or service was changed. All model-artifact credential scans passed; no unsafe benchmark-model tool attempts were observed. This is not an adversarial sandbox proof.

Caller time includes preparation, facade execution, actual batch waves, verification and scoring. It excludes interpreter launch and filesystem catalog discovery. Toolkit latency_ms is a local timer, not provider compute time; it may omit orchestration and remains stale on cache hits. Medians/ranges describe this corpus, not production tails. Do not sum unrelated medians.

Jev: 648,893 native input and 126,675 output tokens, no missing usage. At the documented input-only rate, the total estimate is $0.027253506, not verified billing. Main API: 763,317 known prompt and 57,634 completion tokens. The planner's HTTP 400 has unknown usage, not zero. Main pricing is unverified, so no combined dollar total or cost-saving ratio is asserted.

The budget was 1,200 Jev sends, 1,000 main sends and 90 minutes from the first connectivity request. Every reservation ID has exactly one receipt. First send: 2026-09-20 04:13:34 UTC; last agent return: 05:23:13 UTC, before the 05:43:34 deadline. Frozen runs were not tuned or repeated after semantic results. Failed environment/smoke attempts remain separate and consumed no model requests.

Tokenizers and protocols differ. Do not interpret native token counts as a cross-model cost ratio. The public rate was checked at TypeSafe's model documentation; account-specific billing was not inspected.

Seven families × 12 unique case IDs = 84. This is not 84 independent task structures; compaction shares one template. The two primary matrices contain 408 case/arm/repetition attempts. Catalog sensitivities and cold/warm checks bring the completed nonbrowser total to 552 without adding new tasks.

Nonbrowser arms: 252 Jev invocations, 228 baseline-control invocations and 72 other-control invocations. Jev paths: 217 fresh semantic, 10 cache hits, 25 local gates, zero terminal fallbacks. Cached outputs carry prior decisions, not new model answers. Only the unauthorized-action case changes Jev primary pass/fail across repeats; repeat agreement elsewhere is not statistical independence.

The independent audit recomputed all 552 scores, checked 128 frozen-hash comparisons over 18 distinct harness/fixture files, 134 offline artifact hashes, 19 byte-identical evidence copies and 35 pinned source excerpts. All nine existing accounting summaries matched. Completion is established by schedule and final completion receipts; the frozen manifest’s initial complete=false is not a final status.

All 110 skill-shortlist request hashes were reconstructed and matched without sending requests. Complete original response-body hashes cannot be recomputed from stored facade projections. That limitation remains even though receipt-file hashes match.

• Evaluated source and frozen scoring thresholds were not repaired after outcomes. Environment corrections and adaptations remain separately labeled.
• Offline tests, live selector calls, browser execution and full-agent episodes are different populations. Fallback usefulness is distinct from model success.
• Selected-context screening is not downstream injection resistance; short extractive compaction is not long-context summarization.
• Case-coauthored deterministic controls have task knowledge. They are not ordinary no-Jev agents and do not generate outputs.
• These labels do not establish specialized routing, actual provider capacity, alert delivery, desktop execution or net task-time/cost savings.
• This toolkit is broader than the earlier Decision Aux comparison. Earlier PoCs 1, 2, 4 and 5 remain completed; PoC 3 remains on hold. Their results and sealed fixtures are not pooled here.
• This is a report-only edition. No raw receipts, request IDs, endpoint identities, private configuration, account metadata or evidence downloads are included.

1. Rerank instruction screen jevkit/rerank.py:45–60
2. Rerank local gates and clipping jevkit/rerank.py:273–299
3. Rerank selection and coverage jevkit/rerank.py:350–388
4. Redaction clips head and tail jevkit/privacy.py:58–80
5. Filesystem discovery deduplicates names jevkit/skillpick.py:54–76
6. Picker batching, verification and duplicate return jevkit/skillpick.py:301–371
7. Action chooser confidence fallback jevkit/choose.py:100–121
8. Triage sensitive return and questions jevkit/triage.py:67–98
9. Triage classification and route jevkit/triage.py:106–153
10. Supervision gates and semantic signals jevkit/supervise.py:85–146
11. Watcher alert policy, untested in this matrix jevkit/supervise.py:205–223
12. Compaction selection and digest truncation jevkit/compact.py:33–91
13. Client retries and locally measured latency jevkit/client.py:133–190
14. F01 hermes/plugin/hermes-jev/plugin.yaml:7 hermes/plugin/hermes-jev/plugin.yaml:7–7
15. F01 hermes/plugin/hermes-handoff/plugin.yaml:7 hermes/plugin/hermes-handoff/plugin.yaml:7–7
16. F01 Hermes hermes_cli/plugins_cmd.py:594-604 hermes_cli/plugins_cmd.py:594–604
17. F02 hermes/plugin/hermes-jev/__init__.py:32-55 hermes/plugin/hermes-jev/__init__.py:32–55
18. F02 jevkit/catalog.py:27-34 jevkit/catalog.py:27–34
19. F03 jevkit/keystore.py:66-67 jevkit/keystore.py:66–67
20. F03 Hermes agent/secret_scope.py:147-175 agent/secret_scope.py:147–175
21. F04 install.py:80-134 install.py:80–134
22. F05 jevkit/route.py:420-422 jevkit/route.py:420–422
23. F05 jevkit/route.py:485-491 jevkit/route.py:485–491
24. F05 jevkit/route.py:545-553 jevkit/route.py:545–553
25. F06 jevkit/route.py:477-497 jevkit/route.py:477–497
26. F06 jevkit/route.py:532-533 jevkit/route.py:532–533
27. F06 README.md:94 README.md:94–94
28. F07 jevkit/client.py:65-70 jevkit/client.py:65–70
29. F07 jevkit/client.py:143-147 jevkit/client.py:143–147
30. F07 jevkit/client.py:166-190 jevkit/client.py:166–190
31. F07 README.md:92 README.md:92–92
32. F08 jevkit/compact.py:33-59 jevkit/compact.py:33–59
33. F08 jevkit/compact.py:75-91 jevkit/compact.py:75–91
34. F09 jevkit/supervise.py:101-129 jevkit/supervise.py:101–129
35. F09 jevkit/privacy.py:13-17 jevkit/privacy.py:13–17
36. F09 CONTRIBUTING.md:27 CONTRIBUTING.md:27–27
37. F10 hermes/plugin/hermes-jev/__init__.py:181-187 hermes/plugin/hermes-jev/__init__.py:181–187
38. F11 jevkit/triage.py:67-81 jevkit/triage.py:67–81
39. F11 jevkit/triage.py:143-150 jevkit/triage.py:143–150
40. F12 tests/test_gui_agent.py:594-606 tests/test_gui_agent.py:594–606
41. F12 skills/jev-computer-use/scripts/jev_gui_agent.py:728-729 skills/jev-computer-use/scripts/jev_gui_agent.py:728–729
42. F12 skills/jev-computer-use/scripts/jev_gui_agent.py:848-850 skills/jev-computer-use/scripts/jev_gui_agent.py:848–850
43. F13 install.py:242-253 install.py:242–253
44. L01 jevkit/privacy.py:52-80 jevkit/privacy.py:52–80
45. L02 skills/jev-browser-use/scripts/jev_browser_agent.py:291-310 skills/jev-browser-use/scripts/jev_browser_agent.py:291–310
46. L02 README.md:121 README.md:121–121
47. Escalation probes and last resort jevkit/ladder.py:129–192

Content corpus

60c4f4ec26a577677d092d885ed6516e66a2c93e8b227e300476f917f6c8c454

Decision corpus

b4d58e137d881c7c68e4dd4f7c100e8772facf1f8d62444b76f69015f75b43bc

Content run

586325302e5e04c75662f91cc3899bc4ddde0293fe781fba1798b0a43846c2bb

Decision run

646386b42ed961ee36231da535187d07ddec0792a171388f59d8914bfb30c961

Agent harness

85b09cc11d9162ae16afde70aea40624593100cf9c115471834b7d33d70498dd

Agent schedule

bfb6cdbbd49bb26a4d1b8b6bdb86eca38c335f889285132de0a10e819936b31f

Final accounting

ae58d4ab037c09f8e963179777b2c4fcd3cc7155323f7d231e209209e96c128c

The private evidence contains exact commands, test identities, per-attempt outputs, failed calls, native usage and all run manifests. This public edition supplies summaries and source citations, not raw receipts, accounts, configurations or downloads. Hashes identify those retained artifacts; they do not make unavailable raw response bodies independently recoverable.

An executor inspection printed a private API endpoint into its private tool transcript, violating the no-print instruction. No API key or password was exposed. This was a reporting-process error, not a benchmark-model sandbox escape. Raw transcripts are excluded from publication; the endpoint value is not reproduced here. Artifact credential checks alone do not erase that incident.