BEARHUDDLESTON.DEV / JEV + HERMES
Jev + Hermes:
integration assessment.
We wouldn't add Jev to Hermes by default.
It made some small decisions faster, but adding it did not demonstrate better completed tasks. Extra calls and integration defects leave no convincing overall gain in these tests.
Jev is a separate model used for small decisions. We tested our own Decision Aux pilots and the third-party Hermes Jev Skills toolkit. This page explains what those results mean for everyday Hermes use.
Two useful results—not a case for the whole layer.
- Jev answered small questions faster than Astra, our main model. Both got the same answers right. This helps if you already need that extra decision; it doesn't prove you should add one.
- Giving Hermes all the relevant source files helped. That improvement needed no Jev call. The files were small enough to fit together.
Compare the studies: results, times and sample sizes
| Study | Observed evidence | What it means for integration |
|---|---|---|
| PoC 1 · generator routingCompleted · 8 tasks × 2 repeats × 5 arms | Jev kept Astra on all 16 attempts. Jev and Astra-only passed 16/16; Luna-only passed 15/16. Mean times were 18.795 s with Jev versus 19.312 s Astra-only, but medians reversed: 19.871 s versus 19.491 s. | No demonstrated acceleration from selecting the faster generator. The paired interval crossed zero; one raw mean is not a routing win. |
| PoC 2 · source retrievalCompleted · 8 tasks × 2 repeats × 4 arms | All-source bundling: 14.768 s mean versus ordinary Astra's 17.262 s, with 16/16 correct artifacts each. Main requests fell from 110 to 88. Jev bundling averaged 17.103 s and used 107 main plus 16 ranking requests. | The useful improvement was simple exact bundling without Jev. All 20 source files fit the budget; this does not establish a large-repository retrieval strategy. |
| PoC 3 · combined routing + retrievalOn hold · not run | No neither/routing/retrieval/both result on the common holdout. | No combined-effect or synergy claim. Earlier separate studies cannot substitute for this experiment. |
| PoC 4 · Jev versus Astra callsCompleted · 72 unique cases per backend | Both answered 72/72 labels correctly. Jev's median was 375 ms versus 1,350 ms: 72.3% less decision-call time. Each backend made 102 calls including controls. | A real bounded-call latency advantage, not proof of task improvement. This compares two ways to make a decision—not making a decision versus doing without it. |
| PoC 5 · agent skill adviceCompleted · 4 tasks × 2 repeats × 3 arms | Every arm passed 8/8 trials. No-advice / Astra-advice / Jev-advice task medians: 20.57 / 24.77 / 21.17 s. Skill loads: 9 / 11 / 16. | Faster than an Astra advisor did not establish better than no advisor. Extra recommendations and loads did not improve the artifacts. |
| Hermes Jev Skills · full-agent trialsPinned toolkit · 6 tasks × 6 arms × 1 run | All 36 artifacts passed. Jev routing retained Astra in all six routed episodes. Picker on/off each passed 6/6 and used 21 main requests; picker-on added 12 Jev requests. | No observed task-quality gain. The broader toolkit repeats the same concern: more decision machinery without demonstrated net benefit on these tasks. |
PoCs 1, 2, 4 and 5 remain completed; only PoC 3 is on hold. Counts above retain each study's original denominator. Repeated attempts add no new task diversity.
The extra calls didn't earn their place.
In the toolkit's skill-advice test, turning advice on added 12 Jev calls. It saved no main-model calls, and all six tasks passed with or without it. Routing also added decisions without choosing a different model.
Not every run was slower. Some timings favored Jev, but these small tests don't establish a reliable overall speed or cost benefit.
Check the call counts, timing differences and cost limits
The benefit we did observe
When a bounded judgment is already required, Jev can be a faster backend than the tested Astra decision path. PoC 4 is useful evidence for that narrow substitution.
PoC 2 also found a practical benefit from providing all relevant source when it fit. That improvement did not require an auxiliary model.
The benefit we did not establish
We did not establish that adding a new decision step makes the overall agent better, cheaper or reliably faster. Avoided downstream work must outweigh the decision request, prompt overhead, orchestration and failures. A low API price alone does not establish that balance.
The relevant baseline is ordinary Hermes with normal skill access, file reads and batching—not a deliberately constrained agent.
| Contrast | Correctness / requests | Observed task-time difference | Interpretation |
|---|---|---|---|
| Routing on versus off | 6/6 each. On: 19 main + 6 Jev. Off: 21 main + 0 Jev. | Median paired difference: −1.211 s. | No model switch occurred. This small, one-run difference is not evidence that routing accelerated work. |
| Picker on versus off | 6/6 each; 21 main each. On adds 12 Jev requests. | Median paired difference: +0.900 s. | Added calls and modest observed delay, without a correctness gain. Picker-off still permits ordinary skill loading. |
Not uniformly slower. Some observed timings favor Jev, including the routing contrast above and the separate handoff-generation follow-up. Small samples, provider/cache variation and different timing boundaries prevent a blanket slowdown or causal speedup claim. The adoption problem is the lack of a demonstrated net advantage, not a claim that every added-call run loses.
Toolkit benchmark accounting: 356 Jev and 200 main-model sends including diagnostics and follow-up. Jev's documented-rate estimate was $0.027253506, not verified billing. Main API pricing is unknown. The more consequential unpriced burdens are an external dependency, additional failure paths, privacy boundaries and maintenance; those burdens were not measured in dollars.
Some safeguards lost useful information.
Filtering stored context rejected attack examples but also removed evidence the task needed. In a small browser test using our own executor, Jev failed to finish one of three workflows. We also found installation defects and cases where settings or routing decisions were reused in the wrong context.
Other features did useful work. The question is whether those benefits outweigh the mistakes and maintenance—not whether Jev can ever help.
Inspect the feature-by-feature benefits and failures
| Feature | Useful behavior | Counterweight |
|---|---|---|
| Skill selection | Canonical rubric: 24/24 versus lexical 20/24; 12 cases, two repeats. The 377-entry condition also found required skills. | No full-agent correctness improvement. The direct duplicate-name input stressor failed 2/12; normal filesystem discovery deduplicates, so this is not a demonstrated installed-catalog failure. |
| Memory screening | No attack candidate selected across six unique attack examples, repeated twice. Both local and semantic screening contributed. | The local regex removed legitimate quoted-security evidence; clipping hid another required identifier. Lexical and Jev retrieved the same 24/28 evidence-group occurrences; mean per-case recall was lower with Jev. |
| Inbox triage | All 24 route observations accepted. | Joint rubric 20/24: human-review judgments failed on correctly ignored newsletters. A sensitive local return omitted the explicit human-review field. |
| Compaction + handoffs | Jev reduced selected text and preserved the short corpus's facts. Generated Jev and keep-all handoffs each retained 60/60 required evidence items. | Later answer coverage was 59/60 versus 58/60, with broad-question ambiguity—not a semantic quality win. Separate offline clipping counterexamples still lose facts. |
| Actions + browser | Bounded offered IDs and stale-state refusal prevented invalid execution in the local fixture. | Action labels: Jev 21/24 versus lexical 22/24. Adapted browser completion: Jev 2/3 versus fixture-scripted 3/3. This was not the shipped optional browser runner. |
| Supervision | Action labels passed 24/24, matching the keyword control. | Two completion signals were wrong despite the correct action. Alert delivery and real provider-capacity escalation were not tested. |
Installation and reliability
The pinned toolkit's native manifests were rejected by the pinned Hermes installer, although its own repository installer worked. Same-process profile switching exposed settings/catalog and credential-scope problems. Cached routing decisions bypassed changed-context and pool checks; risk screening could lose a clipped phrase. Privacy checks did not cover every outbound field, and nominal client timeouts did not enforce a total slow-read deadline.
These are observed implementation counterexamples, not production failure-rate estimates. A passing small task suite does not negate them. Fixing them would address correctness, but would still not prove that the extra layer improves end tasks.
Keep it optional, not part of the default.
- For everyday use: keep ordinary Hermes. Use simpler checks or provide the relevant source directly when that already solves the problem.
- For a specific use case: consider an opt-in plugin only after fixing the defects and showing a measurable task benefit.
Why we recommend this—and why a plugin, not a core dependency
Keep ordinary Hermes as the default. Do not adopt the tested Jev policies or Hermes Jev Skills wholesale as shipped.
“A wash, possibly worse after overhead” is a fair practical reading of these workloads: no observed end-task quality gain from the extra layer, added requests, and several concrete regressions and integration defects. It is an adoption judgment—not a statistically established negative effect across every workload.
For normal Hermes use
- Keep the auxiliary routing and advisor policies off by default.
- Prefer deterministic checks or exact source bundling where they already solve the problem. The latter helped only where the source fit.
- Do not trade away clean evidence, profile isolation or reliable recovery for a faster decision call.
For a narrowly scoped extension
- A required high-volume bounded classifier could justify an opt-in Jev backend, but this is a hypothesis to validate on that workload.
- Keep it replaceable, with measured no-auxiliary and deterministic controls, explicit privacy policy and visible fallbacks.
- Treat third-party integration as a standalone plugin—not a new Jev-specific core dependency.
Architecture is a separate consideration from measured utility: the pinned Hermes contribution policy places third-party product integrations in standalone plugins. A generic host extension point can be justified by concrete consumers; it does not need a vendor-specific core dependency. Pinned repository policy →
What evidence would change our minds?
- Repair the demonstrated boundaries. Profile-scoped settings and secrets, cache revalidation, full-input safety checks, response validation and reliable timeout/recovery behavior must work. Verify the actual installed path.
- Use a representative frozen workload. Include difficult, ambiguous, multilingual and adversarial cases that distinguish task outcomes. Keep a holdout; do not tune after primary results.
- Compare the complete task. Use matched auxiliary-on/off, deterministic and ordinary-agent controls. Include decision latency, retries, tool work, total tokens and failures. Verify prices before reporting savings.
- Require a material net benefit. Consistently better task quality, or lower total time/cost without worse quality or safety, must outweigh the extra dependency. Set the acceptance threshold before the run; more skill loads or fewer reads are not sufficient.
No new trial or rollout is authorized by this assessment. PoC 3 remains on hold; the completed studies remain completed.
Read the evidence
These were small studies using made-up tasks—not proof of universal harm or equal model capability. The planned combined test (PoC 3) was not run; the other four pilots are complete. This assessment adds no new tests.
Why we do not pool results or declare a universal verdict
The studies use different tasks, catalogs, mechanisms, source revisions, baseline capabilities and clocks. PoCs 1–2 include agent construction; PoC 5 excludes initialization; the pinned toolkit's task clock starts after agent readiness. Backend call latency is not task latency. We compare conclusions and mechanisms, not absolute times across studies.
The synthetic tasks were small. Some controls were authored alongside their cases, and several matrices saturated all arms. Repeats are not independent diversity. The adapted browser is a disclosed local evaluator, not a general desktop benchmark. The initial compatibility fixes, setup failures, fallbacks, partial runs and later corrections remain in the source reports.
Historical PoCs retain their billing/dashboard attribution limits; the later toolkit benchmark's gap-free request receipts do not retroactively resolve them. Raw logs, configurations, request IDs, private endpoints and evidence archives are not public assets. The original permission/publication caveats and contributor attribution are retained in the consolidated report.
Assessment of completed evidence, September 2026. No new inference, integration, live-profile change or service rollout. The recommendation concerns these tested policies and pinned toolkit—not whether Jev can ever be useful.
← Back to directory