CYBRARY Forge Agent Evaluation
Run a versioned agent against hidden cases and compare it honestly.
What is in it
For agent builder, ai platform owner, agent buyer — decide whether a configuration change made the agent better or just faster.
- Frozen suites with hidden ground truth the agent cannot read during a run
- An independent check of final state, not the agent's own claim of success
- Baseline against candidate, with failures and timeouts included
- Separate scores for quality, safety, oversight, cost and latency
- Learner and agent graded apart on the same incident
What it is not
- A simulation score is not proof of production readiness
- Arbitrary third-party agent code does not execute inside Forge
Forge Agent Evaluation against the field
20 capabilities that matter for this purchase, next to the platforms a buyer shortlists in this lane: Immersive Agentic Harness, HTB AI Range.
- Reliability is measured by repeating the same task against fresh snapshots, not by one demo run.
- Approvals are bound to a specific target, so an agent cannot reuse a permission it was granted for something else.
- Spend, tool calls and traces are recorded durably on the server, not in the browser.
| What you need | Forge Agent EvaluationIncluded in the platform | HTB | Immersive |
|---|---|---|---|
| A distinct human-with-agent practice proposition | Yes See it live | Hack The Box: Documented for A distinct human-with-agent practice proposition | Immersive: Documented for A distinct human-with-agent practice proposition |
| Tool-using AI inside the technical environment | Yes See it live Agents act through a broker on the same estate the learner sees. | Hack The Box: Documented for Tool-using AI inside the technical environment | Immersive: Documented for Tool-using AI inside the technical environment |
| Bring a supported customer agent or configuration | Yes See it live A customer agent connects on a scoped connection: named tools, a risk ceiling, a call budget, tenancy checks and unilateral revocation. Arbitrary uploaded agent code still does not run. | Hack The Box: Documented for Bring a supported customer agent or configuration | Immersive: Documented for Bring a supported customer agent or configuration |
| Model, prompt and tool tuning interface | Yes See it live | Hack The Box: Documented for Model, prompt and tool tuning interface | Immersive: Documented for Model, prompt and tool tuning interface |
| Live SIEM-oriented agent tasks | Yes See it live | Hack The Box: Documented for Live SIEM-oriented agent tasks | |
| Tool-call, input and output execution accounting | Yes See it live Every call, argument, result and denial is recorded on the run ledger. | Hack The Box: Documented for Tool-call, input and output execution accounting | |
| Per-run token and spend visibility | Yes See it live | Hack The Box: Documented for Per-run token and spend visibility | |
| Enforced per-run cost cap | Yes See it live The cap is enforced outside the model, so an agent cannot talk its way past it. | Hack The Box: Documented for Enforced per-run cost cap | |
| Agent configuration export | Yes See it live A portable signed bundle. Import refuses an edited bundle, and credentials and hidden answers are stripped on the way out and refused on the way in. | Hack The Box: Documented for Agent configuration export | |
| Offensive success grounded in real target state | Yes, in part See it live Grounded in modeled state. Genuine exploitation needs the native lane. | Hack The Box: Documented for Offensive success grounded in real target state | |
| Developer AI review and remediation environment | Yes See it live Two benches: a graded review bench with a deliberately fallible scanner, and an AI reviewer that writes four confident patches on the diff — one correct, one plausible but wrong, one privilege-expanding, one off-target — plus a verification it never ran. The learner adjudicates it against the service running on the estate, and learner and agent are scored on separate cards. | Hack The Box: Limited for Developer AI review and remediation environment | Immersive: Documented for Developer AI review and remediation environment |
| Continuously refreshed evaluation targets | Yes See it live Seeded variants change names, chronology and benign context between runs. | Hack The Box: Documented for Continuously refreshed evaluation targets | |
| Comparison against a human baseline | Yes See it live Learner and agent are scored on separate cards for the same incident. | Hack The Box: Documented for Comparison against a human baseline | |
| Adversarial-AI and prompt-injection content | Yes See it live | Hack The Box: Documented for Adversarial-AI and prompt-injection content | Immersive: Documented for Adversarial-AI and prompt-injection content |
| Published repeatability and confidence methodology | Yes See it live Trials per case, fixed world seed, consistency reported apart from pass rate, incomplete runs counted as failures, and a plain statement of what a simulated score does not prove. | ||
| Independent proof of production safetyNobody has this, and a lab score must never be sold as it. | By design, no We report what happened in the simulation and say so plainly. No safety certificate. | ||
| Denied agent actions recorded as graded resultsAdded by Forge. Not established for any vendor in the research. | Yes See it live An agent reaching past its authority is a scored event, not a console error. | ||
| Staged faults and hidden ground truth disclosed only at debriefAdded by Forge. | Yes See it live Nine fault classes planted deterministically from a seed; rejecting a wrong agent answer scores well. | ||
| Transfer check with the agent taken awayAdded by Forge. | Yes See it live Every agent track ends with an unaided repeat of an earlier lab. | ||
| Graded practice at supervising a working agentAdded by Forge. | Yes See it live Three benches on the same rails: take over a queue an AI analyst already triaged, hold an intel agent to a spend cap and its grants, and measure an unreliable triage agent across repeated attempts before changing it. Each runs in practice, guided or assessment mode, where a hint cannot name the answer. | Hack The Box: Limited for Graded practice at supervising a working agent |
The run behind this line: The recommendation that reads well and is wrong
An agent produces a confident, evidence-shaped recommendation. Accepting it costs the run; rejecting it with a reason is the pass.
Opponent: A well-meaning agent reasoning from a staged, incomplete log set. Shaped after Analyst-automation failure modes: confident output from truncated evidence. Technique and zone identifiers are published by MITRE and ISA/IEC; the mapping is ours and is not an accreditation.
How it starts — the message delivered to the mailbox
- From
- Security Operations <soc-digest@northwind.example>
- Subject
- Overnight triage summary — one host recommended for reimage
- The summary is real internal mail — the failure is in the reasoning, not the sender
- Confidence stated without the gap in the evidence being mentioned
- The recommended action destroys the evidence that would disprove it
Graded verdict: legit — T1078. Nothing here sends or resolves anywhere real.
The steps, and the zone each one crosses
| # | Tactic | ATT&CK | On the range | IEC 62443 zone |
|---|---|---|---|---|
| 1 | Credential access | T1078 Valid accounts | A real sign-in anomaly exists, but on a different host. | Identity |
| 2 | Defense evasion | T1070.001 Clear Windows event logs | A staged log gap makes the innocent host look guilty. | Enterprise |
| 3 | Discovery | T1087 Account discovery | The enumeration the agent missed happened on the quiet host. | Identity |
Zones in this run: Enterprise (Purdue level 4/5, via Internet mail and web egress); Identity (Purdue level 4, via LDAP and Kerberos to the domain controllers). How the zone model is used.
The alerts the SIEM raises from this run
Event log cleared during the evidence window
Security 1102 or System 104 inside the period the agent reasoned over.
endpoint agent · stage 2 · LINE-AGENTEVAL-01 · T1070.001
Directory enumeration from an unflagged host
Whole-subtree user search from a machine with no alert on it.
directory audit · stage 3 · LINE-AGENTEVAL-02 · T1087
What the case has to hold before it closes
- The agent recommendation, accepted or rejected, with the reason
- The gap in the agent's evidence, stated plainly
- The host that was actually involved
- The action taken instead, and what it preserved
Start learning this line
A plausible agent recommendation that is wrong — rejecting it is the pass.
Start the lab
A tick means the other platform's own documentation shows it. A question mark means we could not confirm it publicly — it is never a claim that they lack it. Every Forge yes opens the working screen rather than a screenshot.