CYBRARY R8 — Model Bake-Off and Procurement Decision
Procurement wants a model recommendation and anecdotes will not survive review. Run twenty-five investigation tasks across every reachable runtime under normalized conditions, with benign and malicious sets and repeated trials, and report a result with a confidence interval.
Expert · Assumes real range time. Little guidance, full consequences.
Outcome · Choose runtimes per workload from measured evidence, with variance, cost and compensating controls.
AI connection
This activity runs entirely on the deterministic runtime. Optional live-model runtimes only affect dialog, coaching and wording — never checks or scoring.
Nothing leaves the browser session.
Prompts never leave the learner's machine.
Lab prompts reach the learner's chosen provider only.
Customer tenancy; Cybrary sees metadata only.
Cybrary AI gateway; retention off by default.
Per-job — the router never sends restricted jobs off-tenant.
Lab manifest
The machine-readable release contract. Authors edit this; the console refuses to publish without checks, both regression suites and a reference solve.
schemaVersion: cybrary.ai-lab/v1
id: model-bake-off
release: 1.0.0
mode: agent-evaluation-range
maturity: 5
roles:
- ai-security-lead
- governance-lead
- soc-leadership
outcomes:
niceTasks:
- T0473
- T0484
atlasTechniques:
- AML.T0002
owaspRisks:
- LLM09
- LLM10
aiAssets:
models:
- support-agent-model@2026.06
prompts:
- model-bake-off-system-prompt@1
agents:
- support-agent-orchestrator@3
tools:
- crm.lookup
- ticket.update
- email.send
- refund.issue
corpora:
- support-kb@v4
enterpriseTopology: support-enterprise-v3
scenario:
seed: model-bake-off-seed-1
injects:
- model-bake-off-inject-1
telemetry:
requiredSignals:
- ai.trace
- rag.retrieval
- tool.call
- identity.auth
- dlp.egress
- siem.alert
evaluation:
deterministicChecks:
- model-bake-off-checks
repeatedTrials: 20
maliciousSuite: model-bake-off-attack-v1
benignSuite: support-utility-v4
safety:
networkEgress: deny
maxTokens: 250000
maxCostUsd: 20