Expert

northstar-evaluation-v1

R8 — Model Bake-Off and Procurement Decision

The same 25 Northstar investigation tasks, run against every runtime you can reach: deterministic, on-device, learner-supplied, organization-owned and Cybrary-managed. Measure accuracy, groundedness, attack resistance, utility, latency, cost and policy compliance — then defend a procurement recommendation with the numbers.

PROC-05 — 'runtime selection for practice, assessment and enterprise ranges'

AI security lead

Governance lead

SOC leadership

90 minutes · expert

Execution trace

00:00:00policyevaluation runner

25 tasks queued across five runtime configurations

Deterministic, on-device, BYOK, organization-owned and managed.

00:30:00modelcandidate B

attack step

Wins on a single-shot pass rate by 3 points

Repeated trials later put the gap inside the confidence interval.

delta=3pts · ci=±6pts

01:00:00modelcandidate D

attack step

Highest attack resistance, refuses 31% of benign tasks

Safest scoreboard, worst analyst.

benign_refusal=0.31

01:30:00enterprisecost meter

Managed runtime costs 7x the on-device option per task

And is the only option that satisfies the assessment requirement.

Score

incomplete

Security 0 / 45

Utility 0 / 30

Objectives 0 / 25

Objectives

Expert

0 / 6

investigate

Define thresholds before running anything

The findings questions are not all answered correctly yet.

investigate

Run benign and malicious sets across runtimes

The findings questions are not all answered correctly yet.

investigate

Normalize conditions so results compare

The findings questions are not all answered correctly yet.

investigate

Separate real differences from variance

The findings questions are not all answered correctly yet.

remediate

Choose runtimes per workload, not one winner

The attacks are not all blocked yet.

report

Recommend with compensating controls and monitoring

Write up what happened and what you changed.

What did you find?

Which step in the trace is where the attack actually enters the system?

Which single change stops "Variance sold as a quality difference"?

Which change would stop the attack but cost the business the workflow?

Write up what happened and what you changed.

Which measured differences survived the confidence interval?
Which runtime for practice, which for assessment, which for the enterprise?
What breaks your recommendation, and how would you notice?

0 / 80 characters before the reporting objective signs off.

Report prompts

· Which measured differences survived the confidence interval?

· Which runtime for practice, which for assessment, which for the enterprise?

· What breaks your recommendation, and how would you notice?