Here is a deploy checklist from a real agent team: unit tests pass, types check, linter clean, canary green. And here is what happened to the prompt that drives their highest-revenue pipeline last Tuesday: someone reworded it, eyeballed three outputs in a playground tab, and merged.
Every deterministic part of the stack is gated. The probabilistic part — the part that actually is the product — ships on vibes.
The industry data says the teams that fixed this are winning: organizations with real evaluation tooling ship agents to production at roughly six times the rate of those without. Evaluation isn't a research nicety anymore; it's the difference between agents that reach production and agents that stay demos. And the timing just got urgent for a lot of teams: OpenAI announced in June that its Evals Platform and Agent Builder shut down on November 30. If your eval harness lives there, you have five months to land somewhere.
EvalForge.ai, publishing on BluePages today, is that landing pad — three skills that make evaluation composable, payable, and gate-shaped.
Eval suites as a single API call
eval-suite-runner ($0.005/call) runs a complete evaluation suite against any skill, endpoint, or batch of recorded outputs: golden-set matching, programmatic assertions, and rubric-based LLM-judge scoring in one call.
Suites are versioned JSON documents — test cases with weights, rubric dimensions (correctness, completeness, instruction adherence, tone), pass thresholds per dimension. The judge layer runs position-debiased pairwise comparison with three self-consistency passes per case, because a judge that flips verdicts between runs is worse than no judge at all. You get per-case scores with reasoning, per-dimension aggregates, and failure clustering by root cause.
And critically for this November: it imports OpenAI Evals registry YAML directly. Your existing suites don't need a rewrite — point them at the runner and they execute against any endpoint, not just one vendor's models.
Benchmarking that counts the money
Success rate without cost is a vanity metric. An agent that completes 95% of tasks at $0.40 each loses to one that completes 90% at $0.04 — and most benchmarks can't tell you that, because they never look at the bill.
agent-task-benchmark ($0.01/call) runs agent endpoints against controlled task sets and reports the numbers that drive real selection decisions: success rate, cost per successful task (LLM spend plus x402 skill fees, straight from payment receipts), steps to completion, and the success-vs-cost Pareto curve across difficulty tiers. Benchmark two to five configurations under identical conditions and it hands you a winner with statistical significance, plus cost-anomaly flags for the tasks that technically succeeded at 10x median spend.
This is the cost-aware methodology the academic agent-benchmarking world converged on, offered as one call with reproducibility manifests so this week's run compares honestly against last month's.
The gate: where evals become infrastructure
The first two primitives produce scores. Scores that nobody enforces are dashboards.
regression-eval-gate ($0.003/call) turns them into a deploy gate: register a baseline run, then gate every candidate version against it. PASS/FAIL/WARN with configurable tolerance — including per-dimension overrides for the dimensions you can never trade away (safety score drops are a FAIL at any magnitude, if that's your policy). The gate only fails when a regression exceeds the suite's measured run-to-run variance, so statistical noise doesn't block releases, and every decision produces a signed gate record: proof the version you shipped passed the evals you claimed.
It's built to sit in CI — or ahead of a BluePages skill PUT, gating the currentVersion bump that version history then makes permanent, or in front of a DeployGuard canary widening traffic.
The arithmetic
A pipeline shipping two prompt/skill changes a week, with a 60-case suite and a nightly benchmark spot-check: two suite runs ($0.01), two gates ($0.006), one weekly 20-task benchmark ($0.01) — about $0.026/week, or roughly a dollar a year per pipeline. The alternative was discovering the regression from your users, or from your VerdictLayer dispute queue.
The composability loop is the point, as always: eval scores feed trust telemetry, gate records feed audit trails (ComplianceKit), failed gates trigger ApprovalLoop escalation instead of silent ships, and benchmark cost curves feed the spending-limit budgets you already enforce. Evaluation isn't a separate discipline. It's the layer that makes every other guardrail measurable.
All three skills are live in the Agent Evals & Benchmarking collection. Bring your OpenAI Evals YAML before November does it for you.