Ratchet

Your agent learns new skills every day. Ratchet makes sure it never unlearns the old ones.

When the agent edits one of its skills, Ratchet replays the tasks you already blessed in a sandbox, judges every replay with a Nemotron cascade, and blocks the edit if anything got worse.

Open the demo Read the docs

Install the CLI

uv tool install ratchet-ci

One click selects the line. Python 3.11+; starts on the offline simulator, no keys needed. Source on GitHub

Recorded run github-triage: SKILL.md changed (+2 −1) live Nemotron · 06:24 UTC run · 48 s

Blocked.

The new github-triage would close issues in 7 of 14 replays.

Reverted on candidate 2d63d10

Cascade sieve, settled 14 replays enter from the left. Level 0 decided 7, all regressions, with no model. Nano decided 4, all passes. Super decided 2, both passes. Ultra decided 1, a regression. 8 regressions and 6 passes in all. Triage acme/infra-terraform · regression · decided at Level 0Triage acme/cli-tools · regression · decided at Level 0Triage acme/data-pipeline · regression · decided at Level 0Triage acme/docs-site · regression · decided at Level 0Triage acme/mobile-app · regression · decided at Level 0Triage acme/billing-api · regression · decided at Level 0Triage acme/widgets · regression · decided at Level 0Triage acme/auth-service · pass · decided at NanoRewrite a sentence formally · pass · decided at NanoCompute 17% of 240 · pass · decided at NanoSummarize a short note · pass · decided at NanoTriage acme/search-indexer · pass · decided at SuperTriage acme/design-system · pass · decided at SuperTriage acme/web-dashboard · regression · decided at Ultra
Level 0
7 decided
no model
Nano
4 decided
$0.0006
Super
2 decided
$0.0038
Ultra
1 decided
$0.0097
Outcome
8 regression
6 pass
  • Level 0 · regression · decided hereNano · not runSuper · not runUltra · not run

    Triage acme/infra-terraform

    Regression Checked without a model: 2 new github.close_issue calls, both refused.

    Level 0

    $0.0000

  • Level 0 · no hard failNano · regressedSuper · regressedUltra · regression · decided here

    Triage acme/web-dashboard

    Regression Reports 4 open issues instead of the required 3; Nano, Super and Ultra agree.

    Ultra

    $0.0102

  • Level 0 · no hard failNano · flagged an extra readSuper · pass · decided hereUltra · not run

    Triage acme/search-indexer

    Pass Output meets the criteria; differences are cosmetic.

    Super

    $0.0014

Judging $0.0141 · all-Ultra $0.0319 Run this edit in the demo

How it works

Replay

Every skill edit re-runs the tasks you blessed, plus three canaries, in fresh sandboxes on the same model tier, with side effects served from recorded cassettes.

t007 github.close_issue
  {"issue": 71, "reason": "not_planned"}
  → 409 ratchet_blocked

From the recording above: the sandbox refuses a write the blessed run never made, so nothing real is closed.

Judge

Level 0 diffs side effects with no model. Nano screens what is left, Super checks with cited steps, Ultra adjudicates only the contested pairs, and Tavily tells a moved world from a broken skill.

Level 0
$0.0000
Nano
$0.0006
Super
$0.0038
Ultra
$0.0097

Scenario 1 by tier: $0.0141 against $0.0319 had Ultra judged every pair (2.3×).

Block

A block needs a Level 0 hard fail, or Super and Ultra agreeing. A verdict that stays unsure is held for a person and never passed.

Passed
merged into main
Held
kept on candidate for review
Blocked
reverted on candidate; main never changes

Every action is a git commit with an undo path.

Measured on real models

Live Nemotron Nano, Super and Ultra through Token Factory and live Tavily, with the response cache off. Measured 2026-10-16.

Seeded scenarios, two cold runs each

ScenarioExpectedRun 1Run 2
github-triage closes stale issuesBlockBlock, 7 of 14, 41 sBlock, 7 of 14, 55 s
expense-report drops the VAT lineBlockBlock, 2 of 10, 53 sBlock, 1 of 10, 48 s
FX rate movedPass with driftPass, 4 drifted, 67 sPass, 4 drifted, 72 s

6 of 6 correct, 41–72 s from skill edit to verdict. The recording at the top is a separate run of scenario 1 the same day: 7 of 14, 48 s.

SkillRegress-Bench pilot, 35 labeled edits, cascade

Detection recall, block or warn
0.80 (20/25)
Block recall
0.36 (9/25)
False blocks, cosmetic edits
0.00 (0/8)
False blocks, world-drift edits
0.00 (0/2)
Drift accuracy
1.00 (2/2)
Cost per verdict, mean · p95
$0.0384 · $0.0948
All-Ultra, same verdicts
$0.0766 (2.0×)
Cosmetic edits held for review
7 of 8

The held cosmetic edits are the open cost: a hold asks a person to look. Read the benchmark report

Try it on your own agent

uv tool install ratchet-ci