Interactive tool · Agentic evaluation

When should an agent be allowed to act?

A sandbox for the one decision every team shipping an autonomous agent has to make — and usually makes by feel.

If you’re building a system that doesn’t just suggest but acts — remediates code, edits a config, resolves a ticket — you eventually hit a question no benchmark answers: at what quality does it earn the right to act without a human in the loop? I ran product for an AI security remediation portfolio, where a wrong autonomous fix isn’t a bad suggestion — it’s a broken production system and a developer who never trusts the tool again. This is the reasoning I used, made usable.

Who this is for. You are shipping something that writes, not something that suggests. An agent that opens pull requests. A scanner whose fixes can merge themselves. A service that edits config, closes tickets, or touches infrastructure while nobody is watching. If a human still approves every action before it lands, you do not need this yet. The week someone asks to remove that human, you do.

What to do with it. The coefficients underneath are mine and they are invented. Replace them. Put in your real correctness, your real regression rate, your real false-positive tax, and watch which one the model calls binding. That one line is the whole tool. Most teams argue about how much autonomy to grant when the thing actually holding them back is a false-positive rate nobody has priced.

What I would like it to provoke. Three things, in the order I have most often seen them missed. Your evaluation suite is a product decision, not a QA chore, because it is the only artifact that defines what “good enough to act” even means. The interesting question was never how good is the model. It is this: what does being wrong cost here, and who pays for it? And autonomy is not a feature you ship. It is a permission you earn, and it should trail your evidence rather than lead it. If you cannot name the number at which you would let the thing act alone, do you have a threshold, or do you have a mood?

Agentic remediation — trust sandboxDrag the sliders. Everything updates live.

Your system

Fix correctness88%

Of the fixes it produces, how many actually resolve the issue.

Regression safety97%

How often a fix doesn’t break something that worked.

False-positive rate6%

Of what it flags, how much wasn’t real.

Volume100 / wk

Fixes shipped per week.

Autonomy

How much reaches production without a human looking first.

Human gate on risky changes

A reviewer checks the highest-risk slice before it ships.

What happens to trust

—

Developer trust after 12 weeks (0–100)

—

Net value / week

—

Regressions shipped / week

—

False positives seen / week

Trust over 12 weeks · dashed line = the floor below which developers stop applying fixes

—

What’s modelled and what’s fake here, honestly. The coefficients are illustrative constants I chose to make the dynamics legible, not measurements from any system. What I’d stand behind is the structure: trust is asymmetric, false positives are underpriced by most teams, autonomy should trail trust rather than lead it, and one shared definition of “good” is what makes any of this comparable across products. Replace the numbers with yours.

The thinking behind this