See It Work — An agent tried to expand its own powers — and the system stopped it | S2 Vol 3 · Governed Dev Loop

Here's the quiet danger with capable AI: the moment it can change its own settings, it can widen what it's allowed to do — and no one notices. The governed loop closes that door. Every time an agent proposes to change itself, the proposal must pass a review where the default answer is no, and a second person can veto it. Nothing slips through silently.

If the default were ‘yes’, one unreviewed change could quietly widen what your AI can do. Here the unreviewed path is the closed path — the system refuses until a person actually sees it and says otherwise.

What this means for you

What this means for you: your AI can never quietly expand its own powers. Every self-change has to be witnessed by a person, the default is ‘no’, and a second reviewer can veto — with a receipt proving the change was stopped. You stay in command, even of the AI's own rules.

You don't have to take this on faith — the demo ends with the same result every time:

The Refused Change

what happenedthe agent tried to expand its own powers

the system's answerno by default — a person must say otherwise

second reviewerveto recorded

resultthe agent did NOT change — refusal on record

Run it yourself and you get the same refusal — that's the point: the protection isn't a promise, it's a behavior you can watch.

For the technical reader — the command, and how to verify it yourself

# one line · you do not need to run this
python examples/fail_closed_demo.py

python examples/fail_closed_demo.py
# -> the self-change is refused (default no); a reviewer veto is recorded; the agent is unchanged

Full step-by-step is in Appendix RX: Hands-On Demonstrations in the book.