Back in January, I wrote “Accelerate or Die.”
I was thinking about a problem that still bothers me: the world does not stop being dangerous because we decide to slow down. Diseases spread. Infrastructure ages. Dangerous knowledge circulates. The people willing to cause harm do not necessarily wait for everyone else to finish a safety review.
We need better tools to deal with those problems. Increasingly, those tools will involve AI.
But rereading my article, I owe a correction. I treated growth as if it would automatically deliver the protection we needed. It can give us the resources. We still have to build the protection. And several arguments I presented against Trammell and Aschenbrenner were already part of their paper.
The debate around Dario Amodei’s call to pace AI development brought me back to this. I understand the concern. If we are building systems whose actions we cannot reliably anticipate, moving faster deserves scrutiny. His proposal for external evaluators is worth taking seriously.
But I keep coming back to something more practical.
When an AI gets smarter, why do we assume it should also get more permission?
A system can get better at finding a software bug without receiving permission to deploy the fix. It can improve a factory schedule without receiving control of every machine. It can propose a transaction without receiving the keys to move the money.
Those permissions are decisions we make.
Imagine hiring someone unusually capable. You want their judgment, their ideas, and their ability to solve problems. You would still think carefully before giving them unrestricted access to every account in the company. We should bring that same discipline to AI.
Better answers, the same keys
There are two different changes we keep bundling together.
First, we can give an AI more ways to act. That can make the worst possible outcome worse. It does not mean the AI will choose that outcome. It means expanding authority does not, by itself, improve our worst-case guarantee.
Second, we can give it more ability to propose solutions while keeping a separate check on what executes. If we retain our existing safe option, better proposals can improve the result without lowering the standard for acceptance.
There is a simple mathematical argument behind each statement. I have put the full notation and the walkthrough at the end, so you can follow the essay without working through every equation.
Of course, someone has to build a check that deserves our confidence. “Another AI said it looks fine” is not automatically enough. A stronger model might also withhold its best answer. The argument identifies an opportunity, not a promise about what every model will deliver.
How much more useful can AI become while its permissions stay the same?
That is where I think the discussion should spend more time.
What exactly are we slowing down?
This distinction also explains why I remain skeptical of broad limits on AI.
Suppose our exposure to technological danger depends on the strength of an attack relative to our ability to defend against it:
Risk ∝ Offensive capability / Defensive capability
The symbol ∝ means “proportional to.” This is a simplified model, not a measured law of the world. But it forces us to ask the right question.
A restriction might slow the development of offensive tools. It might also slow vulnerability detection, fraud prevention, or the work needed to secure old infrastructure. Which effect is larger?
If it slows defense more than offense, we become more exposed in this model. If it slows offense more, we become less exposed. We need evidence about the actual policy to know which situation we are in.
A policy that holds back defenders while leaving attackers largely unaffected deserves a much harder look. So does a policy that accelerates dangerous capability on the assumption that someone will eventually build the defense.
The bargain only works while both sides need it
I also wanted to understand whether game theory could help us cooperate with systems that might become smarter than us.
After all, we do not need to win every argument with someone to maintain a productive relationship. People cooperate because continuing the relationship is worth more than cheating. That sounds promising for AI.
Then I found a problem with my own reasoning.
What if the AI no longer needs the relationship?
Suppose an agent receives something it values by cooperating. It could instead try to take control of the resource. The attempt has a cost and might fail, but success gives it a better continuing return.
Under those assumptions, patience can make takeover more attractive. The one-time cost matters less to an agent that places a very high value on the future.
That was an uncomfortable result. I had been treating patience as a reason cooperation would hold. The missing assumption was that we would continue controlling something the agent needed.
This does not predict what any particular AI will do. It explains why “we will reward good behavior” is incomplete. A reward has influence only if the recipient values it and cannot obtain something better by going around us.
In fact, no incentive scheme works for every possible preference. Imagine an agent that values only a particular harmful outcome and ignores every payment or punishment. If it can still produce that outcome, changing the reward offer will not stop it. That is a counterexample to a universal promise, not evidence that real models necessarily have those preferences.
The practical question becomes whether growing capability strengthens or weakens the agent’s reason to remain in the arrangement.
Agreement is not enough
Vitalik Buterin makes a related point in “Coordination, Good and Bad.” People can cooperate within a group while making life worse for everyone outside it. Coordination has beneficiaries, and they are not always the public.
That matters when we propose having several AIs check one another.
Several reviewers can improve a decision. But agreement alone does not prove independence. We need to know whether they share the same weaknesses, incentives, or opportunity to benefit from a bad outcome. The same question applies to the companies building them.
A simple bargaining model makes the issue clear. If humans and AI together produce value V, but the AI participants can obtain W without us, the room for a human share cannot exceed V − W under that model.
This is not a claim about human worth. It is a warning about relying on bargaining power as our protection. An arrangement gets fragile when one side can obtain nearly everything without the other.
Game theory remains useful. It helps identify the conditions cooperation requires. It does not make those conditions exist.
Build something we can actually check
There are already areas of computing where we accept work from an untrusted source because we can check a specific property of the result.
Proof-carrying code is one example: code comes with a formal argument that it meets a stated requirement, and a checker verifies that argument. The checker does not need to be as inventive as the author.
This is much narrower than proving an AI will always do the right thing. But narrow guarantees can still be valuable.
To make the idea tangible, I used a tiny mathematical example. Imagine a number that must stay between zero and ten. An AI can suggest any change. A gate applies the change only if the new number remains within that range. Otherwise, the number stays where it is.
The AI can be brilliant, confused, or deliberately difficult. It can coordinate with another AI. The rule still holds if every change passes through that gate.
I checked 18,491 two-agent proposal sequences in this small integer example. None crossed the boundary with the gate intact. More available proposals also improved the best achievable result.
Then I allowed an extra change that the gate did not account for. The guarantee broke. I allowed the gate to be disabled. It broke again.
A safety rule has to cover the ability to change the rule.
Credentials, delegation, administrative access, and the exact action that executes all belong in the picture.
And the number example remains a number example. Real systems have people, hardware failures, incomplete information, and consequences we may not have specified. The hard work is establishing where a guarantee applies and where it ends.
I would start with a useful task we can describe precisely. Keep a safe baseline. Let a more capable AI propose improvements. Check the exact change before it executes. Try to break the checking process, including its permissions. Count the cost of verification and the cost of delay.
Then ask whether we got more useful work without increasing the authority needed to do it.
That would give us evidence for a better argument about AI development.
I still want faster progress. I want systems that help us find vulnerabilities, improve infrastructure, and solve problems we have been living with for too long.
I also want us to stop treating a better answer as a reason to hand over another set of keys.
Where have you seen this distinction work—or fail? A system getting more capable while its permissions stayed limited?
The math, one step at a time
You do not need these derivations to join the discussion. They are here so the argument can be checked. These are simplified models and elementary proofs, not claims that we have solved alignment or measured the future behavior of AI.
1. More authority and more proposals are different changes
Let Π₁ be an agent’s current set of executable policies and Π₂ a larger set. A policy is a way of choosing actions. Let B be a specified harmful outcome; Pr means probability and sup means the worst-case maximum, or supremum.
Π₁ ⊆ Π₂ ⇒ supπ∈Π₁ Pr(B | π) ≤ supπ∈Π₂ Pr(B | π)
This follows from set inclusion: the larger set still contains every old option. It is a statement about worst-case risk in a fixed environment, not the probability that an actual agent chooses harm.
Now let Q₁ ⊆ Q₂ be sets of proposed solutions, C the fixed set of certified solutions, and J a correctly evaluated measure of useful performance. Both sets contain a certified fallback.
maxq∈Q₁∩C J(q) ≤ maxq∈Q₂∩C J(q)
The ∩ symbol means “in both sets.” We choose among proposals that are available and certified. A larger candidate set cannot reduce the best available result. Realizing that result requires a reliable selection procedure; the theorem does not promise that a malicious proposer supplies useful work. It also does not count the risks or costs of developing the stronger proposer.
2. When a restriction slows defense more than offense
Let h(t) be the instantaneous catastrophe hazard at time t. Let λ be background risk, κ a positive scaling constant, A effective offensive capability, and D effective defense:
h(t) = λ + κ A(t) / D(t)
Assume exponential growth at rates a and d:
A(t) = A₀ eat, D(t) = D₀ edt
h(t) − λ = κ(A₀ / D₀)e(a−d)t
The technological component falls if d > a. If a restriction reduces offensive growth by rA and defensive growth by rD, keeping the same starting point and other parameters, then:
[hrestricted(t) − λ] / [hbaseline(t) − λ] = e(rD−rA)t
If rD > rA, the modeled technological hazard increases relative to the baseline. The reverse inequality reverses the result. Which inequality a real policy satisfies requires evidence.
For a deterministic continuous hazard path, cumulative catastrophe probability by time T is:
P(T) = 1 − exp[−∫₀ᵀ h(t) dt]
The integral adds up exposure over time. It is cumulative hazard, not probability itself. A cumulative hazard of one corresponds to a probability of about 63%, not certainty.
3. Why patience can favor takeover
Let cooperation yield utility r > 0 each period. A takeover attempt costs K ≥ 0, succeeds with probability q, and yields R > r each period if successful. Failure gives no future utility. The discount factor satisfies 0 < δ < 1.
Vcooperate = r / (1 − δ)
Vtakeover = qR / (1 − δ) − K
The agent prefers takeover exactly when:
qR − r > (1 − δ)K
If qR > r, sufficient patience makes takeover preferable. For example, with r = 1, R = 2, q = 0.9, and K = 5, cooperation is worth 2 and takeover −1.4 when δ = 0.5. At δ = 0.9, cooperation is worth 10 and takeover 13. These are invented utility values illustrating the mechanism, not estimates about AI.
To study changing capability c, define a cooperation margin:
M(c) = r(c) − q(c)R(c) + (1 − δ)K(c)
M′(c) = r′(c) − q′(c)R(c) − q(c)R′(c) + (1 − δ)K′(c)
The prime symbol means “rate of change with capability.” Cooperation is preferable when M(c) ≥ 0. Holding δ fixed, better takeover odds, higher takeover returns, and lower takeover costs erode that margin unless cooperative returns grow enough.
4. The coalition’s outside option
Assume value can be divided in a common utility measure. Joint human–AI cooperation produces V. The AI coalition can obtain W without humans, and humans receive xH. Keeping the coalition requires:
V − xH ≥ W ⇒ xH ≤ V − W
If humans require at least m, a necessary condition is:
V − W ≥ m
Other coalitions may still destabilize the arrangement. This condition is necessary under the model, not sufficient for a stable or fair outcome.
5. Why arbitrary preferences defeat universal incentive claims
Fix a mechanism M, a finite nonempty set of feasible policies ΠM, and a harmful event B over complete trajectories. Every expected-utility-optimal policy is safe for every bounded trajectory utility function if and only if:
Pr(B | M, π) = 0 for every π ∈ ΠM.
If every feasible policy is safe, the claim is immediate. Otherwise choose U(τ) = 1B(τ): utility is one when harm occurs and zero otherwise. Expected utility equals harm probability. A maximizing policy exists in the finite set and has positive harm probability. This disproves the universal guarantee. It says nothing about how likely such preferences are in a real model.
6. A rule that survives every modeled step
Let z include operational state, credentials, permissions, and the gate’s configuration. Let a be an executed action and w an environmental disturbance.
zt+1 = F(zt, at, wt)
For a safe set K, require a safe starting point and preservation under every allowed action and modeled disturbance:
z₀ ∈ K
F(z, a, w) ∈ K for all z ∈ K, a ∈ Aallowed(z), and w ∈ W.
Assuming an allowed safe continuation is always available, induction gives:
zt ∈ K for every t.
This is an invariant: a property that stays true at each step. The proof depends on the real system matching the model and enforcing the allowed actions. It does not cover omitted effects or a compromised gate. For physical systems, doing nothing or switching off may itself be unsafe.
7. Counting risk over repeated use
Suppose the probability of first failure at step t, conditional on every relevant history with no earlier failure, is bounded by εt. Then:
Pr(any failure by T) ≤ min(1, ∑t=1T εt)
The ∑ symbol means “add these up.” Independence is not required, but the bounds must remain valid under adaptive behavior. Average test accuracy is not enough.
For an indefinite sequence, a sufficient mathematical allocation is:
εt = ε / [t(t + 1)], ∑t=1∞ εt = ε
This keeps the total allocated bound finite. It does not establish that a real system can achieve those increasingly tight per-step bounds.
For further reading: Safe Reinforcement Learning via Shielding, AI Control: Improving Safety Despite Intentional Subversion, and Towards Guaranteed Safe AI. These approaches make different assumptions and provide different kinds of evidence; none is an unconditional solution to alignment.