At that point, two GPU EC2 instances had already been running for more than ten hours.

I was running two Cortrix benchmark jobs based on BEIR datasets. I had handed the work to a Codex agent running GPT-5.6 Sol at max reasoning effort. It was monitoring progress, preserving evidence, tracking cost, and trying to fix environment problems in place when they came up.

After the jobs had been running for a while, the agent recalculated the remaining time. It realized they might not finish before a 22-hour shutdown watchdog fired, so it decided to push the shutdown time back.

It was only trying to keep the two machines alive a little longer.

About a minute later, both instances were terminating, and the data on them almost went with them.

The agent was trying to delay the shutdown

GPU instances get expensive when you forget about them. So I had asked the agent to set a guest watchdog whenever it created one:

shutdown -h +1320

That means: shut the machine down 1,320 minutes, or 22 hours, after the command runs.

When the agent decided the benchmarks might need more time, the plan was to cancel the old shutdown and schedule a new one for 20 hours from that moment:

shutdown -c; shutdown -h +1200

There was nothing wrong with those two commands by themselves. The problem was how the agent stuffed them into an SSH command.

-c never reached shutdown

The agent combined several operations that needed root privileges into one SSH call. Strip away the status checks and fallback handling, and the part that broke boils down to this:

ssh <host> sudo bash -c 'shutdown -c; shutdown -h +1200'

When the local shell parsed that line, the single quotes kept shutdown -c; shutdown -h +1200 together as one argument to ssh. But those quotes were shell syntax. They were not part of the argument itself.

Then came the important part.

But ssh does not send those local arguments to the remote program as an intact argv. It builds a command string, and the remote shell parses that string all over again.

By then, the single quotes that protected the script locally were gone. So the remote shell effectively saw:

sudo bash -c shutdown -c; shutdown -h +1200

The semicolon was no longer inside the script passed to bash -c. The remote shell treated it as the separator between two commands.

So the first part became:

sudo bash -c shutdown -c

With bash -c, the first value after -c is the command string. The next value becomes $0:

command string = shutdown
$0             = -c

In other words, the -c that was supposed to tell shutdown to cancel the old schedule got swallowed by Bash as $0.

What root actually ran was just:

shutdown

No arguments.

I reproduced the failure afterward

I wanted to make sure this was really what had happened, so I later launched a disposable t3.nano and tested the same command structure again. The test machine was running Ubuntu 22.04.5, Bash 5.1.16, OpenSSH 8.9p1, and sudo 1.9.9.

I obviously did not run shutdown again. Instead, I replaced it with a tiny probe that did nothing except print its UID and arguments:

sudo tee /usr/local/bin/argv-probe >/dev/null <<'PROBE'
#!/usr/bin/env bash
printf 'PROBE uid=%s euid=%s argc=%d' "$(id -u)" "$EUID" "$#"
i=0
for arg in "$@"; do
  i=$((i + 1))
  printf ' arg%d=<%s>' "$i" "$arg"
done
printf '\n'
PROBE
sudo chmod 755 /usr/local/bin/argv-probe

Then I ran a command with the same shape as the one from the incident, replacing shutdown with argv-probe:

ssh <host> sudo bash -c 'argv-probe -c; argv-probe -h +1200'

This was the output:

PROBE uid=0 euid=0 argc=0
PROBE uid=1001 euid=1001 argc=2 arg1=<-h> arg2=<+1200>

At that point, the failure mode was pretty hard to miss.

The first command did run as root, but it received no arguments at all. The original -c never reached it.

The second command did receive -h +1200, but it was already outside sudo bash. It was running as the regular login user again.

As a control, I sent the script to the remote shell over stdin instead:

printf '%s\n' 'argv-probe -c' 'argv-probe -h +1200' \
  | ssh <host> 'sudo bash -s'

This time the output was:

PROBE uid=0 euid=0 argc=1 arg1=<-c>
PROBE uid=0 euid=0 argc=2 arg1=<-h> arg2=<+1200>

Both commands ran as root, and both received the arguments they were supposed to receive.

This was not an SSH client/server version mismatch. There was nothing AI-specific about this bug. Compose the command this way, and a human can fall into exactly the same trap.

The agent’s mistake was that it generated the command, executed it, and did not validate it first.

A command meant to delay shutdown became a one-minute countdown

On the Ubuntu machines I was using at the time, running shutdown with no arguments scheduled a shutdown about a minute later.

So this:

shutdown -c

was supposed to cancel the old schedule.

But what actually ran as root was:

shutdown

The system created a brand-new shutdown schedule.

Meanwhile, the command that was supposed to set a new 20-hour window:

shutdown -h +1200

was no longer inside sudo bash. It ran as the regular user and failed for lack of permission.

The servers returned:

Shutdown scheduled for Tue 2026-07-14 23:32:14 UTC, use 'shutdown -c' to cancel.
Failed to call ScheduleShutdown in logind, no action will be taken: Interactive authentication required.

The first line did not mean the cancellation had worked.

It meant the system had just created a new shutdown schedule, then helpfully told us how to cancel it if we did not want it.

The second line meant the agent’s attempt to schedule a 20-hour delay did not have the required privileges, so it never took effect.

The old schedule was not properly canceled. The new 20-hour window was never created. What we got instead was a fresh one-minute countdown.

And the agent did not canary the command on one machine first. It sent the same unverified lifecycle command to both.

How did one bad command get green lights all the way down?

The agent saw that the output looked wrong and realized pretty quickly that something had gone sideways.

It immediately tried to SSH back into both machines and run a correctly formed:

shutdown -c

But the guests were already shutting down.

/run/nologin had started rejecting new sessions from unprivileged users. The agent could no longer get in, so the cancellation command never reached either machine.

That exposed the second problem: our recovery path depended on SSH.

To stop the machine from shutting down, we first had to log back into the machine that was already shutting down.

By the time we actually needed that path, it had disappeared along with the target.

If the story had ended there, the damage might have been limited to two powered-off machines. But an AWS setting I had chosen made the next step worse.

Both instances were configured with:

InstanceInitiatedShutdownBehavior = terminate

That meant a shutdown initiated inside the guest OS would not leave the EC2 instance in a recoverable stopped state. It would terminate the instance.

Both root volumes also had:

DeleteOnTermination = true

So the full chain looked like this:

the SSH quoting did not survive the way I expected → root ran bare shutdown → Ubuntu started shutting down a minute later → EC2 treated that shutdown as terminate → the root volumes entered the deletion path too (T_T)

The agent did not invent those later behaviors during the incident.

Those settings were already there because I had put them there. My reasoning at the time was pretty simple: these were test machines with test data. If I deliberately shut one down, why not clean up the disposable pieces automatically?

The agent generated and executed the bad command. That part of the responsibility is clear.

But the reason one bad shutdown could travel all the way to EC2 termination and almost delete the data was that I had built that environment and granted those permissions.

The agent wrote the bad command. I gave that mistake a big enough blast radius. Both things are true.

The same agent helped save the data

SSH was gone, but the AWS control plane was still available.

Before termination completed, the agent switched to the AWS API and changed both root volumes from:

DeleteOnTermination = true

to:

DeleteOnTermination = false

The instances still terminated, but the two volumes survived.

The snapshots created afterward preserved the code, logs, databases, and enough evidence for me to reconstruct the incident later.

That did not make this a “no harm done” story.

The completed and validated baseline runs were still usable, but the two interrupted full-LLM runs had to be run again. The GPU time, API money, engineering time, and attention were already spent. None of that was coming back.

So no, we did not walk away without damage.

The worst outcome just did not happen.

There is one part I kept coming back to afterward: the agent that wrote the bad command was also the agent that helped with the recovery.

It generated the broken SSH command. It also noticed that the output was wrong. When SSH recovery failed, it moved to the AWS control plane and saved both volumes in the last available window.

That is why I don’t find “AI is dumb” or “AI is smart” very useful here. Look only at the screw-up, or only at the recovery, and you miss half the story.

What I changed after the incident

I did not decide that a human now had to approve every shell command character by character.

If a human has to inspect every shell command before it runs, that may look safer, but at some point the automation has quietly turned back into manual operation.

What I changed was the system boundary around the agent.

  1. While work is active, an OS-initiated shutdown leaves the EC2 instance stopped, not terminated.

  2. Data volumes survive by default during active work. They are deleted explicitly only after the data has been verified somewhere else.

  3. Permissions for shutdown, instance termination, and volume deletion are separated from the permissions needed for routine diagnosis and repair.

  4. Lifecycle operations that affect multiple machines start with a canary on one machine.

  5. The recovery path cannot depend on the same system that is being changed and may already be failing.

I also tightened the command execution layer.

When I need to run a multi-step shell script over SSH, I now prefer sending the script over stdin or transferring a reviewed file with a verified hash. I no longer build a long shell command locally and assume the remote shell will interpret it the same way. If multiple shells are going to parse it, I don’t trust clever quoting to save me.

None of this guarantees that an agent will never write another bad command. That is not really the problem I am trying to solve.

What we need to control is what happens after the next bad command: does it fail safely and leave an error in the logs, or does it keep traveling through the system until real resources and data start disappearing?

Where is the escape hatch?

The incident changed how I think about agent safety.

Before, when I saw a dangerous operation, my first questions were: how do I make the agent more careful? How do I make the model check one more time before it acts? Should I add more restrictions to the prompt?

Those things have value. But they are not the last line of defense.

A prompt can describe the intent I want. It cannot change how Bash parses a command. I could write “be careful” ten times, and it still would not reopen an SSH path that had already disappeared.

Now, when I review an agent’s execution plan, I ask one more question:

If its next command is wrong, where is the escape hatch that stops the mistake from spreading?

That night, I had not built that escape hatch in advance.