# AI agent crash recovery: quit at step 6, does it start over?

URL: https://systemdesignsimple.com/ai-agent-crash-resume-checkpoints/
Published: 2026-11-13
Category: AI Systems
Publisher: System Design Simple (https://systemdesignsimple.com)

## Summary

If an agent process dies and its workflow restarts without saved progress, completed side effects can repeat. In 1,000 simulated crashes of a 12-step agent, restarting redid 5.68 finished steps on average and repeated an action in 70% of runs. A checkpoint after each step cut redone steps to 0.04 but left 3.9% of runs with a duplicate. Keys the service deduplicated made that 0%.

## Key facts

- **Test setup:** A simulated 12-step agent task with 3 side effects (email at step 4, ticket at step 7, test card charge at step 10), 1,000 runs, each killed at a random step, seed 24, Node 22 on one Apple M2 Pro Mac, run 2026-11-13. Step costs are made up; the crash is a thrown error inside one process
- **Restart from step 1:** 5.68 finished steps redone on average (max 11), 14,300 wasted tokens on average, an action repeated in 70% of runs
- **Checkpoint after each step:** 0.04 finished steps redone, 1,188 wasted tokens on average, an action repeated in 3.9% of runs (the 39 crashes that landed after an action and before the save)
- **Checkpoint plus idempotency keys:** Same 0.04 redone steps, 0% of runs with a duplicate action; the service blocked 39 repeats by key
- **Assumption:** 15% of crashes on a side-effect step land after the action and before the checkpoint save

If an agent process dies and its workflow restarts without saved progress, completed side effects can repeat: the same email, ticket or payment a second time. AI agent crash recovery needs two parts, a checkpoint after each step and an idempotency key on each action. In 1,000 modeled crashes of a 12-step agent, restarting redid 5.68 finished steps on average and repeated an action in 70% of runs; a checkpoint after each step cut the redone work to 0.04 steps, and keys that the receiving service deduplicated brought duplicates to 0%.

## Why a restart is worse than lost time {#problem}

> Definition: **Checkpoint**: a saved record of a run's progress (the next step to run and the outputs of finished steps), written to storage that survives the process. **Idempotency key**: a stable identifier sent with an action, so the receiving service can recognise a retry of the same action and return the first result instead of acting again.

An agent run is a list of steps. Most steps only cost tokens and time: read the request, search the docs, write a summary. A few steps change the outside world: send an email, open a ticket, charge a card. If the process dies at step 6 and the agent starts again from step 1, the cheap steps are wasted money, and the side-effect steps happen twice. The customer gets two emails and two charges.

This is the same problem as a retried network request, which we covered for counters in [why a like count can be wrong but a balance never](/like-count-wrong-balance-never/). An agent just has more steps, so there are more places to crash.

## What the simulation measured {#test}

The script `labs/ai-24/simulate-agent-crash-resume.mjs` (Node 22, standard library only) runs a 12-step support task: steps 4, 7 and 10 send an email, create a ticket and charge a test card through a fake external service that records every action it really performs. It runs 1,000 times with seed 24. Each run is killed at a random step, then resumed, and the same crash plan is replayed against three strategies.

| Strategy | Finished steps redone (avg) | Wasted tokens (avg) | Runs with an action twice |
|---|---|---|---|
| A: restart from step 1 | 5.68 | 14,300 | 70% |
| B: checkpoint after each step | 0.04 | 1,188 | 3.9% |
| C: checkpoint + idempotency keys | 0.04 | 1,188 | 0% |

[Image: Terminal output of the crash and resume simulation: restart redoes 5.68 steps with 70% duplicate runs, checkpoint 0.04 and 3.9%, checkpoint plus idempotency keys 0.04 and 0% | The printed output of the run on 2026-11-13 (labs/ai-24/output.txt), captured in Chrome. Reproduce it with `node labs/ai-24/simulate-agent-crash-resume.mjs`.]

## How does AI agent crash recovery use checkpoints? {#checkpoint}

Strategy B saves `{ runId, nextStep, outputs }` after every finished step. On restart it loads that record and starts at `nextStep` with the earlier outputs in hand, so step 7 can still use what step 2 found.

```text
restart:     redone = steps finished before the crash   avg 5.68 of 12
checkpoint:  redone = 0, except a crash in the save gap  avg 0.04
```

In production the checkpoint must live somewhere that survives the restart, such as a database row, a file on disk or a key in a durable store, because a checkpoint in process memory dies with the process. Agent frameworks expose the same idea; [LangGraph's persistence layer](https://docs.langchain.com/oss/javascript/langgraph/persistence) saves graph state as checkpoints per thread and offers storage-backed checkpointers for real deployments. The step that was half done when the crash hit still starts over from its beginning, and that lost partial work is not in the 0.04 figure, which counts finished steps only.

The simulation assigns 40 ms to each save: 12 × 40 ms = 0.48 s per completed run. Restarting adds 18.3 s on average under the invented step costs.

## The gap checkpoints cannot close {#gap}

A side-effect step does two things in order: perform the action, then save the checkpoint. If the crash lands between them, the email went out but the saved state does not know it. For the email at step 4, the resumed run sees `nextStep = 3` (the code counts from zero) and sends the email again.

The simulation assumes 15% of crashes on a side-effect step land in that gap. That gave 39 of 1,000 runs, which is where the 3.9% comes from. Swapping the order (save first, then act) does not help, because a crash after the save then skips the action entirely and the duplicates turn into missing actions.

## Idempotency keys close the gap {#keys}

Strategy C sends each action with a key built from the run and the step, for example `run-81:step-10:charge`. Deriving the key from the persisted run identity and step keeps it stable across retries. The service stores the first result under the key, and when the same key arrives again it returns that result and does nothing new. In the simulation the service blocked exactly the 39 repeats, and duplicates fell to 0%.

This only works if the receiving service checks the key. [Stripe's idempotent requests](https://docs.stripe.com/api/idempotent_requests) are a documented example: for API v1 it saves the status code and body of the first request with a key and returns them for retries, and errors if the parameters differ. Stripe may prune API v1 keys once they are at least 24 hours old. Reusing a key after it has been pruned creates a new request. So receiver-side deduplication prevents repeated actions while the key remains valid, and a resume days later needs its own check. Part 43 of the main series ("You pressed Pay twice") covers the general rule, idempotent operations. For failover one layer down, see what happens when the [main server dies at 3 am](/main-server-dies-at-3am/).

On your own services, the receiver must claim the key atomically, for example with an insert under a unique constraint, so two concurrent requests with the same key cannot both pass the check. For changes inside one database, commit the key, request parameters, result and business change atomically. For external actions, persist an outbox entry in that transaction, then deliver it with a stable key that the receiver deduplicates. An outbox can deliver duplicates, as [AWS's transactional outbox guide](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html) notes, so the receiver's key check has to stop the second email or charge.

## What we saw when we ran it {#we-ran-it}

We ran `node labs/ai-24/simulate-agent-crash-resume.mjs` on our own Mac on 2026-11-13. It played a 12-step task 1000 times with seed 24, crashed each run at a random step and resumed it three ways. The terminal printed one table. In the restart row, 5.68 finished steps were redone on average and a side effect repeated in 70% of runs.

![Terminal output of our ai-24 run: node simulate-agent-crash-resume.mjs](/img/posts/ai-agent-crash-resume-checkpoints/lab-run.webp)
*Our terminal after the run: three strategies, 1000 seeded runs, one summary table.*

With strategy B, checkpoints cut redone steps to about 0.04 on average (39 repeated steps over 1000 runs, printed rounded), but duplicates did not go to zero: 39 of 1000 crashes landed after a side effect and before the checkpoint save, so those runs sent the email, the ticket or the card charge again, which is 3.9% of runs. Strategy C kept the same average of about 0.04 redone steps, and the fake service blocked all 39 repeats by key, so duplicates dropped to 0%.

![Screenshot of the official page: Stripe API reference: Idempotent requests](/img/posts/ai-agent-crash-resume-checkpoints/source-1.webp)
*Stripe's API reference page on idempotent requests, the receiver-side check we lean on.*

That page says that for API v1, once endpoint execution has begun, Stripe saves the status code and body of the first request with a key and returns them for retries. API v2 handles keys differently. In your own agent, checkpoints save the time and the receiving service's key check prevents the second charge.

## The core, as pseudocode {#code}

This is pseudocode for strategy C. `store` is durable storage scoped by run ID. `step.run` has no side effects: it only computes a step's output, or the request parameters for an action, from saved outputs, so a retry sends the same parameters.

```js
// resume-with-keys (pseudocode): the shape of strategy C
function runAgent(runId, steps, store, service) {
  const saved = store.load(runId);              // durable, scoped to this run
  if (saved && saved.runId !== runId) throw new Error("checkpoint is for another run");
  const id = saved ? saved.runId : runId;       // persisted identity drives the keys
  const start = saved ? saved.nextStep : 0;
  const outputs = saved ? { ...saved.outputs } : {};
  for (let i = start; i < steps.length; i++) {
    const step = steps[i];
    let out = step.run(outputs);                // no side effects
    if (step.effect) {
      const key = `${id}:step-${i + 1}:${step.effect}`; // same on every retry
      out = service.perform(step.effect, key, out);     // service returns the first receipt for a seen key
    }
    outputs[i] = out;
    store.save(id, { runId: id, nextStep: i + 1, outputs });
  }
  return outputs;
}
```

If a model call builds the parameters, save them before the action so a retry cannot change them. To see a crash in the gap and the blocked repeat, run the full simulation: `node labs/ai-24/simulate-agent-crash-resume.mjs`.

## What this test does not show {#limits}

- No real process was killed and no real disk was used; the crash is a thrown error and the checkpoint is a string in memory that is assumed to survive.
- The 15% gap share is an assumption. The 3.9% was observed in the seeded simulation under the assumed crash distribution.
- Token and time costs per step are made up, so the table shows the shape of the effect only.
- Steps here are deterministic. A real model may produce a different output when a step reruns, which is one more reason to save outputs in the checkpoint rather than recompute them.

A related agent failure, acting on a screen that changed since the last look, is covered in [why a computer-use agent checks after every click](/move-one-button-can-ai-finish-the-task/).

## Our take {#our-take}

Save progress and outputs after every step, to storage that outlives the process, and resume from there. Then give every side effect a stable key and make sure the service on the other end checks it. Keys depend on a service you may not own, so read its docs for how long it keeps them.

This article describes general patterns with a small simulation. It does not describe any company's private systems. Numbers come from one run on 2026-11-13 with made-up step costs.

## Frequently asked questions

**Does an AI agent start over if I close the app?**
Not always. Closing the chat window may leave the run going on a server or background worker. If the worker itself stops, an agent with checkpoints in durable storage resumes at the step it was on, and the half-done step runs again from its start.

**Are checkpoints enough to stop duplicate emails or payments?**
No. If the crash lands after the action ran but before the checkpoint was saved, the resumed run does the action again. In our simulation that happened in 3.9% of runs. A stable idempotency key that the receiving service deduplicates closes that gap while the key remains valid.

**How long does a payment service remember an idempotency key?**
It depends on the service. Stripe may prune API v1 keys once they are at least 24 hours old. Reusing a key after it has been pruned creates a new request.

## Sources

- [Stripe API reference: Idempotent requests](https://docs.stripe.com/api/idempotent_requests)
- [AWS Prescriptive Guidance: Transactional outbox pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html)
- [LangGraph documentation: Persistence](https://docs.langchain.com/oss/javascript/langgraph/persistence)
- [RFC 9110, HTTP Semantics, section 9.2.2 Idempotent Methods](https://www.rfc-editor.org/rfc/rfc9110#name-idempotent-methods)
- [Kleppmann, Designing Data-Intensive Applications (exactly-once and idempotence)](https://dataintensive.net/)