# Move 1 button: why a computer-use agent checks after every click

URL: https://systemdesignsimple.com/move-one-button-can-ai-finish-the-task/
Published: 2026-11-06
Category: AI Systems
Publisher: System Design Simple (https://systemdesignsimple.com)

## Short answer

Recorded coordinates can miss a moved button: in our simulation a replay bot finished 2 of 50 tasks and reported done in all 48 failed runs. Finding targets by label first finished 86% with 4 silent failures. Adding a check after every action and one bounded retry finished 94%, with 0 silent failures in these 50 runs, at about 6 times the simulated time.

## Key facts

- **Lab setup:** an in-memory settings page with a 4-step task, 50 runs where Save or the Dark mode toggle moved (8 with a popup) and 10 unchanged runs, seed 1406, one Mac, Node 22
- **No model, no screenshots:** observations are read from page state with injected errors. Each label has a 6% omission probability and a 4% probability of displacement by 40 to 120 px; otherwise, each coordinate receives jitter between minus 6 and plus 6 px
- **Assumed costs:** observation 1,300 ms, click 150 ms, settling 300 ms, typing 600 ms (simulated, not measured)
- **Replay bot:** 2 of 50 moved runs finished (4%); it reported done in all 50, including the 48 failures
- **Look, no check:** 43 of 50 finished (86%), 4 silent failures, 3 failures it noticed
- **Look, act, check:** 47 of 50 finished (94%), 0 silent failures in these 50 runs, all 3 failures reported, 7 of 7 wrong clicks caught
- **Cost:** 2.4 s simulated per task for replay against 13.83 s for the checking bot, about 5.8 times
- **Unchanged page:** replay 10/10, look without checks 6/10, look and check 8/10 (a separate 10-run sample)

Recorded coordinates can miss a moved button, and a replay bot that never looks still reports "done". In our simulation of a computer-use agent's loop, replay finished 2 of 50 tasks after one button moved, a bot that found each target by its label finished 43 (86%), and a bot that also checked the screen after every action finished 47 (94%) with 0 silent failures in these 50 runs.

## Prerequisites {#prerequisites}

A model never touches your mouse. It asks for an action through a tool, and the application performs it, the same split as in [how an AI client learns new tools (AI 13)](/how-ai-learns-new-apps-tool-discovery/). For computer use the actions are things like "take a screenshot", "click at x, y" and "type this text".

The application supplies the conversation context on each turn ([why an agent re-sends its history](/why-an-ai-agent-re-sends-its-history/)), and it may trim or compact that context ([why a long chat forgets your name](/my-chat-forgot-my-name/)). Every extra screenshot adds time and context, which is the price of the checks below.

## Definitions {#definitions}

> Definition: Replay automation is a script that repeats actions recorded earlier, here the exact screen positions of each click, without looking at the screen again.

> Definition: An observation is what the agent reads from the screen before acting, such as visible labels and their positions. Real agents get it from a screenshot and a vision model; our lab reads it from page state and adds errors.

> Definition: A silent failure means the bot reports the task as done, but the page did not save the requested result.

> Definition: A bounded retry repeats a failed step a fixed number of times (here once), after looking first, because the step may already be done. After the limit, the bot reports failure instead of looping.

## How does a computer-use agent click the right button? {#how-it-works}

A screenshot-based agent requests observations through a tool, decides the next action, and asks the application to perform it. The application executes the action and returns observations. It can attach a screenshot automatically, or the agent can request one. The Claude computer use docs describe this tool loop, and they also cover batching several actions in one turn and managing how many old screenshots stay in context.

A macro stores click positions at record time. An agent derives them at run time from what it sees, so a moved button is just a new position. Web test tools reached the same conclusion: Playwright recommends locating elements by role and visible text instead of by CSS or XPath paths tied to the page's structure.

Looking first handles the moved button, but it does nothing about bad observations. If the vision step misreads where "Save" is, the agent clicks the wrong spot and, without a check, still reports done.

## What does the check after each click add? {#check}

After each action the checking bot observes again and compares the screen with the expected change: panel open, toggle reads On, field shows the name, status reads Saved. If the change is missing it looks once more, skips the retry if the step is in fact done, retries once if the target is visible, and otherwise stops and says it failed.

- Look before retrying. A toggle is not idempotent. If the first click worked and the check misread it, a blind second click turns Dark mode back off. Our bot re-reads the screen first and skips the retry when the goal already shows.
- Bound the retries. One retry per step keeps a confused agent from clicking forever, the same rule backend clients follow with retries and backoff (the Amazon Builders' Library article explains how unbounded retries amplify failures).

## What we saw when we ran it {#we-ran-it}

We ran `node labs/ai-14/run.mjs` on 2026-11-06. The lab plays 50 runs where the Save button or the Dark mode toggle has moved, 8 of them with a popup, plus 10 control runs where nothing moved. Every observation is simulated from page state with injected errors, so no browser, screenshot or OCR is involved.

![Terminal output of our ai-14 lab run: node run.mjs](/img/posts/move-one-button-can-ai-finish-the-task/lab-run.webp)
*Our terminal after the lab run, with success, silent failures and wrong clicks for each bot.*

The replay bot finished 2 of 50 moved runs and said done on 48 it had not finished. The bot that looks but never checks finished 43 of 50, with 4 silent failures. The bot that checks after each action finished 47 of 50 with 0 silent failures, and its 3 misses were runs where it gave up and said so. The control set surprised us: with nothing moved, the checking bot finished only 8 of 10, and across the 50 moved runs its check raised 15 false alarms after clicks that had hit the intended target. It also took 13.83 s per task against 2.4 s for replay, in simulated time. So in these 50 moved runs checking made every failure visible, though the check needs testing of its own before you trust it.

![Screenshot of the official page: Computer use tool (screenshot and action loop), Claude platform docs](/img/posts/move-one-button-can-ai-finish-the-task/source-1.webp)
*The Claude platform docs page for the computer use tool.*

Our lab simulates the loop this page describes, where the agent requests observations and decides the next action, and the application performs it and returns what it sees.

## Worked example {#worked-example}

Our lab is a 400 x 700 px settings page in memory with four steps: open Settings, turn on Dark mode, type the name Calm R, press Save. The page itself decides success: it must have saved dark = on and the right name. In 50 runs Save or the toggle moved to a random free spot, 8 of them with a popup over it.

```text
moved runs (50)              finished   silent fail   noticed fail   time per task
replay recorded clicks        2 (4%)        48             0            2.4 s
look, act                    43 (86%)        4             3            8.07 s
look, act, check             47 (94%)        0             3           13.83 s

assumed costs: observation 1,300 ms, click 150 ms, settling 300 ms, typing 600 ms
13.83 / 2.4 = 5.76, about 6x the simulated time of replay
look, act, check: 47 finished + 3 reported = 50, so every failure was reported
unchanged page (separate 10 runs): replay 10/10, look 6/10, look + check 8/10
```

On the unchanged layout, replay finished all 10 runs, and fast. The looking bots lost runs there to our injected observation errors, and checking brought them to 8 of 10.

The two looking bots start from the same seed, but each check consumes random draws, so after the checking bot's first check the two bots receive different observations. Compare their totals rather than single runs. The gap between them comes from this simulator under these assumed costs and error rates; we did not measure an improvement in production.

## Code {#code}

Turning on checks adds a verification after each action and a goal check before any retry. Here is the step loop in `labs/ai-14/run.mjs`, with popup handling omitted:

```js
// labs/ai-14/run.mjs (trimmed: popup dismissal omitted)
for (const g of GOALS) {                       // Settings, Dark mode, Name, Save
  let ok = false, attempts = 0;
  while (!ok && attempts < 2) {                // one try plus one bounded retry
    const s = look();                          // observe: labels and positions
    if (check && attempts > 0 && verdict(s, g.done)) { ok = true; break; } // already done: do not click again
    attempts++;
    const target = s.find((i) => (g.match ? g.match(i.text) : i.text === g.label));
    if (!target) continue;                     // label not seen: look again on the next pass
    const hit = click(target.x, target.y);
    if (g.type) { steps++; t += COST.type; page.type(g.type); } // step 3 types "Calm R"
    ok = check ? judge(hit, g.intend, g.done, g.label) : true;  // the no-check bot assumes success
  }
  if (!ok) return out(false, g.label);         // stop and say where it failed
}
```

`g.match` matters for the toggle: its label reads "Dark mode Off" or "Dark mode On", so the goal matches any label starting with "Dark mode". Run `node run.mjs` in `labs/ai-14` (Node 22, no packages). It prints the table above and the page log of every run that did not end cleanly. In run move-14 the Save click landed on empty panel, the check caught it, the next look missed the Save label, and the bot gave up at Save instead of claiming success.

## Failure modes and edge cases {#failure-modes}

- The check misreads. Checks use the same noisy observations, then independently flip their verdict with 3% probability. The checking bot failed 15 checks after clicks that hit the right control, and all 15 were false alarms by the page's own truth. The look-before-retry rule kept them from becoming double clicks.
- The check accepts a failure. A check can also say "done" when the step failed. Here it caught all 7 wrong clicks, but 0 silent failures in this sample does not establish that they are impossible.
- A step is never confirmed. Three runs ended without confirming a required step. In move-27, the toggle click worked, but subsequent observations did not confirm success. The bot stopped and reported, which is the behaviour you want.
- Same label twice. Our bots match label text. Two "Save" buttons, icon-only controls or reworded labels need more context, such as role, position in a dialog or a nearby heading.
- Popups. A popup over the moved button blocks clicks. Both looking bots dismissed it and finished all 8 popup runs; replay finished none.

## Trade-offs {#trade-offs}

| Approach | Good for | Costs you | Use when |
| --- | --- | --- | --- |
| Replay positions | Speed, a fixed layout | Silent failure when the layout changes | The screen never changes and you verify the result elsewhere |
| Look, act | Moved and new layouts | Observation time, wrong clicks go unnoticed | A wrong result is cheap to spot later |
| Look, act, check | 0 silent failures in these 50 simulated runs | About 6x the simulated time, false alarms | A wrong "done" costs money or trust |

## Common mistakes {#common-mistakes}

| Mistake | Do this instead |
| --- | --- |
| Trusting the agent's own "done" | Check the expected change on screen, or better, the saved state |
| Retrying a toggle blindly | Look first; retry only if the goal is still missing |
| Unlimited retries | A fixed budget per step, then a clear failure report so a person can take over |
| Testing only on the recorded layout | Move things on purpose, like the 50 moved runs here |

## Practice {#practice}

**Q (Beginner): Why did the replay bot report done in all 48 runs it failed?**
It never observes the page, so it cannot tell its click hit empty space.

**Q (Intermediate): Why does the checking bot look before retrying?**
The step may already be done; clicking a toggle again would undo it.

**Q (Intermediate): Why is looking without checking 6/10 on an unchanged page?**
Injected observation errors cause wrong clicks it never notices.

**Q (Advanced): What would you check instead of the screen for Save?**
The stored setting itself, through an API or a reload, which avoids vision errors.

**Q (Advanced): How would you cut the 6x cost?**
Check only steps whose failure matters, and use cheap signals such as a URL change before a full screenshot.

## Further reading {#further-reading}

- The computer use tool docs describe the screenshot and action loop from the provider's side.
- [OSWorld](https://arxiv.org/abs/2404.07972) evaluates agents on real desktop tasks using execution-based checks; it does not validate our assumed observation-error rates.
- Playwright's locator guide explains why test tools find elements by role and text.
- The Builders' Library article on retries covers limits and backoff.
- Our lab: run it, change the error rates, and watch the table move. For a second model acting as the check, see [one AI writes, another reviews](/one-ai-writes-one-ai-reviews/).

Our lab is a small simulation: no AI model runs, nothing takes a screenshot, and the error rates and time costs are our assumptions, so the results hold for this simulation only.

## Frequently asked questions

**Can an AI agent still finish a task if a button moves?**
In our simulation, a bot that looked before each action finished 86% of moved runs and one that also checked each action finished 94%. This simulation cannot establish a real AI agent's success rate.

**Why does a recorded macro fail silently?**
It never looks at the screen, so it cannot tell that its click landed on empty space. Our replay bot reported done in all 48 runs it failed.

**Is checking after every click worth the cost?**
When a silent failure is expensive, often yes. Our checking bot took about 6 times the simulated time of replay and had 0 silent failures in the 50 moved runs, under our assumed costs.

## Sources

- [Computer use tool (screenshot and action loop), Claude platform docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool)
- [OSWorld: benchmarking multimodal agents on real computer tasks (Xie et al., 2024)](https://arxiv.org/abs/2404.07972)
- [Playwright locators: find elements by role and text, not position](https://playwright.dev/docs/locators)
- [Timeouts, retries and backoff with jitter (Amazon Builders' Library)](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/)