Computer-use agents break on pixels: how to switch to stable UI refs via accessibility trees and stop fixing flaky desktop automation

I’m Denis Shokhirev, an Agentic AI Systems Architect in Freiburg im Breisgau, Germany, and at DennisCraft AI Studio I ship production AI systems for DACH B2B teams on Claude, Supabase, n8n, Doppler, and self-hosted Postgres. The desktop automations that hurt most in production usually don’t fail because the model is weak; they fail because the agent keeps clicking the same pixels after the UI has quietly moved.

That is the core mistake in many computer-use setups. Teams treat the screen as the interface. In practice, the screen is only a rendering. If your agent depends on screenshots, coordinates, and visual guesses as the primary control channel, you are building on the least stable layer of the system.

Why pixel control fails in real desktop workflows

Pixel-based automation works well in demos because the environment is frozen: one machine, one theme, one resolution, one language, one known window state. Production is the opposite. Windows get resized, banners appear, DPI changes, localization changes labels, and a minor UI update shifts a button by 20 pixels.

Coordinates are not references

A coordinate points to a location, not to a UI object. That sounds obvious, but many agent loops still behave as if x/y is a durable identifier. It is not. If the app renders one extra status bar or a sidebar expands, your action lands on the wrong target while the execution trace still looks “successful.”

Screenshots carry appearance, not semantics

A screenshot can show a field, but it does not tell your agent whether that field is a textbox, a disabled input, a dropdown trigger, or static text. Accessibility trees expose role, name, state, sometimes value, and parent-child relations. For automation, that is the difference between “click the gray box” and “set the Amount textbox in the Invoice window.”

Visual heuristics degrade under localization

For DACH clients this matters immediately. The same workflow may run in German and English, with different display scaling and different endpoint user settings. OCR and CV can help, but they remain a probabilistic interpretation of a rendered surface.

Anthropic’s documentation for computer use and tool use reflects this broader point: stable tool interfaces beat raw visual guessing whenever they are available. See Anthropic docs on tool use and computer use patterns: https://docs.anthropic.com/ . The same principle appears in browser automation stacks like Playwright, where role-based selectors are preferred over brittle CSS or positional selectors.

What changes when you switch to accessibility-tree references

The practical shift is simple: make the agent operate on semantic UI references first, and keep screenshots for verification, audit, or fallback. Your agent should ask for the “Save” button in the “Invoice Editor” window, not for “the blue control near the lower right corner.”

In other words, you are not trying to improve the model’s eyesight. You are changing the surface area of control so the model can act on durable objects instead of rendered guesses.

A production pattern that actually holds up

I use a thin adapter layer between the agent and the desktop app. That adapter retrieves the accessibility tree, normalizes it, resolves references, executes the action, and writes structured logs to Postgres.

1. Normalize UI elements into stable refs

from dataclasses import dataclass
from typing import Optional

@dataclass
class UIRef:
    app: str
    window: str
    role: str
    name: str
    path: str
    enabled: bool
    value: Optional[str] = None

def can_act(ref: UIRef) -> bool:
    return ref.enabled and ref.role in {
        "button", "textbox", "checkbox", "menuitem", "link"
    }

2. Resolve actions by semantics, not coordinates

def resolve_target(ax_tree: list[dict], role: str, name: str) -> dict | None:
    matches = [
        node for node in ax_tree
        if node.get("role") == role and node.get("name") == name
    ]
    enabled = [node for node in matches if node.get("enabled", True)]
    return enabled[0] if enabled else None

target = resolve_target(ax_tree, role="button", name="Save")
if target is None:
    raise RuntimeError("Target not found")
perform_click(target["path"])

3. Log resolution separately from execution

This is where many teams lose observability. “Agent failed” is not a useful incident class. You need at least three classes: reference not found, wrong element resolved, and correct element clicked but business action rejected. Once you separate those, remediation becomes much faster.

4. Keep screenshots as evidence, not as the only source of truth

Screenshots still matter. They help confirm that a dialog opened, a warning banner appeared, or a state transition is visible. They are excellent for human review and incident replay. They should not be your only addressing mechanism.

Minimal orchestration with Claude, n8n, and Postgres

A simple production loop looks like this: snapshot the accessibility tree, pass task plus UI context to the model, resolve the selected target in the live tree, execute, then persist the event stream. This pattern works well with n8n as an orchestrator and Postgres as the session store.

version: "1"
steps:
  - name: snapshot_ui
    tool: ui-adapter
    action: get_accessibility_tree
  - name: decide
    tool: claude
    input: ax_tree + task + history
  - name: resolve_ref
    tool: ui-adapter
    action: resolve_and_validate
  - name: execute
    tool: ui-adapter
    action: click_or_type
  - name: persist
    tool: postgres
    action: write_event

This also improves auditability. Instead of storing only screenshots, you can store a structured event: session ID, step number, requested intent, resolved role, resolved name, window title, result, and follow-up state. For regulated environments, that is much more useful than a folder full of PNG files with timestamps.

Where this still breaks

Accessibility trees are not magic. Some legacy desktop apps expose poor metadata. Custom controls may hide roles. Virtualized lists can mutate structure between snapshots. That is why I prefer a hybrid pattern: semantic refs as the primary selector, screenshot verification after the action, and a human handoff if expected state does not materialize after 2 or 3 attempts.

If the target app has weak accessibility support, do not compensate with more prompt tricks. Build a local normalization layer: aliases for labels, whitelisted action types, app-specific path cleanup, and state validation rules. That is cheaper than living inside endless flaky retries.

FAQ

When is pixel-based control still acceptable?

When there is no accessibility tree, no automation API, and the process is low frequency with easy manual recovery. Even then, I would keep the scope narrow and avoid unattended high-risk actions.

Is OCR a replacement for accessibility trees?

No. OCR is useful as an observation channel, especially for PDFs, images, or canvas-based apps. It is usually weaker than semantic UI metadata for action selection.

Do I need a different model to make this work?

Usually not. If the main issue is unstable target addressing, changing the control layer often gives a larger gain than changing the model.

How should I store UI refs in Postgres?

I usually store raw or normalized tree snapshots in JSONB and keep action events in a separate table with session_id, step_id, target_ref, and result. That supports replay, incident analysis, and UI version comparisons.

What if labels vary by language?

Use aliases and match on multiple signals: role, localized name, parent group, and path. In multilingual DACH workflows, single-label matching is rarely enough.

If you already run computer-use agents against desktop software, where do your incidents accumulate right now: target resolution, state verification, or post-failure debugging? I’d genuinely like to compare notes with what I see in live DACH systems. I run a free 30-min stack audit for DACH founders building AI in regulated markets. DM me on LinkedIn or write to @ger_dennis_ai.