Your agent breaks production not because it hallucinates, but because it cannot see the UI: why runtime perception for web and desktop is becoming mandatory

I’m Denis Shokhirev, an Agentic AI Systems Architect based in Freiburg im Breisgau, Germany, and at DennisCraft AI Studio I ship AI systems for DACH B2B clients on Claude, Supabase, n8n, Doppler, and self-hosted Postgres. The ugliest production failures I see are usually not “the model made something up,” but “the agent clicked the wrong thing because the UI changed, a modal blocked the page, or the desktop window lost focus and the agent had no current view of reality.”

That distinction matters because teams often diagnose the wrong problem. They call it hallucination, tune prompts for two weeks, and leave the actual failure mode untouched: the agent is acting on stale assumptions about a live interface. If your agent touches web or desktop software in production and its action loop does not include current-state observation, you do not have an intelligence problem first. You have an instrumentation problem.

The failure is often missing runtime state, not model confusion

“Hallucination” is a convenient label because it shifts blame to the model. In many production incidents, the model is not inventing facts. It is following instructions against an environment it cannot actually observe at decision time. It was told to click “Approve,” but the button moved, became disabled, changed language, or is now behind a consent banner. From the system’s perspective, that is a visibility failure.

What runtime perception actually means

I am not describing a branded platform or a speculative research idea. I mean a production pattern where the agent receives current evidence from the interface before taking a critical action: screenshot, accessibility tree, DOM snapshot, OCR output, or a combination. The agent then maps its intended action to the live state of the application, not to a remembered or assumed state.

Anthropic’s computer use guidance is built around an observe-reason-act loop, which is the right mental model for UI agents. If an interface is the execution surface, perception must be part of execution, not an optional extra. See Anthropic documentation for computer use patterns and tool loops: https://docs.anthropic.com/. The same design logic shows up across practical agent systems: action without fresh context is blind automation.

Why web and desktop are now riskier execution surfaces

Enterprise interfaces are less stable than teams assume. React rerenders, role-based rendering, iframes, localization, A/B flags, SSO redirects, MFA prompts, and vendor-side layout changes all break simplistic selector-driven agents. On desktop, the situation is worse: VDI sessions, Citrix, legacy ERP screens, variable resolution, and focus loss create state transitions that are hard to represent through text instructions alone.

This does not mean “use perception everywhere.” My bias is still API first, then event-driven integrations, then UI automation only where the business process is trapped in the interface. But if the interface is in scope, runtime perception becomes a production safety layer, not a nice-to-have.

A production pattern that holds up better

The design I trust is not “give the agent more freedom.” It is a constrained action loop with verification gates. The minimum viable pattern is: acquire current UI state, state the goal of the next step, limit allowed actions, execute one action, verify the resulting state, and stop or escalate if the result is ambiguous. A confident wrong click is still a wrong click.

Example: screen classification before action

from anthropic import Anthropic
import base64
import os

client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

with open("screen.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode("utf-8")

msg = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=300,
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Identify whether the payment approval button is visible and enabled. If not, return BLOCKED with the reason."},
            {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": image_b64}}
        ]
    }]
)

print(msg.content[0].text)

The important part is not the SDK call. It is the discipline around it. The agent does not immediately act. It first classifies the current screen. If the result is BLOCKED, the workflow branches: request a missing field, capture another screen, or escalate to a human operator.

Verification before writing to stateful systems

type UiDecision = {
  action: "click" | "type" | "blocked";
  target?: string;
  reason: string;
};

function verifyDecision(decision: UiDecision) {
  if (decision.action === "blocked") throw new Error(decision.reason);
  if (!decision.target || decision.target.length < 3) {
    throw new Error("Unverified UI target");
  }
  return decision;
}

const approved = verifyDecision({
  action: "click",
  target: "button[aria-label='Approve payment']",
  reason: "Control is visible and enabled"
});

This kind of guard does not solve every error class. It does make the failure explicit and observable instead of silently letting the agent proceed into the wrong context.

Why this matters for security and regulated operations

When an agent works inside a regulated workflow, UI mistakes are not just UX bugs. They can become approval errors, access control mistakes, or data written under the wrong customer context. OWASP’s Top 10 for LLM Applications highlights excessive agency and insufficient action controls as real risk categories, not theoretical ones: https://owasp.org/www-project-top-10-for-large-language-model-applications/.

The most common failure modes I see are boring and expensive:

• the agent clicks a visually similar control after a layout change;
• a modal blocks the page while old DOM references still exist;
• desktop focus shifts and keystrokes go to the wrong window;
• SSO or MFA introduces an unplanned step the workflow never modeled;
• localization changes labels and text matching hits the wrong target.

None of these are solved by “better prompting” alone. They are solved by giving the agent a way to inspect reality at the moment of action and by forcing explicit stop conditions when reality does not match plan.

What teams should do next

If you already have an agent touching web or desktop software, ask five blunt questions. What exactly does it observe before a critical action? How do you verify state after the action? What happens when the screen is unfamiliar? Can you replay the path through logs and screenshots? Where can a human safely interrupt the loop?

If two of those answers are vague, your main issue is probably not prompt quality. It is that the agent is operating half-blind. That is an architecture issue, and architecture issues do not disappear with another system prompt rewrite.

FAQ

Is API integration still preferable to UI automation?

Yes. If the API fully covers the process, use the API. Runtime perception matters when key business actions exist only in the UI or when legacy and partner systems do not expose what you need programmatically.

Is a DOM snapshot enough without screenshots?

Often no. DOM does not reliably capture overlays, visual visibility, disabled state after render, or window focus. For critical steps, combining DOM with visual evidence is safer.

Does runtime perception make agents too slow or too expensive?

It adds cost per critical step, but that tradeoff is usually favorable compared to production incidents. You do not need it on every click, only on high-risk transitions and approvals.

Does this apply to desktop and VDI environments?

Absolutely. In desktop environments it matters even more because APIs are often absent and focus, resolution, and session state directly affect action correctness.

Can you remove humans from the loop entirely?

Sometimes, if the action space is narrow and the verification logic is strong. For money movement, access changes, or personal data workflows, I still prefer explicit escalation paths.

Which stage in your LLM pipeline causes more production trouble right now: planning, tool invocation, or the moment the agent actually touches a live interface? I’d genuinely like to compare that with what I’m seeing across DACH B2B systems.

I run a free 30-min stack audit for DACH founders building AI in regulated markets. DM me on LinkedIn or write to @ger_dennis_ai.