AI agents break at the UI boundary: how to give coding agents access to Android, iOS, web, and desktop without custom glue code

I’m Denis Shokhirev, an Agentic AI Systems Architect in Freiburg im Breisgau, Germany, and at DennisCraft AI Studio I ship AI systems for DACH B2B clients on Claude, Supabase, n8n, Doppler, and self-hosted Postgres. In production, the same failure keeps showing up: the coding agent writes decent backend code, patches SQL, and wires APIs, then hits the UI boundary and falls apart because nobody gave it a stable way to operate Android, iOS, the browser, or the desktop.

This is where many “agent” projects quietly turn into shell scripts with a prompt attached. Teams expose Playwright for one flow, adb for another, a folder of screenshots for a third, and a few manual steps for whatever did not fit. The result is not an autonomous system. It is custom glue code that becomes expensive the moment you need repeatability, incident review, or regulated-client evidence.

The actual problem is not model quality

Most teams frame this as a model problem: “the agent is not good enough at UI tasks.” Usually the model is not the main issue. The issue is missing control architecture. A coding agent needs a constrained way to observe state, execute an action, collect artifacts, and feed those artifacts back into the planning loop. If you do not provide that, the model improvises around incomplete context.

That gap gets wider across four surfaces:

What works is not “give the model more freedom.” What works is reducing freedom and making each action observable.

Use one action contract across all UI surfaces

Do not let the agent invent shell commands on the fly

The cleanest pattern I have found is to expose a narrow action layer rather than raw environment access. The agent should call tools like observe, act, artifact, and state, while adapters translate those requests into Playwright, adb, simulator commands, or desktop automation.

import subprocess
from typing import TypedDict

class ToolResult(TypedDict):
    ok: bool
    stdout: str
    stderr: str

def adb_tap(serial: str, x: int, y: int) -> ToolResult:
    cmd = ["adb", "-s", serial, "shell", "input", "tap", str(x), str(y)]
    proc = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
    return {"ok": proc.returncode == 0, "stdout": proc.stdout.strip(), "stderr": proc.stderr.strip()}

def adb_logcat(serial: str, lines: int = 100) -> ToolResult:
    cmd = ["adb", "-s", serial, "logcat", "-d", "-t", str(lines)]
    proc = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
    return {"ok": proc.returncode == 0, "stdout": proc.stdout, "stderr": proc.stderr.strip()}

The point here is not that adb is special. The point is that the agent is not free-writing terminal commands every time it wants to touch a device. The action layer enforces timeouts, logs every call, and returns predictable data.

Keep one event log for screenshots, DOM, logs, and traces

Another common mistake is splitting evidence across five systems. Screenshots go to object storage, DOM snapshots sit in temp folders, logs stay on the worker, and humans investigate by scrolling Slack threads. That setup breaks the moment a client asks what happened during a failed run.

create table agent_ui_events (
  id bigserial primary key,
  run_id uuid not null,
  step_index int not null,
  surface text not null,
  action text not null,
  status text not null,
  artifact_path text,
  log_excerpt text,
  created_at timestamptz default now()
);

I prefer a single event table in Postgres, with artifact references stored next to each step. That gives you replayability, auditability, and a clean handoff from the agent to a human operator.

Security controls matter more here than in pure code generation

Once an agent can click, type, and navigate, it can do damage faster than a code-only workflow. OWASP’s LLM Top 10 for 2025 explicitly calls out prompt injection and excessive agency as material risks in agentic systems: https://genai.owasp.org/llm-top-10/. At the UI layer, those risks become operational, not just theoretical.

Anthropic’s documentation on tool use and computer use also pushes the same direction: tools should be constrained, observable, and scoped to the task rather than broadly attached to a production environment. That is the right default if you build around Claude Code or the Anthropic SDK.

I usually put four guardrails in place:

• an allowlist of devices, domains, and applications;
• step limits and hard timeouts per session;
• mandatory artifacts before and after important actions;
• static analysis on generated code before execution.

#!/usr/bin/env bash
set -euo pipefail

bandit -r ./generated_code -f json -o bandit-report.json
gitleaks detect --source ./generated_code --report-format json --report-path gitleaks-report.json
semgrep --config auto ./generated_code --json --output semgrep-report.json

echo "checks finished"

None of these tools makes the system safe by itself. Together, they remove obvious failure modes before the agent gets execution privileges.

A practical architecture for Android, iOS, web, and desktop

Build one control plane and four adapters

You do not need four independent automation stacks. You need one control plane that receives tasks, stores state, writes events, and exposes a normalized tool catalog to the agent. Under that layer, adapters execute surface-specific commands.

tools:
  - name: browser_open
    surface: web
    requires_artifact: true
  - name: android_tap
    surface: android
    requires_artifact: true
  - name: ios_screenshot
    surface: ios
    requires_artifact: true
  - name: desktop_type
    surface: desktop
    requires_artifact: true

The transport can vary. Web can run through Playwright. Android through emulators or physical devices over adb. iOS through simulators and system tooling. Desktop through controlled virtual machines. The important part is that the agent always sees the same capability grammar and the same response format.

This also cuts maintenance cost. When a new surface appears, you add or replace an adapter rather than rewriting orchestration logic. If you already use n8n as your orchestration layer, this maps cleanly to an event-driven loop: tool call, adapter execution, artifact save, planner retry, human escalation if confidence drops.

FAQ

Is DOM access enough for UI-capable agents?

For some web tasks, yes. For mobile and desktop, usually no. You need screenshots or accessibility-tree data, otherwise the agent loses too much context.

Can Playwright cover everything?

No. It is excellent for web automation, but it does not replace Android, iOS, or desktop-specific control layers. Those need their own adapters and state handling.

Why not just expose a shell and log everything?

Because logging does not fix uncontrolled agency. A constrained tool layer is easier to audit, easier to test, and much easier to explain to enterprise clients.

What should I store first: screenshots or logs?

Store both. A screenshot tells you what the agent saw; logs tell you what the system did. You need both to debug production failures quickly.

Where should a small team start?

Start with one surface, usually web or Android, and define a stable tool result schema. Once event logging and artifact storage are clean, expanding to iOS and desktop becomes much easier.

If you already run coding agents in production, which stage breaks most often in your stack today: browser session state, Android device control, iOS simulator orchestration, desktop interaction, or artifact feedback into the planner? I’d genuinely like to know where your highest-cost failure point sits right now.

I run a free 30-min stack audit for DACH founders building AI in regulated markets. DM me on LinkedIn or write to @ger_dennis_ai.