About Portfolio Cases Services Blog Contact 🎙 Talk to AI
EN DE RU
🎙 Talk to AI
November 21, 2026 · 5 min read

Your AI agent breaks on mobile UI: how to connect iOS and Android via MCP and run real workflows instead of screenshot demos

I’m Denis Shokhirev, an Agentic AI Systems Architect based in Freiburg im Breisgau, Germany, and at DennisCraft AI Studio I ship production AI systems for DACH B2B clients on Claude, Supabase, n8n, Doppler, and self-hosted Postgres. The failure pattern I keep seeing is simple: the agent looks smart on desktop, then falls apart the moment the workflow moves into a real mobile app with permissions, keyboards, deep links, and state the model cannot actually control. This is why screenshot demos ar

Denis Shokhirev
Denis Shokhirev
Agentic AI Systems Architect
Telegram LinkedIn

I’m Denis Shokhirev, an Agentic AI Systems Architect based in Freiburg im Breisgau, Germany, and at DennisCraft AI Studio I ship production AI systems for DACH B2B clients on Claude, Supabase, n8n, Doppler, and self-hosted Postgres. The failure pattern I keep seeing is simple: the agent looks smart on desktop, then falls apart the moment the workflow moves into a real mobile app with permissions, keyboards, deep links, and state the model cannot actually control.

This is why screenshot demos are misleading. A model can describe a screen, infer the likely next tap, and still fail the real task because mobile execution is not the same thing as mobile observation. If you want an agent to operate on iOS or Android in a way that survives production conditions, the device has to be exposed as a controlled tool surface, not as a sequence of images pasted into a chat.

Why mobile breaks agents that looked fine on web

On desktop web you usually have a URL, a DOM, test selectors, browser logs, and mature automation layers. On mobile you have a different control model: accessibility trees, native widgets, system dialogs, app state, overlays, keyboards, network transitions, and frequent context switches between app, webview, and OS. A model that only “sees” pixels has very little grounding in that environment.

OWASP’s Mobile Application Security Testing Guide and MASVS are a useful anchor here because they document how much sensitive behavior lives outside the visible app surface: local storage, inter-app communication, auth state, transport, and permissions. For agent builders, the takeaway is operational rather than academic: if you cannot inspect state and execute bounded actions through a stable interface, you are not testing a mobile workflow. You are guessing at one. Source: OWASP MASVS/MASTG, current project materials at https://owasp.org.

The first things that usually fail

  • OTP or push approval in a second app
  • Camera, notifications, location, or biometric permission prompts
  • Soft keyboard covering the primary CTA
  • Deep links that open the wrong app state
  • Network changes that trigger retries or stale sessions

What “connect mobile to MCP” actually means

I’m using MCP here as an interface pattern: the model gets a standard set of tools to read state and take actions. For mobile, that usually means four categories of tools: inspect the current UI hierarchy, capture a screenshot, execute an interaction, and collect logs or artifacts. Appium is a practical automation layer for both iOS and Android, and an MCP server can sit in front of it as a thin control plane that exposes only the actions you want the model to use.

The important part is not branding. The important part is constraint. Do not give the agent raw shell access to a device. Give it a tool surface with bounded verbs: tap by accessibility id, swipe, input text, wait for element, open deep link, fetch page source, capture screenshot. That moves the system from “vision-only improvisation” to “stateful execution with auditability.” Anthropic’s documentation on tool use follows the same principle: models do better when the environment provides clear, explicit actions and external state handling. Source: Anthropic docs, https://docs.anthropic.com.

from appium import webdriver
from appium.options.ios import XCUITestOptions

options = XCUITestOptions().load_capabilities({
    "platformName": "iOS",
    "appium:deviceName": "iPhone 15",
    "appium:platformVersion": "17.5",
    "appium:automationName": "XCUITest",
    "appium:bundleId": "com.example.app",
    "appium:noReset": True
})

driver = webdriver.Remote("http://127.0.0.1:4723", options=options)
print(driver.page_source[:1000])
driver.quit()

That snippet matters because it shows the right abstraction level. The agent should not reason from screenshots alone if your goal is reliable execution. It needs access to inspectable state.

A minimal production architecture for mobile agent workflows

A setup I trust in production is straightforward: orchestration in n8n or a Python service, model calls through the Anthropic SDK, state and audit data in Postgres or Supabase, secrets in Doppler, and device execution through Appium on simulators, emulators, or real devices. MCP is not the center of the system. It is the contract between the model and the execution environment.

LayerToolRole
Model orchestrationAnthropic SDK / Claude CodePlanning, tool selection, retries, error handling
Workflow runnern8nScheduling, webhooks, alerts, branching
State storePostgres / SupabaseRuns, steps, artifacts, audit trail
SecretsDopplerEnvironment variables, test credentials, separation by stage
Mobile automationAppiumUI inspection and action execution on iOS/Android

Why step logging matters more than a final pass/fail

The biggest operational mistake is storing only the verdict. A mobile agent loop needs step-level logs: which tool was called, what target it used, whether the element existed, what screenshot was taken, what the page source looked like, and what exception came back from the driver. Without that, every failure gets flattened into “the model was wrong,” which is often false. In practice, many mobile failures are caused by app state, expired sessions, transient rendering delays, or unstable selectors.

type StepLog = {
  run_id: string;
  step: number;
  action: string;
  target?: string;
  status: "ok" | "error";
  screenshot_path?: string;
  error_message?: string;
};

async function saveStep(db, log: StepLog) {
  await db.query(
    `insert into agent_step_logs
     (run_id, step, action, target, status, screenshot_path, error_message)
     values ($1,$2,$3,$4,$5,$6,$7)`,
    [log.run_id, log.step, log.action, log.target, log.status, log.screenshot_path, log.error_message]
  );
}

iOS and Android fail in different ways

Android usually gives you more flexibility in automation and device coverage, but you pay for that with fragmentation. OEM differences, DPI variation, OS versions, and WebView behavior introduce more edge cases. iOS is more uniform, but system restrictions and app sandboxing require tighter scenario control, especially around permissions, app switching, and biometric flows.

If you operate in regulated environments, separate functional mobile automation from production operations. The agent that validates onboarding in staging should not have the same credentials, artifact retention, or approval path as a system that can trigger real user-impacting actions. This is less about theory and more about avoiding accidental damage from one repeated tap, one stale session, or one unredacted screenshot containing personal data.

What to verify before your first real run

  • Stable accessibility labels or content descriptions exist
  • The flow is repeatable on at least 3 device profiles
  • Artifacts are stored with retention rules and access control
  • Screenshots are redacted if they may include PII
  • Allowed actions are constrained at the MCP tool layer

Security and policy control beat “autonomy” every time

For teams building in Europe, this also lines up with compliance reality. If screenshots, logs, and session traces contain personal data, you have a GDPR issue, not just a test artifact issue. I strongly recommend artifact TTLs, redaction for screenshots, stage-separated secrets, and explicit approval boundaries for sensitive actions such as payments, account deletion, or KYC steps.

If you want a concrete engineering reference for tool design, the OpenAI Cookbook and Anthropic docs both push the same pattern: explicit tools, explicit schemas, and externalized control over side effects. Different vendors, same operational lesson. Source: OpenAI Cookbook, https://cookbook.openai.com and Anthropic docs, https://docs.anthropic.com.

export APPIUM_HOST=http://127.0.0.1:4723
export DEVICE_NAME="Android Emulator"
export RUN_ID=$(date +%s)

python run_mobile_flow.py \
  --run-id "$RUN_ID" \
  --platform android \
  --start-url "myapp://login" \
  --artifact-dir "./artifacts/$RUN_ID"

FAQ

Can computer vision alone replace Appium?

For demos, sometimes. For repeatable production workflows, rarely. Vision-only agents struggle as soon as the task depends on hidden state, permission dialogs, timing, or precise control over native elements.

Do I need MCP to automate mobile apps?

No. You can build the same pattern without MCP. But MCP is useful when you want one consistent contract across browser tools, database tools, and mobile tools inside the same agent loop.

Should I use real devices or emulators first?

Start with emulators and simulators for early iteration. Move to real devices once the flow includes push notifications, biometrics, variable connectivity, or device-specific behavior that affects user outcomes.

How do I deal with flaky mobile steps?

Log page source, timings, screenshots, and driver exceptions around every critical action. Then fix the tool layer and selectors before you keep editing prompts. Prompt changes are often treating the symptom, not the cause.

What is the smallest useful starting point?

One high-value workflow, one Android emulator, one iOS simulator, Appium, Postgres, and a thin MCP adapter. If you cannot get repeatability there, scaling infrastructure will not save the design.

Where does your mobile agent loop fail most often in production: state inspection, permission handling, artifact governance, or action control? I’d genuinely like to know, because that gap is where polished demos and working systems usually split apart.
I run a free 30-min stack audit for DACH founders building AI in regulated markets. DM me on LinkedIn or write to @ger_dennis_ai.

For your business

I build agents like this for businesses in Germany and the EU.

Tell me in two sentences what eats your team’s time – reply within 24 hours.

Custom AI agents (German page) →
Continue reading
How not to drown in MCP chaos: 50+ remote MCP servers already exist, but without curation your production agent becomes a security hole
One stolen key put 5,000 developers at risk: what the Cursor breach and trojanized Claude Code extension change for AI IDE security
Your customers ask ChatGPT for a tradesman. Does it name your business
Monday morning at a medical practice: what an AI phone assistant takes over and what it does not
All articles →
Where this is applied
Services — what we build
Talk to the voice agent
Case studies
Ready to build?

Turn your process into an AI system

Production quality. DACH B2B focus.

Start a project → ← All articles