Why Your AI Agents Break Prod: How to Locally Capture and Use Agent Failures for Regression and Security

I'm Denis Shokhirev, an Enterprise AI architect based in Erlangen. At DennisCraft AI Studio, I ship production AI systems for DACH B2B clients on a stack of Claude, Supabase, n8n, Doppler, and self-hosted Postgres. Last quarter, I shipped 14 agents — and in at least 4 cases, I watched a “green” test suite pass, only to have a real-world prompt or edge-case input break a live integration in a client workflow. No runtime error, no stack trace — just a silent agent failure that nobody captured.

Why Local Error Logging Beats Cloud Black Boxes

Many teams rely on vendor dashboards or cloud logs, hoping they’ll catch everything. In reality, most LLM agent failures don’t trigger exceptions. Instead, the agent “thinks wrong” — incorrect function call, hallucinated API parameter, or a silent refusal to act. Unless you’re collecting agent-level failure data locally, you’re blind to 80% of failure modes. On my last logistics deployment, a missed edge case in a Claude-generated SQL query cost the client a full day of manual reconciliation.

Production-Grade Logging: What’s Actually Needed

How to Capture Agent Failures Locally

Forget “just let the API log it.” For real regression and security, you need a local capture layer that logs:

• Agent ID and invocation context
• Input prompt and all parameters
• Raw output (including intermediate states)
• Error messages and stack traces (if any)
• Timestamped, auditable records

I use Postgres (self-hosted or via Supabase) as the central log store. Each agent or orchestrator (n8n workflow) sends structured error events to a persistent table. All PII is masked or hashed before logging — both for GDPR and internal compliance.

Sample Table for Agent Failures

CREATE TABLE agent_failures (
    id SERIAL PRIMARY KEY,
    agent_id TEXT NOT NULL,
    input JSONB NOT NULL,
    output JSONB,
    error_message TEXT,
    stack_trace TEXT,
    intermediate_state JSONB,
    created_at TIMESTAMP DEFAULT now()
);

This pattern is agent-agnostic: it works for Claude, GPT-4, or your own code. Just send every “unexpected” agent outcome, not only hard exceptions.

Implementation on Claude, Supabase, n8n Stack

n8n as Orchestrator

In every production workflow, I add a “Failure Logging” step after any agent call or critical function. This step writes all failure context to Supabase/Postgres in real time.

const { createClient } = require('@supabase/supabase-js');
const supabase = createClient(process.env.SUPABASE_URL, process.env.SUPABASE_KEY);

async function logFailure(agentId, input, output, errorMsg, stackTrace, intermediate) {
    await supabase
        .from('agent_failures')
        .insert([{
            agent_id: agentId,
            input: input,
            output: output,
            error_message: errorMsg,
            stack_trace: stackTrace,
            intermediate_state: intermediate
        }]);
}

I use the same method for Claude Code or Anthropic API agents — the logging call is external, not tangled in agent logic.

PII Masking and Compliance

Never log raw emails, names, or sensitive data. At minimum, hash user emails/IDs, redact names, and flag columns containing PII. Use tools like semgrep or bandit for pre-commit checks against accidental leaks. For EU clients, double-check GDPR and local data retention policies.

Using Captured Failures for Regression and Security

Locally stored agent failures aren’t “just logs.” I actively use them to:

• Build a regression suite: every new agent release is run against 10–20 real-world failure cases
• Test for security bugs: search for SQL injection, prompt injection, and token leaks (using semgrep, gitleaks, bandit)
• Track error trends: dashboards in Supabase or Grafana reveal which agents degrade fastest

Python Example: Automated Failure Replay

import requests

def replay_failure(agent_url, failure):
    response = requests.post(agent_url, json=failure['input'])
    assert response.status_code == 200
    # Optionally, check for expected output

# Fetch last 10 failures from Postgres
failures = get_failures_from_db(limit=10)
for f in failures:
    replay_failure('https://your-agent/api/invoke', f)

On my last three projects, 60% of critical bugs were only reproducible by replaying real failure cases — not synthetic test data.

FAQ

Do I need to keep all failures forever?

No. I recommend 3–6 months retention, then archive or anonymize. Some sectors (fintech, healthcare) require stricter policies — check your domain rules.

How do I retrofit this into an existing system?

Add a middleware or decorator that logs agent errors to a dedicated table. Don’t mix error logging into core agent logic — keep it orthogonal and maintainable.

How is this different from Sentry/New Relic?

Sentry is great for code exceptions, but it can’t see agent-level logic failures or prompt issues. You need a layer that logs prompt, inputs, and agent “thought process.”

What about false positives?

Tag failures manually at first. As you scale, semi-automate with bandit or semgrep to filter out non-critical errors. Build a simple UI for annotation if needed.

How do I secure the failure log?

Restrict access, encrypt storage, and run regular audits with gitleaks. Set up alerts for suspicious insert patterns or PII leakage.

Which stage in your LLM pipeline catches the most issues in prod — static analysis, runtime sandbox, or human review? I run a free 30-min stack audit for DACH founders building AI in regulated markets. DM me on LinkedIn or write to @ger_dennis_ai.