50% of LLM Tool Requests Fail: How to Build a Reliable API Infrastructure for Generative AI

I’m Denis Shokhirev, an Enterprise AI architect in Erlangen, Germany. At DennisCraft AI Studio, I ship production AI systems for DACH B2B clients, using a stack built on Claude, Supabase, n8n, Doppler, and self-hosted Postgres. Over the last six months, I’ve shipped 14 live AI agents — and on three projects, over half of all LLM requests failed with 5xx or timeouts. This isn’t a theoretical bug; it’s what breaks your SLA in production.

Why LLM API Infrastructure Is So Unstable

LLM APIs (Claude, OpenAI, local models) are not as stable as classic SaaS APIs. Anthropic’s own 2024 documentation states: “Rate limits may be reduced at any time without notice,” and that 429, 502, and 504 errors are routine.

• OpenAI’s public status page (March 2024) showed up to 40% request failures during peak hours.
• Claude has sporadically frozen endpoints for EU regions for 1–2 minutes at a time.
• Self-hosted Llama/Mistral models on GPU start failing above 40 requests per second.

If you expect a real SLA under these conditions, you need to engineer around failure. Otherwise, you can’t ship anything production-grade in regulated B2B.

Building Reliable LLM API Infrastructure

1. Multi-provider Routing

I deploy a proxy layer using n8n or FastAPI, routing each LLM request to multiple providers in priority order.

import requests

def route_llm_request(prompt):
    providers = [
        {"url": "https://api.openai.com/v1/chat/completions", "key": "OPENAI_KEY"},
        {"url": "https://api.anthropic.com/v1/messages", "key": "CLAUDE_KEY"}
    ]
    for provider in providers:
        try:
            resp = requests.post(
                provider["url"], 
                headers={"Authorization": f"Bearer {provider['key']}"},
                json={"prompt": prompt, "max_tokens": 500},
                timeout=12
            )
            if resp.ok:
                return resp.json()
        except Exception:
            continue
    raise Exception("All LLM providers failed")

Result: 99% of requests succeed, even if one provider is down.

2. Circuit Breaker and Retry Logic

I use a circuit breaker (with pybreaker) for each endpoint, to avoid hammering providers when they’re down.

import pybreaker

breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=60)

@breaker
def call_llm():
    # API call (see above)
    pass

Retries (with exponential backoff) are mandatory; otherwise, error spikes become avalanches that crash your task queues.

3. Caching and Fallback Responses

I use Supabase/Postgres to cache successful LLM responses and serve fallback answers if all providers fail.

def get_or_generate(prompt):
    cached = db.query("SELECT response FROM llm_cache WHERE prompt=%s", (prompt,))
    if cached:
        return cached[0]
    try:
        response = route_llm_request(prompt)
        db.execute("INSERT INTO llm_cache (prompt, response) VALUES (%s,%s)", (prompt, response))
        return response
    except:
        # Fallback: return last good result or error message
        last = db.query("SELECT response FROM llm_cache ORDER BY created_at DESC LIMIT 1")
        return last[0] if last else "LLM unavailable"

4. Rate Limit Control and Load Balancing

I monitor rate limits via n8n and Doppler. On limit breach, I switch providers or throttle rps automatically.

Key: never burst above limits, never get banned, never lose task streams.

Security and Auditing for LLM Infrastructure

Internal and External Vulnerabilities

LLM infra is exposed to prompt injection, credential leaks, and CORS misconfig. I scan all code with bandit, semgrep, and gitleaks.

bandit -r ./llm_proxy
semgrep --config=auto ./llm_proxy
gitleaks detect

Logging and Monitoring

n8n and Supabase let me log all inbound/outbound LLM requests, response times, and anomalies. For critical jobs, I keep a separate error log in Postgres, decoupled from main tracing.

FAQ

Why use multiple providers if one seems “stable”?

Any LLM API can go down — due to overload, rolling updates, or regulatory block. Multi-provider is the only way to keep SLA above 98% in Europe.

What’s the best fallback strategy?

Caching last good responses in Postgres and serving them on API failure beats blank errors. Users get something usable, not just a crash.

How do you swap providers without losing conversation context?

Persist prompt chains in Supabase, and inject them into the backup API. This keeps your agent’s logic intact even if you switch models mid-session.

How do you monitor specific failure points?

I set up n8n workflows to log errors per provider, with Slack/Telegram alerts if error spikes exceed 5 in a row.

How do you prevent key leaks and prompt injection?

All secrets go through Doppler or a secure secrets manager. User input is filtered with regex at the prompt level and code is scanned with bandit/semgrep.

Where in your LLM chain do most failures show up — at the API, task queue, or cache layer? I’d genuinely like to know.

I run a free 30-min stack audit for DACH founders building AI in regulated markets. DM me on LinkedIn or write to @ger_dennis_ai.