LiteLLM got hijacked again: how to avoid losing SSH keys and API secrets through your AI gateway in production

I’m Denis Shokhirev, an Agentic AI Systems Architect based in Freiburg im Breisgau, Germany, and at DennisCraft AI Studio I ship production AI systems for DACH B2B clients using Claude, Supabase, n8n, Doppler, and self-hosted Postgres. The ugly production moment is not usually model latency or a bad completion. It’s realizing your AI gateway can see SSH keys, .env files, service tokens, and internal API secrets it never needed in the first place.

That is why another LiteLLM compromise matters. An AI gateway is rarely just a routing layer once it reaches production. It becomes the concentration point for model credentials, request logs, tool outputs, callback URLs, internal metadata, and sometimes file access through agent tooling. If that layer gets exposed, the incident is no longer “an LLM problem.” It is a standard infrastructure breach with an LLM-sized blast radius.

The real issue is privilege concentration, not one library name

Teams adopt gateways because the operational case is obvious: one endpoint, provider failover, usage accounting, model abstraction, centralized logging. Those are real benefits. The problem is that convenience quietly turns the gateway into a trust sink. The same process often ends up holding Anthropic keys, OpenAI-compatible credentials, tracing payloads, automation hooks, and enough context to reconstruct sensitive business workflows.

OWASP’s Top 10 for LLM Applications 2025 calls out Sensitive Information Disclosure and Excessive Agency as core risks in LLM systems, not edge cases. Source: OWASP GenAI Project, 2025, https://genai.owasp.org/. In practice, that risk shows up in very boring ways: full request logging, tools with filesystem access, inherited environment variables, and developers passing “just in case” context into prompts that later lands in traces.

Where secrets usually leak

Design as if the gateway will eventually be compromised

The useful design question is not “is this gateway secure enough?” It is: if this process is compromised for 15 minutes, what can an attacker actually reach? If the answer is “all model keys, production database access, and deployment secrets,” the architecture already failed before any specific CVE enters the story.

1. Separate secret access from tool execution

If your agents can run shell commands, read files, call webhooks, or trigger automations, that runtime should not sit next to long-lived secrets. I prefer issuing the smallest possible credentials to the smallest possible process, with short TTL where feasible. The point is to reduce lateral movement after compromise.

#!/usr/bin/env bash
set -euo pipefail

export APP_ENV=production
export ANTHROPIC_API_KEY="$(doppler secrets get ANTHROPIC_API_KEY --plain)"
export DB_URL="$(doppler secrets get READONLY_DB_URL --plain)"

exec env -i \
  PATH="/usr/bin:/bin" \
  APP_ENV="$APP_ENV" \
  ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
  DB_URL="$DB_URL" \
  python3 worker.py

The important pattern here is not Doppler specifically. It is the clean environment via env -i, a read-only database URL, and no inherited shell state from a developer laptop or CI runner.

2. Remove access to home directories and local credentials

A lot of teams think they are safe because a tool is “only reading the repo.” That misses how many secrets live outside the repository: ~/.ssh, ~/.aws, Docker credentials, Git credentials, editor tokens, and random copied .env files. If a tool can resolve arbitrary paths, that is enough for a bad day.

services:
  agent-runner:
    image: python:3.12-slim
    working_dir: /app
    volumes:
      - ./workspace:/app:ro
    read_only: true
    tmpfs:
      - /tmp
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL

This is not perfect isolation, but it removes a common failure mode immediately: the agent cannot wander through a developer’s home directory just because a tool call asked for “related config files.”

What to enforce in CI before production

Gateway incidents get worse when the surrounding codebase is already permissive. That is why I still start with plain static analysis instead of pretending AI needs completely new security mechanics. Mature tools already catch a lot of self-inflicted damage.

#!/usr/bin/env bash
set -euo pipefail

gitleaks detect --source . --no-git --exit-code 1
bandit -r . -x tests
semgrep --config auto .

echo "Static checks passed"

gitleaks catches committed tokens and keys, bandit finds risky Python patterns, and semgrep covers a broad range of rules including shell execution, path traversal, and unsafe input handling. This is cheap, fast, and production-friendly.

Anthropic’s documentation on tool use and the Anthropic SDK examples are useful, but they are not a production isolation model by themselves. Source: Anthropic documentation, 2025, https://docs.anthropic.com/. The same is true for the OpenAI Cookbook: excellent for implementation guidance, not a guarantee that your runtime boundaries are correct. Source: OpenAI Cookbook, 2025, https://cookbook.openai.com/.

Minimum policy for any AI gateway

• Disable full request and response body logging by default.
• Redact Authorization headers, API keys, and tool output fields likely to contain secrets.
• Physically separate staging and production credentials.
• Restrict egress from the tool runner to approved destinations only.
• Make secret rotation possible without a full platform rebuild.

A safer production pattern for multi-agent systems

In my DACH client systems, I try to keep the gateway intentionally boring. It should know how to talk to model providers and little else. Database access should go through a narrow backend with scoped queries, not through a broad service credential embedded in the agent context. Supabase is productive, but dropping a service_role key into an agent pipeline is usually an architectural mistake.

from anthropic import Anthropic
import os

client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

message = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=512,
    system="You may call only approved tools. Never request secrets.",
    messages=[{"role": "user", "content": "Summarize ticket status"}]
)

print(message.content)

The important thing is what is missing here: no SSH material, no database superuser, no general filesystem tool by default. Anything privileged sits behind a separate service boundary with auditability.

FAQ

Should I stop using LiteLLM entirely?

Not necessarily. The bigger problem is usually over-privilege around the gateway, not the gateway concept itself. If the process is isolated, minimally logged, and cannot see long-lived secrets, the blast radius is much smaller.

Are environment variables still acceptable for API keys?

Yes, with constraints. They are fine for short-lived processes if you control logging, crash dumps, child process inheritance, and access to runtime inspection on the host.

Do I need a separate container for tool execution?

If the agent can access files, shell, or the network, yes. Even basic container isolation with read-only mounts and dropped capabilities cuts off several very common failure paths.

What is the first check I should run this week?

Inspect exactly which secrets and paths the gateway process can access, then disable full payload logging. That usually exposes more unnecessary privilege than teams expect.

What do I do after a suspected compromise?

Assume every secret visible to the gateway is exposed: rotate keys, review logs, audit tool permissions, and inspect outbound traffic. If that process takes days instead of hours, your incident path needs work.

Which stage in your LLM pipeline catches the most issues in production right now: static analysis, runtime isolation, egress control, or human review of privileged tools? I’d genuinely like to know because that answer usually reveals where the real architecture risk sits.

I run a free 30-min stack audit for DACH founders building AI in regulated markets. DM me on LinkedIn or write to @ger_dennis_ai.