AI Agents for Cloud Ops: Automating Incident Investigation and Remediation in Azure Without Data Leaks

I'm Denis Shokhirev, Enterprise AI architect based in Erlangen, Germany. At DennisCraft AI Studio, I deploy production AI systems for DACH clients in logistics, fintech, and industrial automation. My stack includes Claude, Supabase, n8n, Doppler, and self-hosted Postgres. Just last quarter, I caught a leaked Azure token in a workflow generated by an LLM — the only reason it never hit production was a semgrep pre-commit hook.

The Challenge: Incident Automation in Azure Without Data Leaks

Automating incident triage and remediation in Azure sounds simple: wire up runbooks, give your agent a service account, and let it act. In practice, this approach almost always grants excessive permissions and risks leaking sensitive data. I've seen agents ingest raw log streams, escalate privileges, and accidentally forward secrets to LLMs during analysis.

There are two main risks here:

• AI agents with excessive Azure privileges (Contributor, Owner) can become a single point of failure if compromised.
• Sensitive data (secrets, tokens, logs) can be exfiltrated in LLM queries, especially when using external APIs.

Microsoft's 2023 Identity Threat Landscape report attributes 86% of cloud breaches to misconfigured identity and permissions (source).

Architecture: Securely Integrating AI Agents into Azure

Production Stack

Pattern: Minimize Data Shared With LLMs

I never pass raw logs or environment variables directly to Claude or any LLM. All sensitive fields are masked with regex preprocessing before being added to prompts.

import re

def mask_secrets(log: str) -> str:
    # Hide tokens, keys, secrets (Azure, JWT, etc.)
    log = re.sub(r'(token|key|secret)\s*=\s*["\']?[\w-]+["\']?', r'\1=***', log, flags=re.IGNORECASE)
    # Mask email addresses
    log = re.sub(r'[\w\.-]+@[\w\.-]+', '[EMAIL]', log)
    return log

This preprocessing runs in every n8n module that might send data to an LLM. Static analysis is enforced in CI/CD using semgrep and bandit.

Agent Isolation and Permission Control

Service Principals With Least Privilege

Each agent uses a dedicated Azure Service Principal with scope-limited permissions. For example, an agent that restarts VMs gets only the "Virtual Machine Contributor" role, scoped to a specific resource group — never subscription-wide Contributor or Owner.

az ad sp create-for-rbac --name "incident-agent" \
  --role "Virtual Machine Contributor" \
  --scopes "/subscriptions/<sub-id>/resourceGroups/<rg-name>/providers/Microsoft.Compute/virtualMachines"

Centralized Secrets Management With Doppler

All environment variables, tokens, and credentials are managed exclusively through Doppler, with audit logs and rotation. Agents never persist secrets to databases or logs.

Workflow: Automated Investigation and Remediation

Incident Trigger

1. Azure Monitor sends an alert to n8n (via webhook or Event Grid). 2. n8n triggers a workflow: parses the incident details, invokes an LLM agent for analysis and remediation suggestions. 3. Agent sends masked data to Claude for stepwise remediation plan generation.

// n8n custom function node
const incident = items[0].json;
const maskedLog = $evaluateExpression('={{ $json["log"] }}', { mask: mask_secrets });
return [{ json: { incidentId: incident.id, log: maskedLog } }];

Remediation Validation and Auditing

Every remediation playbook that touches critical resources requires manual approval (a human-in-the-loop step in n8n). Agent actions are logged to a dedicated Postgres table for post-mortem review.

Automated Remediation Without Leaking Secrets

Where possible, remediation scripts use ephemeral tokens (Azure Managed Identity), never static secrets or keys passed to LLMs. Sensitive operations are always gated by manual review.

Security Verification: Static and Dynamic Analysis

semgrep, bandit, and gitleaks

I run semgrep and bandit in every CI/CD pipeline to catch secret leaks, CWE patterns, and authorization logic flaws. gitleaks scans all code repos for accidental credential exposure.

semgrep --config=auto src/
bandit -r src/
gitleaks detect --source .

Cloud Security Posture Management

I review agent and service principal configurations with Azure Security Center and run regular compliance scans (Azure Policy) to ensure privilege boundaries are enforced.

FAQ

What permissions do your AI agents have?

Only the minimum required for their specific task and resource scope. No blanket Owner or Contributor roles.

Do you ever pass raw logs to LLMs?

No. All logs and incident data are masked before being used as LLM context. Full logs are only available for manual, secure review.

How do you handle secret rotation?

All secrets are managed and rotated through Doppler, with audit logs for every access or change.

Which static code analysis stack do you use?

semgrep for pattern detection, bandit for Python-specific issues, and gitleaks for secret scanning in repos.

Can the agent fix infrastructure bugs autonomously?

Only for non-critical resources and only after explicit manual approval of the remediation plan. All network and database actions require a human in the loop.

Which stage in your LLM pipeline catches the most issues in prod — static analysis, runtime sandbox, or human review? I run a free 30-min stack audit for DACH founders building AI in regulated markets. DM me on LinkedIn or write to @ger_dennis_ai.