Going Autonomous: How Auto-SRE Automates Incident Response and Self-Healing with LangGraph and Splunk MCP
Going Autonomous: How Auto-SRE Automates Incident Response and Self-Healing
A deep dive into an enterprise-grade agentic site reliability assistant powered by LangGraph, Splunk MCP, and automated rollback workflows.
The Dawn of Agentic Site Reliability Engineering
In modern cloud systems, microservice architectures have solved the challenges of scale and agility, but they have simultaneously turned debugging into a complex, distributed puzzle. When an outage occurs in production, every second represents lost revenue, compromised SLA thresholds, and customer frustration. The typical workflow of resolving high-severity incidents has historically remained manual. An engineer is paged at 3:00 AM, logs into an observability panel like Splunk or Datadog, sifts through hundreds of traces, correlations, and Git VCS checkins, manually attempts to isolate the root cause, and finally triggers a deployment rollback or schema migration script to resolve the degradation. This process, while reliable, can take anywhere from 30 minutes to several hours depending on the scale and complexity of the blast radius.
This is where Agentic Ops comes in. By utilizing state-of-the-art Large Language Models (LLMs) configured as autonomous agents, we can automate the diagnostics and remediation loop. This isn't just about runbook automation or writing scripts; it is about building an intelligent, reasoning system that can dynamically adapt to unforeseen anomalies, run safe and secure database profiling queries, correlate metrics with commit histories, and execute self-healing steps under human supervision.
Built for the prestigious Splunk Agentic Ops Hackathon 2026, Auto-SRE is an enterprise-grade, autonomous SRE assistant designed to triage, diagnose, and remediate application degradations in minutes instead of hours. Let’s explore how this system works under the hood, how it utilizes the Splunk Model Context Protocol (MCP) to access telemetry, and how a LangGraph state machine ensures safe and structured execution.
The Core Problem: Bridging Observability and Remediation
Observability platforms are exceptional at telling you when something is broken. Modern telemetry pipelines pull logs, capture tracing spans, and aggregate CPU and memory metrics, surfacing alerts to Slack or PagerDuty. However, these systems do not possess the agency to repair what is broken. A static alert like "payment-service checkout latency exceeds 5000ms" only flags the symptom. Finding the cause—be it a missing database index on a newly deployed table, a hostname typo in a production configuration file, or an accidental reduction of CPU core limits in a Kubernetes manifest—requires cognitive overhead.
Static runbook scripts are fragile. They operate on rigid assumptions (e.g., if latency spikes, restart the server). If the latency is actually caused by a locked database row, restarting the service might temporarily resolve the issue but will trigger cascading failures elsewhere. Auto-SRE bridges this gap by using a dynamic planning and tool-calling agent framework.
To highlight the paradigm shift Auto-SRE introduces, consider the side-by-side comparison below. While manual resolution cycles waste valuable engineering time, Auto-SRE autonomously traces, analyzes, and proposes fixes within seconds.
| Incident Step | Manual Investigation (45 - 60 Minutes) | Auto-SRE Autonomous Flow (2 Minutes) |
|---|---|---|
| Triage | A human monitors alerts, locates the affected service, and manually gathers stakeholders into a war room. | The Triage Agent intercepts the alert and maps the affected service, dependencies, and blast radius within 5 seconds. |
| APM & Logs Scan | Engineers execute manual SPL (Splunk Search Processing Language) queries, scanning transaction traces and db locks. | The Diagnostician Agent issues automated MCP calls, running Splunk queries to analyze trace logs. |
| VCS Correlation | Devs crawl git commits, checking who deployed what, matching deployment timestamps with error occurrences. | The agent queries VCS records on Splunk, identifies the exact commit modifying configurations, and produces a git diff. |
| Remediation | Engineers decide on a repair, manually executing SQL migrations or writing rollback commands in terminal sessions. | The Remediation Agent produces a precise rollback configuration or SQL index patch, ready for execution. |
System Architecture: The LangGraph State Machine
The core intelligence of Auto-SRE is structured as a state machine using LangGraph. Unlike traditional agent chains that execute linear sequences (e.g., Prompt -> Output), real SRE processes are cyclical. If a diagnosis is incomplete, the agent must query more metrics. If a remediation script fails to deploy, the system must fallback, re-evaluate, or alert a human operator.
Below is the logical workflow representing the incident lifecycle managed by Auto-SRE:
The state machine processes incidents through five specialized agents:
- Triage Agent: Listens to alerts, categorizes severity, lists affected services, and identifies the blast radius using topology maps.
- Diagnostician Agent: Utilizes the Splunk Model Context Protocol (MCP) server to issue secure, automated SPL queries. It retrieves logs, traces database execution paths, and fetches git commit diffs from repository version control.
- Remediation Agent: Evaluates the logs and outputs recovery options (e.g., executing a database hotfix or applying a git rollback patch).
- Human-in-the-Loop (HITL) Gate: Provides safety guardrails. No destructive action (like modifying a database schema or pushing code adjustments) occurs without an explicit operator approval click via Slack ChatOps or the React Web UI.
- Auditor Agent: Records the completed action, the prompt history, and the system output, formatting the incident log and writing it directly to a dedicated Splunk index (index=sre_audit) for historical compliance.
Feature Deep Dive: Telemetry, ChatOps & Self-Healing
To understand the strength of the Auto-SRE platform, we must examine the specific systems and integrations that make up its core features.
1. Splunk Model Context Protocol (MCP) Server
Model Context Protocol (MCP) is an open-source standard created by Anthropic that allows AI applications to safely access files, read databases, and trigger local API tools. Auto-SRE implements a custom Splunk MCP integration. Instead of writing custom API code for every agent query, the agent dynamically asks the Splunk MCP server to execute commands like:
- query_logs(query_string: str): Runs an SPL query in Splunk Cloud to check HTTP status rates or latency spikes.
- get_apm_traces(trace_id: str): Traces span durations across nested service calls to highlight slow database queries.
- fetch_vcs_history(service: str): Inspects Splunk-indexed VCS commit records to match timestamps of recent deployments.
2. ChatOps Slack Integration & Dashboard
One of the project's standout features is its interaction design. Outages are managed via an Apple-style, glassmorphic SRE Control Dashboard and a Slack workspace simulation. When the Triage Agent identifies a critical issue, it creates a new incident room on the dashboard and pushes a Slack card containing:
- A summary of the incident (e.g., DB lock contention in Payment microservice).
- A visual git diff highlighting what changed in the recent deployment.
- Interactive buttons: "Approve Schema Hotfix", "Revert Configuration Commit", or "Escalate to On-Call Engineer".
3. Automated Remediation Drivers
Rather than simply alerting, Auto-SRE implements concrete executors capable of resolving three specific incident classes:
- Database Migrations: Resolves missing index locks by dynamically generating and applying a schema-compliant DDL index script on target tables.
- Git Configuration Rollbacks: Leverages version control history to extract previous configuration payloads, rollback production configs, and trigger microservice hot-reloading.
- Deployment Manifest Corrections: Reverts container manifest modifications (e.g., CPU throttling rules) that starved API gateways of core execution capacity.
System Architecture & Directory Layout
The codebase is split into a modular backend server and a clean React-based control panel. Below is the directory structure layout showing how the core features are distributed:
auto-sre-demo/ ├── backend/ # Python FastAPI Backend │ ├── app/ │ │ ├── agents/ # Specialized SRE Agent Definitions │ │ │ ├── triage.py # Outage detection & blast mapping │ │ │ ├── diagnose.py # Splunk MCP logs & traces analysis │ │ │ ├── remedy.py # Hotfix execution logic │ │ │ └── auditor.py # Splunk audit trails indexer │ │ ├── mcp/ # Splunk MCP client configuration │ │ └── main.py # FastAPI entrypoint and WebSocket routing │ ├── requirements.txt # Python runtime dependencies │ └── venv/ # Local python virtual environment ├── frontend/ # React control panel dashboard │ ├── public/ # Static dashboard visual mocks │ ├── src/ # React dashboard components & topology │ └── package.json # Node dependencies ├── run.sh # Unified startup execution script └── README.md # System overview and local guide
Step-by-Step Installation & Local Execution
To run the Auto-SRE demonstration environment locally, follow these configuration instructions. Ensure that your development environment meets the core prerequisites before building the service.
1. Core Prerequisites
Before launching the backend and frontend components, make sure you have the following packages installed on your local host:
- Python: Version 3.10 or higher.
- Node.js: Version 18.x or higher (along with npm/pnpm).
- Ollama (Optional): Required for offline local LLM reasoning. Ensure Ollama is running and has the required coding model pulled:
ollama run qwen2.5-coder:1.5b
2. Backend Setup
Clone the codebase, navigate to the backend subdirectory, initialize a clean Python virtual environment, and install all required modules:
cd backend python3 -m venv venv source venv/bin/activate pip install -r requirements.txt
Configure your environment variables. If you do not have a Gemini API Key available, Auto-SRE will automatically fall back to using local Ollama models or a structured fallback simulator, enabling you to test the flow without external API dependencies:
export GEMINI_API_KEY="your-gemini-key"
3. Frontend Setup
Open a secondary terminal session, navigate to the frontend React directory, and download the web interface dependencies:
cd frontend npm install
4. Booting the Application
From the root folder of the repository, execute the unified startup script. This launches the backend server on port 8000 and the React dashboard on port 3000 concurrently:
./run.sh
Once running, access the local system web interfaces at:
- Control Panel Dashboard: http://localhost:3000
- FastAPI backend routing: http://localhost:8000
Exploring Interactive Outage Scenarios
To demonstrate its self-healing capabilities, the Auto-SRE playground features three built-in outage simulation scenarios. You can trigger these from the Control Panel UI to observe the agents in action:
These scenarios mirror production incidents common in large-scale deployments, showing how AI can safely mitigate issues in databases, configuration updates, and resource allocation.
Scenario A: Payment-Service Checkout Latency
A sudden spike in latency occurs during Checkout transactions. The Diagnostician Agent runs database telemetry analysis and identifies that a slow SQL query is locking tables because of a missing database index on the SQLite database. The agent automatically drafts a DDL schema hotfix to add the missing index. Once you click Approve in the UI or via the simulated Slack channel, the agent applies the DDL index schema, and checkout latency drops back to baseline immediately.
Scenario B: Auth-Service Cache Failures
An engineer merges a new configuration commit, causing cache communication to break and authentication calls to fail. The agent checks Splunk's VCS index logs, finds a recent commit modifying the hostname in config/production.json, and presents a visual git diff showing the typo. The agent drafts a git rollback patch. Once approved, the rollback is executed, and authentication service cache operations are restored.
Scenario C: Gateway CPU Resource Starvation
The system experiences packet drops at the API Gateway due to CPU resource limits. The agent scans host telemetry logs, detects container throttling in gateway-deployment.yaml, and determines that the CPU limit was recently restricted to an unsafe value. The agent proposes a roll-back to the previous stable gateway deployment manifest, restoring CPU core limits and resolving the starvation.
Conclusion & The Future of Autonomous SRE
The Auto-SRE project, built for the Splunk Agentic Ops Hackathon 2026, demonstrates the potential of agentic AI in modern system operations. By combining the data-gathering capabilities of Splunk MCP with the multi-agent orchestration of LangGraph, Auto-SRE handles the root cause analysis, diagnostic, and resolution steps in under two minutes—reducing MTTR by up to 95% while keeping humans firmly in control. If you want to contribute, test the interactive simulator, or deploy Auto-SRE nodes within your lab environments, check out the repository on GitHub!
Interested in autonomous SRE architectures? View the complete source code, review the agent templates, and star the repository on GitHub!
⭐️ Explore Auto-SRE on GitHub