From Alert to Verified Recovery: Building a Guardrailed Multi-LLM LinuxOps Agent on Kubernetes
Most Kubernetes operators already have monitoring. They can see when error rates climb and pods misbehave. The expensive part is everything that happens next: correlating metrics with workload state, deciding whether a restart is safe, getting approval, executing the change, and then proving the service is actually healthy again. That human loop is where latency lives.
This post walks through a project that closes that loop with an AI agent, but does it without handing the keys to the model. The agent detects an elevated HTTP 5xx rate, proposes a bounded remediation, and executes a rollout restart only after deterministic guardrails and explicit verification pass. The result is not a demo that simply restarts a pod; it is a system that proves the application recovered before declaring victory.
Starting from a Book, Not a Product
The project began with the Chapter 6 autonomous-agent examples from The Ultimate AI Guide for Linux Engineers by Packt. The examples are useful teaching material: a ServiceRecoveryAgent shows service tooling, memory, token budgeting, and dry-run recovery, while a DeploymentAgent demonstrates planning, approvals, resource limits, and rollback.
What the repository does not provide is one integrated production application. The copied code references interfaces that are not fully implemented in the same tree: config.settings.Settings, memory.episodic.EpisodicMemoryStore, memory.semantic.SemanticMemoryStore, tools.registry.ToolRegistry, observability.metrics.MetricsCollector, guardrails.approvals.ApprovalManager, guardrails.resource_limits.ResourceLimitEnforcer, utils.circuit_breaker.CircuitBreaker, utils.retry.retry_with_backoff, core.reasoning.ReasoningEngine, and core.executor.PlanExecutor. The production planner signature also differs from the synchronous template planner shown in the chapter.
So the work became an integration and refactoring project rather than a simple copy-paste exercise. The goal was to take the concepts, fill the gaps, and produce something that actually runs end-to-end against a Kubernetes workload.
The Multi-LLM Abstraction
One early decision was to avoid tying the agent to a single provider. 
The reasoning layer was wrapped behind a provider-neutral interface, with OpenAI as the default and Anthropic and Ollama selectable through the same configuration.
def create_provider(s):
p = s.llm_provider.lower().strip()
if p == 'openai':
return OpenAIProvider(s.openai_api_key, s.openai_model)
if p == 'anthropic':
return AnthropicProvider(s.anthropic_api_key, s.anthropic_model)
if p == 'ollama':
return OllamaProvider(s.ollama_url, s.ollama_model)
raise ValueError(f'Unsupported LLM_PROVIDER={s.llm_provider!r}')
This separation matters because the tools, safety layer, and Kubernetes RBAC stay constant regardless of which model is reasoning. Switching from OpenAI to a local qwen3-coder:latest instance running on a Mac does not require rewriting the agent logic. The health endpoint confirmed the active provider and model:
{"status":"healthy","provider":"openai","model":"gpt-5.2","approval_mode":"manual"}
The kind Demo Architecture
The demo runs on a local kind cluster with three namespaces: linuxops-system, linuxops-demo, and monitoring.
OpenAI / Anthropic / Ollama
|
v
LinuxOps Agent
/ | \
Prometheus | PostgreSQL memory
|
restricted K8s tools
|
web-api demo
The agent runs as a Deployment in linuxops-system. It talks to Prometheus in monitoring for telemetry, to PostgreSQL for operational memory, and to the Kubernetes API through a namespace-scoped Role that can only read Pods and patch specific Deployments in linuxops-demo. The target workload is web-api, a small FastAPI service continuously exercised by a load generator.
Real Deployment Failures Before the Demo Worked
Building the environment surfaced the kind of failures that appear in real projects. Applying the agent failed immediately because ServiceMonitor CRDs were missing:
no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first
After installing Prometheus Operator CRDs, Grafana entered CrashLoopBackOff with a disk-pressure error:
mkdir: can't create directory '/var/lib/grafana/plugins': No space left on device
Freeing Docker/kind storage resolved that. Then the LinuxOps Deployment existed but had no Pod. The ReplicaSet event explained why:
error looking up service account linuxops-system/linuxops-agent:
serviceaccount "linuxops-agent" not found
Once the ServiceAccount, Role, and RoleBinding were applied, the agent Pod started and /health returned 200. These are not footnotes; they are reminders that agentic demos must survive the same deployment reality as any other service.
Injecting a Fault That Kubernetes Readiness Misses
The web-api exposes a /fault/enable endpoint that sets an in-process flag. While the flag is true, every request returns HTTP 500, but the container keeps running and the readiness probe still passes.
@app.get('/')
def root():
if fault:
REQ.labels(status='500').inc()
return Response('transient demo failure\n', status_code=500)
REQ.labels(status='200').inc()
return {'service': 'web-api', 'status': 'ok'}
This is the critical detail. A system that trusts Kubernetes alone would see a Ready pod and conclude everything is fine. The failure is application-level, so the remediation must be driven by application telemetry, not just pod phase.
After enabling the fault, Prometheus showed web_api_http_requests_total for status 5xx climbing while the pod reported zero restarts.
Perception, Reasoning, and Dry-Run
When asked to investigate elevated HTTP 5xx errors, the agent runs an observation step that gathers exactly what a human would check first:
- Prometheus one-minute 5xx ratio
- Deployment status
- Pod list and restart counts
- Relevant previous episodes from PostgreSQL
The reasoning engine then asks the LLM to return structured JSON with a fixed schema: summary, identified issues, root-cause hypothesis, confidence, and recommended steps. The prompt explicitly restricts tools to the allow-listed set and reminds the model that the web-api fault lives in process memory, so a rollout restart can clear it.
With dry_run=true, the plan is simulated. The agent reports what it would do without touching Kubernetes. This lets a human review the reasoning and the proposed action before anything changes.
Approval Guardrails
The deterministic layer, not the LLM, decides which actions require approval. Any Kubernetes write tool such as k8s_rollout_restart or k8s_scale_deployment is marked as approval-requiring, and the target namespace and deployment are overwritten from trusted task context before execution.
if st.tool_name in {"k8s_rollout_restart", "k8s_scale_deployment"}:
st = st.model_copy(update={
"parameters": p,
"requires_approval": True,
"risk_level": "medium" if st.risk_level == "low" else st.risk_level,
})
With APPROVAL_MODE=manual, a live task that includes a rollout restart returns:
"status": "approval_required"
The model proposed the action, but the policy engine blocked execution. That separation is the heart of the safety design: the LLM reasons and proposes; deterministic code decides what can actually happen.
The First Recovery Was Wrong
With demo auto-approval enabled, the first implementation executed the rollout restart and declared success as soon as Kubernetes reported the Deployment as available. But the immediate Prometheus 5xx ratio was still 1.0. Every request was still failing.
This exposed the core flaw in the initial verification model: Kubernetes availability is not sufficient evidence of application recovery. A pod can be Ready while the application inside is still returning errors. Declaring the incident closed based only on Deployment status would have left the service broken.
Hardened Recovery Logic
The recovery flow was rewritten to require a chain of postconditions before the task status becomes completed:
rollout restart
↓
wait for Deployment rollout completion
↓
require a NEW replacement Pod
↓
require Running + Ready
↓
query Prometheus repeatedly
↓
normalize empty 5xx vector to 0.0
↓
require 5xx ratio < 5%
↓
require 3 consecutive healthy checks
↓
COMPLETED
The constants were set as follows:
RECOVERY_5XX_THRESHOLD = 0.05
RECOVERY_REQUIRED_CONSECUTIVE_CHECKS = 3
RECOVERY_CHECK_INTERVAL_SECONDS = 10
RECOVERY_MAX_CHECKS = 12
If any postcondition fails, the task returns verification_failed instead of completed. The empty-vector normalization is important because immediately after a restart there may be no 5xx samples yet; that case is treated as zero rather than unknown.
Final Verified Recovery
The final incident-response run executed this sequence:
prometheus_query
k8s_deployment_status
k8s_list_pods
k8s_rollout_restart
wait_for_rollout
wait_for_ready_replacement_pod
prometheus_query
application_recovery_verification
The old pod web-api-5d8fc88b7d-p8mfb was replaced by a new Ready pod web-api-59d546895b-bcs75. Then Prometheus was sampled every ten seconds:
| Attempt | 5xx ratio | Healthy | Consecutive |
|---|---|---|---|
| 1 | 1.0000 | No | 0 |
| 2 | 0.8680 | No | 0 |
| 3 | 0.6983 | No | 0 |
| 4 | 0.5317 | No | 0 |
| 5 | 0.3652 | No | 0 |
| 6 | 0.1946 | No | 0 |
| 7 | 0.0000 | Yes | 1 |
| 8 | 0.0000 | Yes | 2 |
| 9 | 0.0000 | Yes | 3 |
Only after three consecutive samples below the 5% threshold did the agent return:
"status": "completed"
The total time from restart initiation to verified recovery was roughly ninety seconds, measured by nine telemetry checks at ten-second intervals plus the rollout time.
What This Means for Operations Teams
The financial case is similar to other automation projects: if a team currently spends one hour investigating and remediating each of ten weekly incidents, reducing that to fifteen minutes of review saves roughly six and a half hours per week, or about three hundred hours annually. But the bigger gain is reliability. An agent that stops as soon as it sees a Ready pod can close a ticket while users are still seeing errors. An agent that waits for application telemetry to confirm recovery is far less likely to leave a latent outage running.
The architecture also keeps operational risk bounded. The agent has no arbitrary shell tool, no Docker socket, no privileged container, and no cluster-admin binding. Its Kubernetes Role is scoped to a single namespace and a small set of verbs. Write actions require approval by policy, and every incident is stored in PostgreSQL for later retrieval.
The Engineering Takeaway
The final demo is not “AI restarted a Kubernetes pod.” It is a guardrailed, multi-LLM-capable, closed-loop LinuxOps agent that uses Prometheus and Kubernetes to detect, reason about, remediate, and verify an application incident.
The principle is worth repeating: the LLM reasons and proposes; deterministic code, Kubernetes RBAC, approval policy, bounded tools, and post-action verification control what actually happens and whether recovery succeeded. That is the pattern that makes agentic operations trustworthy enough to run in production.