Skip to main content
Hashnode
Jenna's Runbooks

Projects

The *Track suite: ML and AI applied to operational signals. Each project answers one question an on-call engineer asks.

ConsoleTrack

What changed, and what broke because of it?

ConsoleTrack catches out-of-band AWS changes within minutes, records who made them, how, and which Terraform-managed resource they touched, and links later EKS incidents back to the change that most likely caused them. A read-only AI agent triages incidents through two MCP servers. Every AWS-side fix needs a human's approval.

Try the demo · Watch the 3-minute walkthrough · View the code

How it works

  1. Intake: CloudTrail → EventBridge → SQS catches console, CLI, CloudShell, and SDK writes, with a canary heartbeat to prove the pipeline is alive.
  2. Change records: each change is matched to its Terraform address and analyzed, then a reviewer can Accept it with a note, Revert it, or Codify it.
  3. Incidents: Alertmanager opens an incident, and the triage agent investigates through a read-only k8s-mcp and aws-mcp, then proposes a fix.
  4. Remediation: a policy gate decides. One narrow cluster fix, raising a memory limit after an OOM kill, runs automatically behind a deterministic validator, a server-side dry run, rollout verification, and automatic rollback. Everything else waits for a human.
  5. Observability: one OpenTelemetry trace follows each incident from alert to rollout, in Grafana (Tempo, Loki, Prometheus).

Design principles

  • The agent only reads. Both MCP servers are read-only by design: an allowlist of API calls, checked twice; IAM scoped by path; RBAC verified on a real cluster.
  • Untrusted text stays untrusted. Text a user could have written, such as a resource tag or CloudTrail parameter, is returned to the agent labeled as untrusted data, a defense against prompt injection.
  • Autonomy is earned, not assumed. The OOM fix graduated to automatic only after an eval: across 20 runs, the agent proposed the correct fix every time. The eval also showed that model confidence was poorly calibrated, so safety lives in the deterministic layers, not in the model's self-assessment.

Built with

Terraform · Amazon EKS · Pod Identity · Helm and Helmfile · FastAPI · PostgreSQL · React · Claude API · Model Context Protocol · OpenTelemetry · Grafana · GitHub Actions with OIDC

Built in 13 days as the final project for the DataTalks.Club AI Dev Tools Zoomcamp.

Read more

Posts in the ConsoleTrack series are coming soon.

FaultTrack

What kind of fault is this? Classifying microservice faults from metrics with classical ML. Coming November 2026.