Complete Build Tutorial · Signal to Verified Recovery
RelayOps
Incident Workflow Platform
A production incident workflow platform with grounded AI diagnosis. Process financial integration incidents through eight quality gates (environment, vertical slice, simulator, integrations, diagnosis, safe actions, evaluation, operations) with deterministic recovery. Built completely end-to-end from empty folder to fully working portfolio.
Status: Complete Build
A 6-week core build + 2-week hardening and packaging tutorial. Covers all eight no-skip gates, deterministic testing, production deployment, and operational resilience. Uses synthetic financial data; no real customer or credentials in any phase.
The problem
Financial integration incidents are chaotic and slow to resolve. When a broker token expires, data entitlement fails, or a payment webhook duplicates, operators dig through logs manually:
- •Identify the root cause by reading logs and system state
- •Design the recovery action (rotate tokens, replay webhooks, repair data)
- •Execute it against live systems, hoping the repair is idempotent
- •Verify recovery actually worked and didn't make things worse
How it works
RelayOps ingests incidents, surfaces ranked root-cause hypotheses, proposals safe recovery actions with explicit approval, and verifies recovery through deterministic checks. Every stage has a gate; you pass only when that capability is proven.
Architecture
Next.js frontend, FastAPI backend with SQLAlchemy models, PostgreSQL with pgvector for evidence retrieval, deterministic scenario engine, grounded AI diagnosis, MCP tools for safe actions, and evaluation harness with quality gates that block releases on model regression.
Why deterministic scenarios
A human tester or external SLA could inject incidents, but you lose reproducibility. RelayOps seeds scenarios with known answers, injects them at the evaluator level, and keeps the application seeing only observable, repeatable synthetic data. The model never sees expected root cause, approved tools, or recovery condition.
Every scenario verifies that the model can rank real root causes above plausible decoys, and that proposed actions stay within the approval contract. Quality gates ensure the model never regresses.
Eight no-skip production gates
Each gate verifies a specific piece of production readiness. You cannot skip a gate and you cannot soft-fail one. The tutorial walks through all eight.
Eight incident families
Expired broker token, missing data entitlement, duplicate payment webhook, settlement schema drift, reconciliation mismatch, processor retry storm, audit pipeline interruption, model provider outage, each with known root cause, approved recovery and deterministic check.
Grounded diagnosis
AI turns evidence into ranked hypotheses without inventing claims. Every hypothesis cites evidence IDs. The model never sees root cause, approved tools or recovery condition.
Safe action proposals
Tool proposals include exact arguments bound to an approval record. Execution reads approvals from the database. The executor cannot run a tool outside its approval contract.
Deterministic scenario engine
Same seed reproduces the same incident, evidence timeline, and recovery check. Injection happens at the evaluator level. The application sees only observable, repeatable synthetic data.
Production readiness gates
Every gate verifies a specific capability: environment setup, vertical slice, deterministic repetition, real integrations, grounded diagnosis, safe actions, quality evaluation and operational resilience.
Stack
See the complete build
The complete tutorial covers every phase from environment setup through portfolio-ready deployment. Real production practices built in from week one, not retrofitted at the end.