PythonProduction Verified

ChaosAgent: Multi-Agent Fault Injector

Autonomous Chaos Engineering & Byzantine Fault Injection for LLM Agents

Role

Lead Systems & Security Engineer: fault injector harness, Byzantine interceptor, resilience benchmark runner

Primary Stack

Python · LangGraph · FastAPI · Docker · Pytest · Prometheus

View on GitHub →

Production multi-agent systems suffer from cascading failures when tool APIs hang, prompt drift introduces hallucinations, or Byzantine agent nodes return corrupted outputs.

CONSTRAINT 01

Must inject 6+ discrete failure modes: latency spikes, dropped tokens, hallucinated tool schemas, partial network partitions

CONSTRAINT 02

Automated resilience scoring and recovery time objective (RTO) benchmarking

CONSTRAINT 03

LangGraph cyclic agent loop compatibility with zero production downtime

Why deterministic fault injection at the SDK interceptor layer?

Injecting chaos at the network level (e.g., iptables) tests infrastructure but misses agent-specific semantic failures. ChaosAgent intercepts tool invocations in Python user space, corrupting JSON schemas, injecting contradictory context, and simulating rate-limit backpressure to test LLM recovery reasoning.

Why automated Recovery Time Objective (RTO) scoring?

Teams need to quantify how quickly an agent detects a failed tool and routes to an alternate execution path. ChaosAgent computes a Resilience Index based on recovery steps, retry token waste, and state divergence.

● LIVE INTERACTIVE

Distributed System Architecture

Explore the multi-tier topology below. Switch between the interactive blueprint canvas, standard Mermaid.js flowcharts, and the step-by-step request simulator.

ChaosAgent: Multi-Agent Fault Injector System Blueprint
TOPOLOGY:

Chaos engineering framework for multi-agent LLM systems with configurable fault schedules, Byzantine interceptors, and automated recovery telemetry.

🌐INGRESS PLANEDeclarative YAML experiment runner and fault scheduler
↓ DATA PIPELINE FLOW
BROKER & PIPELINEDynamic proxy injecting delays, corrupted JSON, and API drops
↓ DATA PIPELINE FLOW
⚙️COMPUTE & AGENT ENGINELangGraph agent loops under simulated stress
ingress planeACTIVE

Chaos Orchestrator

Python / AsyncIO

Component Role & Scope

Schedules fault injection experiments (latency, drop, schema mutate) across agent execution rounds.

🛡 Fault Tolerance & Recovery

Safe-abort emergency switch restores normal agent routing instantly.

Topology Linkages (1)

● Root Edge Component (Direct client intake)
→ Egress tobyzantine-interceptor

Arms chaos interceptor with specific fault injection recipe

Click any node on canvasLive Synchronized
6+chaos fault vectors
100%circuit break capture
<250msfailover recovery (RTO)

Engineering Post-Mortem & Next Iteration

I would add automated fuzzing against prompt injection safety filters to benchmark adversarial jailbreak resilience during chaos runs.