All Projects
08 / 08
08PythonProduction Verified

TaskMesh

Distributed Workflow Orchestration Engine

Role

Lead Engineer: queue topology, consumer groups, worker pool, idempotency engine

Primary Stack

Python · FastAPI · Redis Streams · PostgreSQL · Prometheus · Docker

View on GitHub →
01 // Problem Space

“Process arbitrary background tasks across concurrent workers without dropping a single job during traffic spikes or partial worker crashes on a single low-cost node.”

02 // Engineering Constraints
CONSTRAINT 01

Had to run efficiently on a single low-cost VPS without heavy clustering software

CONSTRAINT 02

No external message broker cluster available: build lean from scratch

CONSTRAINT 03

Zero tolerance for task loss under worker crash or node reboot

03 // Architectural Decisions
Why Redis Streams over RabbitMQ / Celery?▼

RabbitMQ requires a separate Erlang runtime, complicated cluster persistence setup, and memory overhead we could not spare on a constrained VPS. Redis Streams gave us Consumer Groups, message acknowledgement (XACK), and a durable append-only log with sub-millisecond dispatch within the existing Redis instance.

Why PostgreSQL for state transitions vs. Redis alone?▼

While Redis is ultra-fast, Redis memory is volatile without heavy AOF syncing. State transitions (PENDING to PROCESSING to COMPLETED to FAILED) require ACID compliance and idempotency keys to ensure that crashed workers never execute duplicate side effects upon restart.

04 // High-Fidelity Architecture System
● LIVE INTERACTIVE

Distributed System Architecture

Explore the multi-tier topology below. Switch between the interactive blueprint canvas, standard Mermaid.js flowcharts, and the step-by-step request simulator.

TaskMesh System Blueprint
TOPOLOGY:

Event-driven distributed task execution pipeline with Redis Streams message broker, stateless worker consumer groups, and PostgreSQL ACID idempotency verification.

🌐INGRESS PLANENon-blocking HTTP gateway & payload validation
↓ DATA PIPELINE FLOW
⚡BROKER & PIPELINEDurable partitioned streams & consumer groups
↓ DATA PIPELINE FLOW
⚙️COMPUTE & AGENT ENGINEStateless concurrent execution & circuit breaking
↓ DATA PIPELINE FLOW
💾PERSISTENCE & STATEACID state transitions & Dead Letter Queue
↓ DATA PIPELINE FLOW
📡TELEMETRY & ALERTSPrometheus metrics scraper & Grafana health alerts
ingress planeACTIVE

FastAPI Ingress Gateway

FastAPI / Uvicorn

Component Role & Scope

Receives task triggers, generates deterministic UUIDv7 idempotency keys, and immediately returns HTTP 202 Accepted.

🛡 Fault Tolerance & Recovery

Rate limits incoming bursts via Token Bucket algorithm; drops unauthenticated payloads at the edge.

Topology Linkages (1)

● Root Edge Component (Direct client intake)
→ Egress toredis-streams

Publishes validated task payloads with deterministic idempotency keys

Core Implementation Logic

@app.post('/v1/tasks', status_code=202)
async def enqueue_task(payload: TaskSchema):
    task_id = uuid7()
    await redis.xadd('stream:tasks', {'id': str(task_id), 'data': payload.json()})
    return {'task_id': task_id, 'status': 'QUEUED'}
Click any node on canvasLive Synchronized
05 // Validated Outcomes & SLAs
2,400ops/min sustained
22msP95 latency
100%task retention

Engineering Post-Mortem & Next Iteration

I would add dead-letter queues from day one because we lost 3 hours debugging a poison pill message causing cascading consumer failures. I would also expose metrics via OpenTelemetry instead of raw Prometheus scraping, and add per-task replay capability.

Next Case Study

01 // NEXT

Nuuvixx AgentGovern

Autonomous Multi-Agent Governance & Execution OS

→