Aspect	Celery	Dirty Arbiters
Dependencies	Redis/RabbitMQ + Celery	None (built into Gunicorn)
Deployment	Multiple processes/containers	Single process
State	Stateless workers	Stateful workers (keep models loaded)
Progress	Polling or WebSocket	Native streaming
Configuration	Separate config	Same gunicorn.conf.py

Quick Start

Local Development

# Install dependencies
pip install flask requests pytest
pip install -e ../..  # Install gunicorn from source

# Run the application
gunicorn -c gunicorn_conf.py app:app

# In another terminal, test it
curl http://localhost:8000/health
curl -X POST http://localhost:8000/api/email/send \
  -H "Content-Type: application/json" \
  -d '{"to": "test@example.com", "subject": "Hello", "body": "World"}'

Docker

# Build and run
docker compose up --build

# Run with tests
docker compose --profile test up --build --abort-on-container-exit

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Gunicorn Master                          │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │ HTTP Worker │  │ HTTP Worker │  │ HTTP Worker │  ...    │
│  │  (gthread)  │  │  (gthread)  │  │  (gthread)  │         │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘         │
│         │                │                │                 │
│         └────────────────┼────────────────┘                 │
│                          │                                  │
│                    Unix Socket IPC                          │
│                          │                                  │
│         ┌────────────────┼────────────────┐                 │
│         │                │                │                 │
│  ┌──────▼──────┐  ┌──────▼──────┐  ┌──────▼──────┐         │
│  │ EmailWorker │  │ ImageWorker │  │ DataWorker  │  ...    │
│  │  (2 procs)  │  │  (2 procs)  │  │  (4 procs)  │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
│                                                             │
│                    Dirty Arbiter                            │
└─────────────────────────────────────────────────────────────┘

Task Workers

EmailWorker

send_email(to, subject, body) - Send single email
send_bulk_emails(recipients, subject, body) - Bulk send with progress streaming
stats() - Worker statistics

ImageWorker

resize(image_data, width, height) - Resize image
generate_thumbnail(image_data, size) - Generate thumbnail
process_batch(images, operation, **params) - Batch process with streaming

DataWorker

aggregate(data, group_by, agg_field, agg_func) - Aggregate data
etl_pipeline(source_data, transformations) - ETL with progress streaming
cached_query(query_key, ttl) - Cached query execution

ScheduledWorker

cleanup_old_files(directory, max_age_days) - File cleanup
generate_daily_report() - Daily report generation
sync_external_data(source) - External data sync

API Endpoints

Email

POST /api/email/send - Send single email
POST /api/email/send-bulk - Bulk send (SSE streaming)
GET /api/email/stats - Worker stats

Image

POST /api/image/resize - Resize image
POST /api/image/thumbnail - Generate thumbnail
POST /api/image/process-batch - Batch process (SSE streaming)
GET /api/image/stats - Worker stats

Data

POST /api/data/aggregate - Aggregate data
POST /api/data/etl - ETL pipeline (SSE streaming)
POST /api/data/query - Cached query
GET /api/data/stats - Worker stats

Scheduled

POST /api/scheduled/cleanup - Run cleanup
POST /api/scheduled/daily-report - Generate report
POST /api/scheduled/sync - Sync data
GET /api/scheduled/stats - Worker stats

Streaming Progress Example

import requests
import json

# Start bulk email with streaming progress
resp = requests.post(
    "http://localhost:8000/api/email/send-bulk",
    json={
        "recipients": ["a@x.com", "b@x.com", "c@x.com"],
        "subject": "Newsletter",
        "body": "Hello!",
    },
    stream=True,
)

for line in resp.iter_lines():
    if line and line.startswith(b"data: "):
        progress = json.loads(line[6:])
        if progress["type"] == "progress":
            print(f"Progress: {progress['percent']}%")
        elif progress["type"] == "complete":
            print(f"Done! Sent: {progress['sent']}, Failed: {progress['failed']}")

Celery Migration Guide

Before (Celery)

# tasks.py
from celery import Celery

app = Celery('tasks', broker='redis://localhost')

@app.task
def send_email(to, subject, body):
    # Send email
    return {"status": "sent"}

@app.task(bind=True)
def send_bulk(self, recipients, subject, body):
    for i, to in enumerate(recipients):
        send_email(to, subject, body)
        self.update_state(state='PROGRESS', meta={'current': i})

# views.py
from tasks import send_email, send_bulk

def send_view(request):
    send_email.delay(to, subject, body)  # Async
    return {"status": "queued"}

After (Dirty Arbiters)

# tasks.py
from gunicorn.dirty.app import DirtyApp

class EmailWorker(DirtyApp):
    workers = 2  # Limit workers

    def init(self):
        self.smtp = connect_smtp()  # Stateful!

    def __call__(self, action, *args, **kwargs):
        return getattr(self, action)(*args, **kwargs)

    def send_email(self, to, subject, body):
        return {"status": "sent"}

    def send_bulk(self, recipients, subject, body):
        for i, to in enumerate(recipients):
            self.send_email(to, subject, body)
            yield {"type": "progress", "current": i}  # Native streaming!

# views.py
from gunicorn.dirty import get_dirty_client

def send_view(request):
    client = get_dirty_client()
    result = client.execute("tasks:EmailWorker", "send_email", to, subject, body)
    return result  # Sync result, no polling!

Configuration

# gunicorn_conf.py

# HTTP workers
workers = 4
worker_class = "gthread"
threads = 4

# Task workers (replace Celery)
dirty_apps = [
    "tasks:EmailWorker",
    "tasks:ImageWorker",
    "tasks:DataWorker",
]
dirty_workers = 9
dirty_timeout = 300

Running Tests

# Unit tests (no server needed)
pytest tests/test_tasks.py -v

# Integration tests (server must be running)
APP_URL=http://localhost:8000 pytest tests/test_integration.py -v

# All tests via Docker
docker compose --profile test up --build --abort-on-container-exit