In Python 3.7+, PEP 479 converts StopIteration raised inside coroutines
to RuntimeError. Changed _read_into() to raise NoMoreData instead, which
is already properly handled by the protocol layer.
- Bump version to 25.0.2
- Update copyright year to 2026 in LICENSE and NOTICE
- Add license headers to all Python source files
- Add changelog entry for 25.0.2
- Fix nginx config to use keepalive with upstream (was sending
Connection: close which caused premature connection closure)
- Add _safe_write() to handle socket errors (EPIPE, ECONNRESET,
ENOTCONN) gracefully when client disconnects
- Fix ASGI scope server/client to always be 2-tuples for IPv6
compatibility (IPv6 sockets return 4-tuples)
- Add write_eof() before close() to ensure buffered data is flushed
- Bind to [::] for dual-stack IPv4/IPv6 support in test containers
Closes#3484
When a client disconnects during an ASGI request, the worker now:
1. Sends http.disconnect message to the app's receive queue
2. Allows a configurable grace period for cleanup (default: 3 seconds)
3. Only cancels the task after the grace period expires
This follows the ASGI HTTP Connection Scope spec which defines
http.disconnect as the message apps should receive when clients
disconnect: https://asgi.readthedocs.io/en/latest/specs/www.html#disconnect-receive-event
The grace period prevents CancelledError from propagating to async
database operations, allowing SQLAlchemy and other async DB libraries
to properly reset their connection pools.
New config option: --asgi-disconnect-grace-period (default: 3 seconds)
Closes#3482
The dirty module (which uses asyncio and concurrent.futures) was being
imported at gunicorn startup via gunicorn.arbiter. This caused
concurrent.futures to be imported before user code could call
gevent.monkey.patch_all(), breaking gevent's monkey-patching.
Changes:
- gunicorn/arbiter.py: Import DirtyArbiter and set_dirty_socket_path
lazily inside spawn_dirty_arbiter() instead of at module level
- gunicorn/dirty/worker.py: Import ThreadPoolExecutor lazily inside
run() method instead of at module level
- Add tests/workers/test_gevent_import_order.py with 5 tests verifying:
- concurrent.futures is NOT imported when gunicorn.arbiter loads
- gevent patching works correctly with gunicorn
- Reproduces the exact scenario from the bug report gist
This ensures gevent's monkey.patch_all() can run before concurrent.futures
is imported, allowing proper patching of threading primitives.
Add chunked transfer encoding support to the ASGI worker for HTTP/1.1
streaming responses that don't have a Content-Length header. This fixes
SSE (Server-Sent Events) connections not closing properly.
Without chunked encoding or Content-Length, HTTP/1.1 clients wait for
the connection to close to determine end-of-response, causing streaming
endpoints to hang.
Also updates the celery_alternative example to use FastAPI with the
native ASGI worker and uvloop, demonstrating async task execution with
proper SSE streaming.
- SIGINT/SIGQUIT triggers immediate shutdown, skipping connection waits
- SIGTERM triggers graceful shutdown, waiting for connections
- Arbiter forwards SIGQUIT to workers if received during graceful shutdown
- Workers have 2s to exit cleanly after quick shutdown before SIGKILL
- Add get_app_workers_attribute() to read workers class attribute
- Update _parse_app_specs() to check class attribute when no config override
- Add Docker-based e2e tests for per-app worker allocation
- Add test apps: HeavyModelApp (workers=2), LightweightApp
- Add unit tests for get_app_workers_attribute function
- Add integration tests for class attribute detection
Allow dirty apps to specify how many workers should load them, enabling
significant memory savings for heavy applications like ML models.
- Add `workers` class attribute to DirtyApp (None = all workers)
- Add `parse_dirty_app_spec()` to parse "module:Class:N" format
- Add `DirtyNoWorkersAvailableError` for app-specific error handling
- Update DirtyArbiter with per-app worker tracking and routing
- Maintain backward compatibility when no dirty_apps configured
Example: 8 workers x 10GB model = 80GB RAM needed
With workers=2: 2 x 10GB = 20GB RAM (75% savings)
Configuration formats:
- Class attribute: `workers = 2` on DirtyApp subclass
- Config format: `module:class:N` (e.g., `myapp.ml:HugeModel:2`)
- Add explicit do_handshake() in base_async.py before ALPN check
when do_handshake_on_connect is False
- Mark eventlet worker as deprecated (removal in 26.0)
- Add HTTP/2 gevent example with Docker and tests
- Update documentation to reflect eventlet deprecation
- Remove eventlet websocket example (gevent version exists)
The ALPN fix ensures HTTP/2 works correctly with gevent and eventlet
workers when do_handshake_on_connect config is False (the default).
Without explicit handshake, selected_alpn_protocol() returns None.
- Reorder exception handlers: specific exceptions before ProtocolError
- Extract _wait_for_flow_control_window() to reduce return statements
- Refactor flow control waiting to avoid too-many-return-statements
- Send GOAWAY with correct error codes for protocol violations
- Handle StreamClosedError and FlowControlError gracefully
- Return False instead of raising for missing/closed streams
- Handle flow control window overflow per RFC 7540
- Fix reader race condition and add h2 exception handling
- Wait for WINDOW_UPDATE when flow control window is zero/negative
- Use h2 exception's error_code for INITIAL_WINDOW_SIZE violations
- Guard early_hints callback against calls after response started
- Fix :authority precedence over Host header per RFC 9113 section 8.3.1
- Add nginx early_hints documentation link
- Use standard port 443 in curl examples
- Add pylint disable comments for global-statement in lazy import pattern
- Remove unnecessary pass statements in error subclasses
- Remove useless return None at end of _handle_request_received methods
Implement HTTP 103 Early Hints as modern replacement for HTTP/2 Server Push.
This allows servers to send resource hints before the final response,
enabling browsers to preload assets in parallel.
WSGI support:
- Add wsgi.early_hints callback to environ dict
- Apps can call environ['wsgi.early_hints'](headers) to send 103 responses
- Silently ignored for HTTP/1.0 clients (don't support 1xx responses)
ASGI support:
- Handle http.response.informational message type
- Apps can await send({"type": "http.response.informational", "status": 103, ...})
HTTP/2 support:
- Add send_informational() method to HTTP2ServerConnection
- Add async send_informational() method to AsyncHTTP2Connection
- Wire up early hints in gthread worker for HTTP/2 requests
Includes unit tests and Docker integration tests for all protocols.
Integrate HTTP/2 support with gunicorn workers:
- sync worker: Add warning that HTTP/2 is not supported
- gthread worker: Full HTTP/2 support
- ALPN negotiation with explicit handshake for deferred SSL
- HTTP/2 connection lifecycle management
- Per-stream request handling with WSGI
- AsyncHTTP2Connection: Async version for ASGI workers
- Same features as sync version with async/await
- Proper flow control with chunked data sending
- ASGI worker: HTTP/2 support via AsyncHTTP2Connection
- AsyncWorker base: HTTP/2 connection handling
- tornado worker: Add warning that HTTP/2 is not supported
Also exports helper functions from http2 module.
Force early setproctitle initialization by calling getproctitle()
immediately after import. This ensures setproctitle captures the
argv/environ memory layout before systemd.listen_fds() modifies
the environment by removing LISTEN_FDS and LISTEN_PID.
Without this fix, if LISTEN_FDS is the first environment variable,
setproctitle fails to detect argv correctly and silently fails.
Fixes#3430
The wsgi.create() function expects req._expected_100_continue but
UWSGIRequest didn't have this attribute, causing an AttributeError.
Set to False since uWSGI runs behind a frontend server that handles
100-continue negotiation.
Add uWSGI protocol support to ASGI worker
- Implements AsyncUWSGIRequest class extending sync UWSGIRequest to reuse parsing logic with async I/O
- ASGI protocol handler selects between HTTP and uWSGI based on --protocol config option
- Allows gunicorn's ASGI worker to receive requests from nginx using uwsgi_pass directive
- Includes unit tests and Docker integration tests
Add comprehensive Docker integration tests verifying dirty arbiter
lifecycle under realistic conditions:
- Parent death detection via ppid monitoring
- Orphan cleanup on restart
- Dirty arbiter respawning after crash
- Graceful shutdown with SIGTERM
Also fix race condition in manage_workers() by checking self.alive
before spawning new workers during shutdown.
Add ppid monitoring to dirty arbiter's worker monitor loop. If the
main arbiter dies unexpectedly (SIGKILL, crash, OOM), the dirty
arbiter detects the parent change and shuts itself down gracefully.
This complements the existing orphan cleanup on startup.
When the main arbiter crashes and restarts, orphaned dirty arbiters
may continue running. This adds detection and cleanup:
- Add well-known PID file location based on proc_name
- Dirty arbiter writes PID on startup, removes on exit
- Main arbiter checks for orphans on fresh start (not USR2)
- Uses self.proc_name for USR2 compatibility (myapp vs myapp.2)
During USR2 upgrade, old and new dirty arbiters coexist with
separate PID files, preventing the old from removing the new's file.
Add support for streaming responses when dirty app actions return
generators (sync or async). This enables real-time delivery of
incremental results for use cases like LLM token generation.
Features:
- Streaming protocol with chunk/end/error message types
- Worker support for sync and async generators
- Arbiter forwarding of streaming messages
- Deadline-based timeout handling
- Async client streaming API
Protocol:
- Chunk messages (type: "chunk") contain partial data
- End messages (type: "end") signal stream completion
- Error messages can occur mid-stream
New files:
- benchmarks/dirty_streaming.py: Streaming benchmark suite
- tests/dirty/test_*_streaming*.py: Streaming test coverage
- docs/content/dirty.md: Streaming documentation with examples
Replace lock-based request serialization with queue-based approach:
- Each worker now has a dedicated asyncio.Queue and consumer task
- route_request() submits (request, future) to queue and awaits future
- Consumer task processes requests sequentially per worker
- No lock contention - pure async queue operations
Benefits:
- Clearer separation of concerns
- Better visibility into request backlog (queue.qsize())
- Eliminates lock contention under high concurrency
Changes:
- worker_locks dict replaced with worker_queues and worker_consumers
- Added _start_worker_consumer() to create queue and consumer per worker
- Added _execute_on_worker() for actual worker communication
- Updated _cleanup_worker() to cancel consumer tasks
- Updated stop() to cancel all consumers before shutdown
Benchmark results (4 workers, isolated):
- throughput_10ms: 333 req/s, 0 failures
- overload_10ms (200 clients): 334 req/s, 0 failures
- All tests pass with perfect round-robin distribution
1. Split respawning logic from reap_dirty_arbiter() into manage_dirty_arbiter()
to avoid respawning during shutdown/re-exec (follows reap_workers pattern)
2. Reduce public API surface in __all__:
- Keep errors, DirtyApp, client functions as public
- Internal protocol helpers remain importable from submodules
- DirtyArbiter and set_dirty_socket_path kept for gunicorn core
- Use dirty_threads config for thread pool size (default: 1)
- Enforce dirty_timeout at worker level via asyncio.wait_for
- Heartbeat runs independently, not blocked by task execution
- Document thread safety and state persistence in docstrings
Introduce Dirty Arbiters - a separate process pool for executing
long-running, blocking operations (AI model loading, heavy computation)
without blocking HTTP workers. Inspired by Erlang's dirty schedulers.
Key features:
- Completely separate from HTTP workers - can be killed/restarted independently
- Stateful - loaded resources persist in dirty worker memory
- Message-passing IPC via Unix sockets with JSON serialization
- Explicit execute() API from HTTP workers
- Asyncio-based for clean concurrent handling
Architecture:
- DirtyArbiter: manages the dirty worker pool, routes requests
- DirtyWorker: executes functions, maintains state, handles requests
- DirtyClient: sync/async API for HTTP workers to call dirty apps
- DirtyProtocol: length-prefixed JSON messages over Unix sockets
- DirtyApp: base class for dirty applications
Configuration options:
- dirty_apps: list of import paths for dirty applications
- dirty_workers: number of dirty workers (default: 0)
- dirty_timeout: task timeout in seconds (default: 300)
- dirty_graceful_timeout: shutdown timeout (default: 30)
Lifecycle hooks:
- on_dirty_starting(arbiter)
- dirty_post_fork(arbiter, worker)
- dirty_worker_init(worker)
- dirty_worker_exit(arbiter, worker)
Includes comprehensive test suite with 164 tests covering:
- Protocol encoding/decoding
- Worker and arbiter lifecycle
- Client sync/async APIs
- Signal handling
- Error handling and timeouts
- Integration tests
setproctitle causes segfaults on macOS due to fork() safety issues
introduced in newer macOS versions. The mere import of setproctitle
can trigger crashes in forked worker processes.
Fixes#3021
* Python3: refactor returned traceback
Exceptions provide __traceback__ reference since Python 3.0
(and creating cyclic references has not been big deal since Python 2.2)
* --reload: publish entire exception, not just traceback
This is dangerous insofar as the exception text is more
likely to contain secrets than the quoted lines from traceback are.
However, the difference between the two is minor compared to the
primary danger of enabling this on a production machine, so focus
on that instead!
