Add tests to verify that when multiple dirty apps are configured,
messages are correctly routed to the appropriate app based on app_path.
New files:
- tests/support_dirty_apps.py: CounterApp and EchoApp test apps
- tests/dirty/test_multi_app_routing.py: 13 routing tests covering
app loading, routing, state separation, error handling, and
concurrent requests
Replace lock-based request serialization with queue-based approach:
- Each worker now has a dedicated asyncio.Queue and consumer task
- route_request() submits (request, future) to queue and awaits future
- Consumer task processes requests sequentially per worker
- No lock contention - pure async queue operations
Benefits:
- Clearer separation of concerns
- Better visibility into request backlog (queue.qsize())
- Eliminates lock contention under high concurrency
Changes:
- worker_locks dict replaced with worker_queues and worker_consumers
- Added _start_worker_consumer() to create queue and consumer per worker
- Added _execute_on_worker() for actual worker communication
- Updated _cleanup_worker() to cancel consumer tasks
- Updated stop() to cancel all consumers before shutdown
Benchmark results (4 workers, isolated):
- throughput_10ms: 333 req/s, 0 failures
- overload_10ms (200 clients): 334 req/s, 0 failures
- All tests pass with perfect round-robin distribution
- Use dirty_threads config for thread pool size (default: 1)
- Enforce dirty_timeout at worker level via asyncio.wait_for
- Heartbeat runs independently, not blocked by task execution
- Document thread safety and state persistence in docstrings
Introduce Dirty Arbiters - a separate process pool for executing
long-running, blocking operations (AI model loading, heavy computation)
without blocking HTTP workers. Inspired by Erlang's dirty schedulers.
Key features:
- Completely separate from HTTP workers - can be killed/restarted independently
- Stateful - loaded resources persist in dirty worker memory
- Message-passing IPC via Unix sockets with JSON serialization
- Explicit execute() API from HTTP workers
- Asyncio-based for clean concurrent handling
Architecture:
- DirtyArbiter: manages the dirty worker pool, routes requests
- DirtyWorker: executes functions, maintains state, handles requests
- DirtyClient: sync/async API for HTTP workers to call dirty apps
- DirtyProtocol: length-prefixed JSON messages over Unix sockets
- DirtyApp: base class for dirty applications
Configuration options:
- dirty_apps: list of import paths for dirty applications
- dirty_workers: number of dirty workers (default: 0)
- dirty_timeout: task timeout in seconds (default: 300)
- dirty_graceful_timeout: shutdown timeout (default: 30)
Lifecycle hooks:
- on_dirty_starting(arbiter)
- dirty_post_fork(arbiter, worker)
- dirty_worker_init(worker)
- dirty_worker_exit(arbiter, worker)
Includes comprehensive test suite with 164 tests covering:
- Protocol encoding/decoding
- Worker and arbiter lifecycle
- Client sync/async APIs
- Signal handling
- Error handling and timeouts
- Integration tests
The CIDR network support added in 24.1.0 changed forwarded_allow_ips
and proxy_allow_ips from string lists to ipaddress.ip_network objects.
This broke external tools like uvicorn that expect strings.
This fix validates IP/CIDR format during config parsing but keeps the
string representation. Network objects are cached in Config methods
(forwarded_allow_networks() and proxy_allow_networks()) for efficient
IP checking without repeated conversions.
Also uses strict mode for ip_network validation to detect mistakes like
192.168.1.1/24 where host bits are set (should be 192.168.1.0/24).
Fixes#3458
Extend --proxy-protocol to accept version values (off, v1, v2, auto) instead
of being boolean-only. This allows explicit control over which PROXY protocol
versions are accepted.
Changes:
- Add InvalidProxyHeader exception for v2 binary header errors
- Add validate_proxy_protocol() validator with backwards compatibility
- Update ProxyProtocol setting with nargs="?" and const="auto"
- Add PROXY v2 constants (PP_V2_SIGNATURE, PPCommand, PPFamily, PPProtocol)
- Add _parse_proxy_protocol_v1() and _parse_proxy_protocol_v2() methods
- Update both sync (message.py) and async (asgi/message.py) parsers
- Add hex escape handling in treq.py for v2 binary test data
- Add test cases for v2 TCPv4 and TCPv6
Backwards compatible: --proxy-protocol alone (or True) maps to "auto".
Closes#2912
On keepalive connections, finish_request() sets the socket to non-blocking
for selector registration. When the connection is reused, handle() calls
conn.init() which returns early (already initialized) without restoring
blocking mode. This caused SSLWantReadError when WSGI apps read the
request body on SSL connections.
Fix by explicitly setting blocking mode at the start of handle().
Fixes#3448
Use Python's ipaddress module to support IP networks in allow lists.
Individual IP addresses are converted to /32 (IPv4) or /128 (IPv6)
networks. CIDR notation (e.g., 192.168.0.0/16) is now supported.
Fixes#1485Closes#2390
SIGTERM is expected during graceful shutdown and reload operations.
Logging it as warning level causes unnecessary noise in error logs.
SIGKILL remains at error level (suggests OOM), other signals at warning.
Closes#3094
The unread method was incorrectly appending data to the end of the
buffer instead of prepending it to the beginning. This caused issues
when reading partial data and then unreading it.
Closes#2915Closes#2346
The finish_body() function can raise ssl.SSLWantReadError when
discarding unread request body data on SSL connections. This causes
TLS requests to fail intermittently with "Invalid request" errors.
Handle SSLWantReadError by treating it as "no more data to read".
This is safe because finish_body() only discards leftover data before
keepalive - if SSL says "need to wait for more data", there's nothing
left to discard.
Fixes#3448
Upgrade minimum eventlet version to 0.40.3 to address security
vulnerabilities:
- CVE-2021-21419 (Moderate 6.9): Websocket memory exhaustion via
large/compressed frames (fixed in 0.31.0)
- CVE-2025-58068 (Moderate 6.3): HTTP Request Smuggling via improper
trailer handling (fixed in 0.40.3)
Also restructure module to call monkey_patch() at import time for
better patching coverage, while keeping hubs.use_hub() in the worker's
patch() method since it creates OS resources that don't survive fork.
Add comprehensive tests for the eventlet worker.
Address CVE-2023-41419 (Critical - remote privilege escalation via
WSGIServer) by requiring gevent 23.9.0 or higher.
Changes:
- Update minimum gevent version from 1.4.0 to 23.9.0
- Remove legacy server.kill() code path (gevent < 1.0)
- Update documentation to reflect new version requirement
- Add comprehensive tests for gevent worker
Update minimum Tornado version to 6.5.0 to address:
- CVE-2024-52804 (Medium): HTTP Cookie Parsing DoS
- CVE-2025-47287 (High 7.5): Multipart/Form-Data Parser DoS
This simplifies the tornado worker by removing legacy code paths
for Tornado < 5.0 and < 6.0, reducing the codebase by ~30%.
Changes:
- pyproject.toml: Update tornado requirement to >=6.5.0
- gtornado.py: Remove TORNADO5 constant and legacy code paths
- tornadoapp.py: Update example to use async/await syntax
- test_gtornado.py: Add comprehensive test suite
Add support for the uWSGI binary protocol, enabling gunicorn to work
with nginx's uwsgi_pass directive.
New module gunicorn/uwsgi/ with:
- UWSGIRequest: Parses 4-byte binary header and key-value vars block
- UWSGIParser: Protocol parser following existing Parser pattern
- Error classes: InvalidUWSGIHeader, UnsupportedModifier, ForbiddenUWSGIRequest
New configuration options:
- --protocol: Select 'http' (default) or 'uwsgi' protocol
- --uwsgi-allow-from: IP allowlist for uWSGI requests (default: localhost)
Worker integration via get_parser() factory in gunicorn/http/__init__.py,
updates to sync, gthread, and base_async workers.
Example nginx config:
upstream gunicorn {
server 127.0.0.1:8000;
}
location / {
uwsgi_pass gunicorn;
include uwsgi_params;
}
Add a new ASGI worker type that provides native async support using
gunicorn's own HTTP parsing infrastructure adapted for asyncio.
Features:
- HTTP/1.1 with keepalive support
- WebSocket connections (RFC 6455)
- ASGI lifespan protocol for startup/shutdown hooks
- Optional uvloop support for improved performance
- Full proxy protocol support (inherited from gunicorn)
New configuration options:
- --asgi-loop: Event loop selection (auto/asyncio/uvloop)
- --asgi-lifespan: Lifespan protocol control (auto/on/off)
- --root-path: ASGI root path for reverse proxy setups
Usage: gunicorn -k asgi myapp:app
* ci: Remove failing macos-13 from test matrix
* ci: Add FreeBSD testing workflow
* ci: Document test matrix rationale
* ci: Update cross-platform-actions to v0.32.0 for FreeBSD 14.2 support
* ci: Use FreeBSD 14.1 (14.2 has SSH connectivity issues)
* ci: Switch to vmactions/freebsd-vm for FreeBSD testing
* ci: Fix FreeBSD package names (pip included in Python)
* ci: Simplify FreeBSD matrix and fix package names
* ci: Use specific Python version command on FreeBSD
* ci: Add sqlite3 package for FreeBSD
* tests: Increase signal integration test timeouts for CI
The signal integration tests were flaky in CI environments,
especially FreeBSD VMs, due to 10-second timeouts being too short.
Increased timeouts to 30 seconds to handle slower CI environments.
* tests: Add tests for current signal handling behavior
Add tests for arbiter signal handling:
- TestSignalHandlerRegistration (4 tests): Verify signal handler
registration, pipe creation, SIGCHLD separate handler, and
expected signals list
- TestSignalQueue (4 tests): Test signal queueing, max queue size,
wakeup writes to pipe, and sleep returns on pipe data
- TestReapWorkers (6 tests): Test worker reaping for normal exit,
error exit codes, WORKER_BOOT_ERROR, APP_LOAD_ERROR, signal
termination, and SIGKILL OOM hint
These tests establish baseline coverage before refactoring the
signal handling code for safety and reliability improvements.
* tests: Add tests for SIGHUP reload and worker lifecycle
Add tests for reload and worker management:
- TestSighupReload (3 tests): Verify reload spawns configured number
of workers, calls manage_workers, and logs hang up message
- TestWorkerLifecycle (4 tests): Test spawn_worker adds to WORKERS
dict, kill_worker sends correct signal, murder_workers sends
SIGABRT first then SIGKILL on subsequent timeout
* arbiter: Fix waitpid status parsing using POSIX macros
Use os.WIFEXITED/WEXITSTATUS and os.WIFSIGNALED/WTERMSIG instead
of manual bit shifting for waitpid status interpretation. This
correctly distinguishes between normal exits and signal termination.
The previous code used 'status >> 8' which only worked for normal
exits, and used raw status values for signal detection which was
incorrect.
Fixes part of #3435 and #3056 (signal name display issues)
* arbiter: Change SIGTERM log level to warning
Log signal termination at warning level for expected signals
(SIGTERM, SIGQUIT) since these typically occur during normal
graceful shutdown. SIGKILL remains at error level with the
OOM hint since it indicates abnormal termination.
Fixes#3311, #3050 (SIGTERM logged as error)
* arbiter: Remove logging from SIGCHLD signal handler
Move reap_workers() call from signal handler context to main loop.
The signal handler (now signal_chld) only queues the signal and
wakes up the main loop. The actual reap_workers() is called from
handle_chld() in the main loop where logging is safe.
This fixes potential deadlocks caused by logging from signal
handler context when holding the logging lock.
Fixes#3198, #3004 (logging in signal handlers unsafe, deadlock)
* arbiter: Replace PIPE+select with queue.SimpleQueue
Use queue.SimpleQueue for signal handling instead of PIPE+select.
SimpleQueue is reentrant-safe and can be used from signal handlers.
Changes:
- Remove PIPE-based wakeup mechanism
- Add SIG_QUEUE as SimpleQueue instance
- Add WAKEUP_REQUEST sentinel for non-signal wakeups
- Replace sleep() with wait_for_signals() using queue.get()
- Simplify signal handler to just put_nowait()
- Update main loop to iterate over wait_for_signals()
- Add reap_workers() call in stop() to properly clean up workers
since SIGCHLD is no longer processed during shutdown
This simplifies the code and removes the dependency on select().
Also adds integration tests for signal handling that verify:
- Basic request/response
- Graceful shutdown with SIGTERM/SIGINT
- SIGHUP reload
- Multiple concurrent requests
* arbiter: Wait for old workers on SIGHUP reload
After spawning new workers during reload, wait for old workers to
terminate before returning from reload(). This prevents the issue
where old workers could receive double SIGTERM - once from
manage_workers() and again from the arbiter loop.
The reload now tracks worker_age before spawning, then waits up to
graceful_timeout for workers older than that age to exit.
Fixes#3312, #3274 (SIGHUP can send double SIGTERM)
* arbiter: Log SIGCHLD at debug level
SIGCHLD is received frequently (whenever a worker exits) and doesn't
need to be logged at info level. Log it at debug level to reduce
noise in the logs while still making it available for debugging.
* tests: Fix lint warnings in test_arbiter.py
Add tests for:
- Worker liveness reporting to arbiter via WorkerTmp
- SIGTERM graceful shutdown behavior
- SIGQUIT immediate shutdown behavior
- Worker-arbiter integration (parent death detection, timeout)
- Signal interaction edge cases (multiple signals, ordering)
These tests ensure the gthread worker properly:
- Calls notify() in the main loop for arbiter heartbeat
- Handles SIGTERM by setting alive=False and waking the poller
- Handles SIGQUIT by immediately shutting down the thread pool
- Drains connections during graceful shutdown within timeout
- Cleans up resources properly on exit
Replace RLock-based synchronization with a pipe-based method queue
for lock-free coordination between worker threads and main thread.
Key changes:
- Add PollableMethodQueue class using os.pipe() for wake-up signaling
- Non-blocking pipe (both ends) for BSD compatibility (FreeBSD, OpenBSD)
- Unified event loop using single poller.select() - no more futures.wait()
- Better graceful shutdown with connection draining within grace period
- Rename _keep to keepalived_conns, remove _lock entirely
- Add handle_exit() for SIGTERM, improve handle_quit() for SIGQUIT
- Add set_accept_enabled() for dynamic connection acceptance control
- Add wait_for_and_dispatch_events() with EINTR handling
Performance improvement: ~8% at high concurrency due to reduced
lock contention and non-blocking pipe operations.
Tests: 40 tests covering PollableMethodQueue, graceful shutdown,
keepalive management, error handling, and BSD compatibility.
Fixes#3146Closes#3157
This commit addresses three issues with the gthread worker:
1. Request body handling on keepalive
- Add finish_body() method to Parser to discard unread body bytes
- Call it before returning connections to the poller
- Prevents socket appearing readable due to leftover body
Fixes#3301
2. Timeout reliability with monotonic clock
- Replace time.time() with time.monotonic() in set_timeout()
- Replace time.time() with time.monotonic() in murder_keepalived()
- Prevents timeout issues caused by NTP adjustments
3. SSL error handling
- Move conn.init() from enqueue_req() to handle()
- SSL handshake now runs in worker thread, not main thread
- ENOTCONN errors during ssl_wrap_socket are caught per-connection
- Prevents entire worker crashes on SSL handshake failures
Also adds comprehensive unit tests for the gthread worker.
Closes#3303Closes#3308
Strip whitespace also *after* header field value.
Simply refuse obsolete header folding (a default-off
option to revert is temporarily provided).
While we are at it, explicitly handle recently
introduced http error classes with intended status code.
changes:
- Just follow the new TE specification (https://datatracker.ietf.org/doc/html/rfc9112#name-transfer-encoding)
here and accept to introduce a breaking change.
- gandle multiple TE on one line
** breaking changes ** : invalid headers and position will now return
an error.
New parser rule: refuse HTTP requests where a header field value
contains characters that
a) should never appear there in the first place,
b) might have lead to incorrect treatment in a proxy in front, and
c) might lead to unintended behaviour in applications.
From RFC 9110 section 5.5:
"Field values containing CR, LF, or NUL characters are invalid and
dangerous, due to the varying ways that implementations might parse
and interpret those characters; a recipient of CR, LF, or NUL within
a field value MUST either reject the message or replace each of those
characters with SP before further processing or forwarding of that
message."
chunk extensions are silently ignored before and after this change;
its just the whitespace handling for the case without extensions that matters
applying same strip(WS)->rstrip(BWS) replacement as already done in related cases
half-way fix: could probably reject all BWS cases, rejecting only misplaced ones
Note: This is unrelated to a reverse proxy potentially talking HTTP/3 to clients.
This is about the HTTP protocol version spoken to Gunicorn, which is HTTP/1.0 or HTTP/1.1.
Little legitimate need for processing HTTP 1 requests with ambiguous version numbers.
Broadly refuse.
Co-authored-by: Ben Kallus <benjamin.p.kallus.gr@dartmouth.edu>
Do the validation on the original, not the result from unicode case folding.
Background:
latin-1 0xDF is traditionally uppercased 0x53+0x53 which puts it back in ASCII
If we promise wsgi.input_terminated, we better get it right - or not at all.
* chunked encoding on HTTP <= 1.1
* chunked not last transfer coding
* multiple chinked codings
* any unknown codings (yes, this too! because we do not detect unusual syntax that is still chunked)
* empty coding (plausibly harmless, but not see in real life anyway - refused, for the moment)
Ambiguous mappings open a bottomless pit of "what is user input and what is proxy input" confusion.
Default to what everyone else has been doing for years now, silently drop.
see also https://nginx.org/r/underscores_in_headers
- Unify HEADER_RE and METH_RE
- Replace CRLF with SP during obs-fold processing (See RFC 9112 Section 5.2, last paragraph)
- Stop stripping header names.
- Remove HTAB in OWS in header values that use obs-fold (See RFC 9112 Section 5.2, last paragraph)
- Use fullmatch instead of search, which has problems with empty strings. (See GHSA-68xg-gqqm-vgj8)
- Split proxy protocol line on space only. (See proxy protocol Section 2.1, bullet 3)
- Use fullmatch for method and version (Thank you to Paul Dorn for noticing this.)
- Replace calls to str.strip() with str.strip(' \t')
- Split request line on SP only.
Co-authored-by: Paul Dorn <pajod@users.noreply.github.com>
