* ci: Remove failing macos-13 from test matrix
* ci: Add FreeBSD testing workflow
* ci: Document test matrix rationale
* ci: Update cross-platform-actions to v0.32.0 for FreeBSD 14.2 support
* ci: Use FreeBSD 14.1 (14.2 has SSH connectivity issues)
* ci: Switch to vmactions/freebsd-vm for FreeBSD testing
* ci: Fix FreeBSD package names (pip included in Python)
* ci: Simplify FreeBSD matrix and fix package names
* ci: Use specific Python version command on FreeBSD
* ci: Add sqlite3 package for FreeBSD
* tests: Increase signal integration test timeouts for CI
The signal integration tests were flaky in CI environments,
especially FreeBSD VMs, due to 10-second timeouts being too short.
Increased timeouts to 30 seconds to handle slower CI environments.
* tests: Add tests for current signal handling behavior
Add tests for arbiter signal handling:
- TestSignalHandlerRegistration (4 tests): Verify signal handler
registration, pipe creation, SIGCHLD separate handler, and
expected signals list
- TestSignalQueue (4 tests): Test signal queueing, max queue size,
wakeup writes to pipe, and sleep returns on pipe data
- TestReapWorkers (6 tests): Test worker reaping for normal exit,
error exit codes, WORKER_BOOT_ERROR, APP_LOAD_ERROR, signal
termination, and SIGKILL OOM hint
These tests establish baseline coverage before refactoring the
signal handling code for safety and reliability improvements.
* tests: Add tests for SIGHUP reload and worker lifecycle
Add tests for reload and worker management:
- TestSighupReload (3 tests): Verify reload spawns configured number
of workers, calls manage_workers, and logs hang up message
- TestWorkerLifecycle (4 tests): Test spawn_worker adds to WORKERS
dict, kill_worker sends correct signal, murder_workers sends
SIGABRT first then SIGKILL on subsequent timeout
* arbiter: Fix waitpid status parsing using POSIX macros
Use os.WIFEXITED/WEXITSTATUS and os.WIFSIGNALED/WTERMSIG instead
of manual bit shifting for waitpid status interpretation. This
correctly distinguishes between normal exits and signal termination.
The previous code used 'status >> 8' which only worked for normal
exits, and used raw status values for signal detection which was
incorrect.
Fixes part of #3435 and #3056 (signal name display issues)
* arbiter: Change SIGTERM log level to warning
Log signal termination at warning level for expected signals
(SIGTERM, SIGQUIT) since these typically occur during normal
graceful shutdown. SIGKILL remains at error level with the
OOM hint since it indicates abnormal termination.
Fixes#3311, #3050 (SIGTERM logged as error)
* arbiter: Remove logging from SIGCHLD signal handler
Move reap_workers() call from signal handler context to main loop.
The signal handler (now signal_chld) only queues the signal and
wakes up the main loop. The actual reap_workers() is called from
handle_chld() in the main loop where logging is safe.
This fixes potential deadlocks caused by logging from signal
handler context when holding the logging lock.
Fixes#3198, #3004 (logging in signal handlers unsafe, deadlock)
* arbiter: Replace PIPE+select with queue.SimpleQueue
Use queue.SimpleQueue for signal handling instead of PIPE+select.
SimpleQueue is reentrant-safe and can be used from signal handlers.
Changes:
- Remove PIPE-based wakeup mechanism
- Add SIG_QUEUE as SimpleQueue instance
- Add WAKEUP_REQUEST sentinel for non-signal wakeups
- Replace sleep() with wait_for_signals() using queue.get()
- Simplify signal handler to just put_nowait()
- Update main loop to iterate over wait_for_signals()
- Add reap_workers() call in stop() to properly clean up workers
since SIGCHLD is no longer processed during shutdown
This simplifies the code and removes the dependency on select().
Also adds integration tests for signal handling that verify:
- Basic request/response
- Graceful shutdown with SIGTERM/SIGINT
- SIGHUP reload
- Multiple concurrent requests
* arbiter: Wait for old workers on SIGHUP reload
After spawning new workers during reload, wait for old workers to
terminate before returning from reload(). This prevents the issue
where old workers could receive double SIGTERM - once from
manage_workers() and again from the arbiter loop.
The reload now tracks worker_age before spawning, then waits up to
graceful_timeout for workers older than that age to exit.
Fixes#3312, #3274 (SIGHUP can send double SIGTERM)
* arbiter: Log SIGCHLD at debug level
SIGCHLD is received frequently (whenever a worker exits) and doesn't
need to be logged at info level. Log it at debug level to reduce
noise in the logs while still making it available for debugging.
* tests: Fix lint warnings in test_arbiter.py
Add tests for:
- Worker liveness reporting to arbiter via WorkerTmp
- SIGTERM graceful shutdown behavior
- SIGQUIT immediate shutdown behavior
- Worker-arbiter integration (parent death detection, timeout)
- Signal interaction edge cases (multiple signals, ordering)
These tests ensure the gthread worker properly:
- Calls notify() in the main loop for arbiter heartbeat
- Handles SIGTERM by setting alive=False and waking the poller
- Handles SIGQUIT by immediately shutting down the thread pool
- Drains connections during graceful shutdown within timeout
- Cleans up resources properly on exit
Replace RLock-based synchronization with a pipe-based method queue
for lock-free coordination between worker threads and main thread.
Key changes:
- Add PollableMethodQueue class using os.pipe() for wake-up signaling
- Non-blocking pipe (both ends) for BSD compatibility (FreeBSD, OpenBSD)
- Unified event loop using single poller.select() - no more futures.wait()
- Better graceful shutdown with connection draining within grace period
- Rename _keep to keepalived_conns, remove _lock entirely
- Add handle_exit() for SIGTERM, improve handle_quit() for SIGQUIT
- Add set_accept_enabled() for dynamic connection acceptance control
- Add wait_for_and_dispatch_events() with EINTR handling
Performance improvement: ~8% at high concurrency due to reduced
lock contention and non-blocking pipe operations.
Tests: 40 tests covering PollableMethodQueue, graceful shutdown,
keepalive management, error handling, and BSD compatibility.
Fixes#3146Closes#3157
This commit addresses three issues with the gthread worker:
1. Request body handling on keepalive
- Add finish_body() method to Parser to discard unread body bytes
- Call it before returning connections to the poller
- Prevents socket appearing readable due to leftover body
Fixes#3301
2. Timeout reliability with monotonic clock
- Replace time.time() with time.monotonic() in set_timeout()
- Replace time.time() with time.monotonic() in murder_keepalived()
- Prevents timeout issues caused by NTP adjustments
3. SSL error handling
- Move conn.init() from enqueue_req() to handle()
- SSL handshake now runs in worker thread, not main thread
- ENOTCONN errors during ssl_wrap_socket are caught per-connection
- Prevents entire worker crashes on SSL handshake failures
Also adds comprehensive unit tests for the gthread worker.
Closes#3303Closes#3308
Strip whitespace also *after* header field value.
Simply refuse obsolete header folding (a default-off
option to revert is temporarily provided).
While we are at it, explicitly handle recently
introduced http error classes with intended status code.
changes:
- Just follow the new TE specification (https://datatracker.ietf.org/doc/html/rfc9112#name-transfer-encoding)
here and accept to introduce a breaking change.
- gandle multiple TE on one line
** breaking changes ** : invalid headers and position will now return
an error.
New parser rule: refuse HTTP requests where a header field value
contains characters that
a) should never appear there in the first place,
b) might have lead to incorrect treatment in a proxy in front, and
c) might lead to unintended behaviour in applications.
From RFC 9110 section 5.5:
"Field values containing CR, LF, or NUL characters are invalid and
dangerous, due to the varying ways that implementations might parse
and interpret those characters; a recipient of CR, LF, or NUL within
a field value MUST either reject the message or replace each of those
characters with SP before further processing or forwarding of that
message."
chunk extensions are silently ignored before and after this change;
its just the whitespace handling for the case without extensions that matters
applying same strip(WS)->rstrip(BWS) replacement as already done in related cases
half-way fix: could probably reject all BWS cases, rejecting only misplaced ones
Note: This is unrelated to a reverse proxy potentially talking HTTP/3 to clients.
This is about the HTTP protocol version spoken to Gunicorn, which is HTTP/1.0 or HTTP/1.1.
Little legitimate need for processing HTTP 1 requests with ambiguous version numbers.
Broadly refuse.
Co-authored-by: Ben Kallus <benjamin.p.kallus.gr@dartmouth.edu>
Do the validation on the original, not the result from unicode case folding.
Background:
latin-1 0xDF is traditionally uppercased 0x53+0x53 which puts it back in ASCII
If we promise wsgi.input_terminated, we better get it right - or not at all.
* chunked encoding on HTTP <= 1.1
* chunked not last transfer coding
* multiple chinked codings
* any unknown codings (yes, this too! because we do not detect unusual syntax that is still chunked)
* empty coding (plausibly harmless, but not see in real life anyway - refused, for the moment)
Ambiguous mappings open a bottomless pit of "what is user input and what is proxy input" confusion.
Default to what everyone else has been doing for years now, silently drop.
see also https://nginx.org/r/underscores_in_headers
- Unify HEADER_RE and METH_RE
- Replace CRLF with SP during obs-fold processing (See RFC 9112 Section 5.2, last paragraph)
- Stop stripping header names.
- Remove HTAB in OWS in header values that use obs-fold (See RFC 9112 Section 5.2, last paragraph)
- Use fullmatch instead of search, which has problems with empty strings. (See GHSA-68xg-gqqm-vgj8)
- Split proxy protocol line on space only. (See proxy protocol Section 2.1, bullet 3)
- Use fullmatch for method and version (Thank you to Paul Dorn for noticing this.)
- Replace calls to str.strip() with str.strip(' \t')
- Split request line on SP only.
Co-authored-by: Paul Dorn <pajod@users.noreply.github.com>
This change defaults SSLContext to Python's ssl.create_default_context() and
marks ssl_version option as deprecated. The option value will be ignored and
warnign will be printed in stderr.
The ssl_version option was depending on old method of setting TLS min/max
version, which has not worked well anymore with modern Python versions.
Fixes#2223.
Unfortunately, eventlet doesn't implement GreenSocket.sendfile, so we have to do it for it.
Add gevent and eventlet to tox.ini and add tests to make sure we can at least import the workers. Some tests that this actually functions would be nice...
Update the gevent and eventlet setup extras to require the versions that are enforced in their worker modules.
* load application from factory function
Use `ast.parse` to validate that the string passed to the CLI is either
an attribute name or a function call. Use `ast.literal_eval` to parse
any positional and keyword arguments to the function. Call the function
to get the real application.
Co-authored-by: Connor Brinton <connor.brinton@gmail.com>
* test coverage for util.import_app
* document app factory pattern
