* tests: Add tests for current signal handling behavior
Add tests for arbiter signal handling:
- TestSignalHandlerRegistration (4 tests): Verify signal handler
registration, pipe creation, SIGCHLD separate handler, and
expected signals list
- TestSignalQueue (4 tests): Test signal queueing, max queue size,
wakeup writes to pipe, and sleep returns on pipe data
- TestReapWorkers (6 tests): Test worker reaping for normal exit,
error exit codes, WORKER_BOOT_ERROR, APP_LOAD_ERROR, signal
termination, and SIGKILL OOM hint
These tests establish baseline coverage before refactoring the
signal handling code for safety and reliability improvements.
* tests: Add tests for SIGHUP reload and worker lifecycle
Add tests for reload and worker management:
- TestSighupReload (3 tests): Verify reload spawns configured number
of workers, calls manage_workers, and logs hang up message
- TestWorkerLifecycle (4 tests): Test spawn_worker adds to WORKERS
dict, kill_worker sends correct signal, murder_workers sends
SIGABRT first then SIGKILL on subsequent timeout
* arbiter: Fix waitpid status parsing using POSIX macros
Use os.WIFEXITED/WEXITSTATUS and os.WIFSIGNALED/WTERMSIG instead
of manual bit shifting for waitpid status interpretation. This
correctly distinguishes between normal exits and signal termination.
The previous code used 'status >> 8' which only worked for normal
exits, and used raw status values for signal detection which was
incorrect.
Fixes part of #3435 and #3056 (signal name display issues)
* arbiter: Change SIGTERM log level to warning
Log signal termination at warning level for expected signals
(SIGTERM, SIGQUIT) since these typically occur during normal
graceful shutdown. SIGKILL remains at error level with the
OOM hint since it indicates abnormal termination.
Fixes#3311, #3050 (SIGTERM logged as error)
* arbiter: Remove logging from SIGCHLD signal handler
Move reap_workers() call from signal handler context to main loop.
The signal handler (now signal_chld) only queues the signal and
wakes up the main loop. The actual reap_workers() is called from
handle_chld() in the main loop where logging is safe.
This fixes potential deadlocks caused by logging from signal
handler context when holding the logging lock.
Fixes#3198, #3004 (logging in signal handlers unsafe, deadlock)
* arbiter: Replace PIPE+select with queue.SimpleQueue
Use queue.SimpleQueue for signal handling instead of PIPE+select.
SimpleQueue is reentrant-safe and can be used from signal handlers.
Changes:
- Remove PIPE-based wakeup mechanism
- Add SIG_QUEUE as SimpleQueue instance
- Add WAKEUP_REQUEST sentinel for non-signal wakeups
- Replace sleep() with wait_for_signals() using queue.get()
- Simplify signal handler to just put_nowait()
- Update main loop to iterate over wait_for_signals()
- Add reap_workers() call in stop() to properly clean up workers
since SIGCHLD is no longer processed during shutdown
This simplifies the code and removes the dependency on select().
Also adds integration tests for signal handling that verify:
- Basic request/response
- Graceful shutdown with SIGTERM/SIGINT
- SIGHUP reload
- Multiple concurrent requests
* arbiter: Wait for old workers on SIGHUP reload
After spawning new workers during reload, wait for old workers to
terminate before returning from reload(). This prevents the issue
where old workers could receive double SIGTERM - once from
manage_workers() and again from the arbiter loop.
The reload now tracks worker_age before spawning, then waits up to
graceful_timeout for workers older than that age to exit.
Fixes#3312, #3274 (SIGHUP can send double SIGTERM)
* arbiter: Log SIGCHLD at debug level
SIGCHLD is received frequently (whenever a worker exits) and doesn't
need to be logged at info level. Log it at debug level to reduce
noise in the logs while still making it available for debugging.
* tests: Fix lint warnings in test_arbiter.py
Replace RLock-based synchronization with a pipe-based method queue
for lock-free coordination between worker threads and main thread.
Key changes:
- Add PollableMethodQueue class using os.pipe() for wake-up signaling
- Non-blocking pipe (both ends) for BSD compatibility (FreeBSD, OpenBSD)
- Unified event loop using single poller.select() - no more futures.wait()
- Better graceful shutdown with connection draining within grace period
- Rename _keep to keepalived_conns, remove _lock entirely
- Add handle_exit() for SIGTERM, improve handle_quit() for SIGQUIT
- Add set_accept_enabled() for dynamic connection acceptance control
- Add wait_for_and_dispatch_events() with EINTR handling
Performance improvement: ~8% at high concurrency due to reduced
lock contention and non-blocking pipe operations.
Tests: 40 tests covering PollableMethodQueue, graceful shutdown,
keepalive management, error handling, and BSD compatibility.
Fixes#3146Closes#3157
This commit addresses three issues with the gthread worker:
1. Request body handling on keepalive
- Add finish_body() method to Parser to discard unread body bytes
- Call it before returning connections to the poller
- Prevents socket appearing readable due to leftover body
Fixes#3301
2. Timeout reliability with monotonic clock
- Replace time.time() with time.monotonic() in set_timeout()
- Replace time.time() with time.monotonic() in murder_keepalived()
- Prevents timeout issues caused by NTP adjustments
3. SSL error handling
- Move conn.init() from enqueue_req() to handle()
- SSL handshake now runs in worker thread, not main thread
- ENOTCONN errors during ssl_wrap_socket are caught per-connection
- Prevents entire worker crashes on SSL handshake failures
Also adds comprehensive unit tests for the gthread worker.
Closes#3303Closes#3308
Strip whitespace also *after* header field value.
Simply refuse obsolete header folding (a default-off
option to revert is temporarily provided).
While we are at it, explicitly handle recently
introduced http error classes with intended status code.
changes:
- Just follow the new TE specification (https://datatracker.ietf.org/doc/html/rfc9112#name-transfer-encoding)
here and accept to introduce a breaking change.
- gandle multiple TE on one line
** breaking changes ** : invalid headers and position will now return
an error.
New parser rule: refuse HTTP requests where a header field value
contains characters that
a) should never appear there in the first place,
b) might have lead to incorrect treatment in a proxy in front, and
c) might lead to unintended behaviour in applications.
From RFC 9110 section 5.5:
"Field values containing CR, LF, or NUL characters are invalid and
dangerous, due to the varying ways that implementations might parse
and interpret those characters; a recipient of CR, LF, or NUL within
a field value MUST either reject the message or replace each of those
characters with SP before further processing or forwarding of that
message."
since 3.3: EnvironmentError, IOError, socket.error and select.error are merged into IOError.
They may now return a more specific subclass - which this commit does not utilize yet.
chunk extensions are silently ignored before and after this change;
its just the whitespace handling for the case without extensions that matters
applying same strip(WS)->rstrip(BWS) replacement as already done in related cases
half-way fix: could probably reject all BWS cases, rejecting only misplaced ones
Note: This is unrelated to a reverse proxy potentially talking HTTP/3 to clients.
This is about the HTTP protocol version spoken to Gunicorn, which is HTTP/1.0 or HTTP/1.1.
Little legitimate need for processing HTTP 1 requests with ambiguous version numbers.
Broadly refuse.
Co-authored-by: Ben Kallus <benjamin.p.kallus.gr@dartmouth.edu>
Do the validation on the original, not the result from unicode case folding.
Background:
latin-1 0xDF is traditionally uppercased 0x53+0x53 which puts it back in ASCII
In common configuration unlikely a big security problem in itself
you are just fooling the remote about https.
However, it is offers an oracle for otherwise invisible proxy request headers,
so it might help exploiting other vulnerabilities.
If we promise wsgi.input_terminated, we better get it right - or not at all.
* chunked encoding on HTTP <= 1.1
* chunked not last transfer coding
* multiple chinked codings
* any unknown codings (yes, this too! because we do not detect unusual syntax that is still chunked)
* empty coding (plausibly harmless, but not see in real life anyway - refused, for the moment)
Ambiguous mappings open a bottomless pit of "what is user input and what is proxy input" confusion.
Default to what everyone else has been doing for years now, silently drop.
see also https://nginx.org/r/underscores_in_headers
