diff --git a/docs/source/faq.rst b/docs/source/faq.rst index dc71f051..b9e24610 100644 --- a/docs/source/faq.rst +++ b/docs/source/faq.rst @@ -160,3 +160,38 @@ How do I fix Django reporting an ``ImproperlyConfigured`` error? With asynchronous workers, creating URLs with the ``reverse`` function of ``django.core.urlresolvers`` may fail. Use ``reverse_lazy`` instead. + +How do I avoid Gunicorn excessively blocking in ``os.fchmod``? +-------------------------------------------------------------- + +The current heartbeat system involves calling ``os.fchmod`` on temporary file +handlers and may block a worker for arbitrary time if the directory is on a +disk-backed filesystem. For example, by default ``/tmp`` is not mounted as +``tmpfs`` in Ubuntu; in AWS an EBS root instance volume may sometimes hang for +half a minute and during this time Gunicorn workers may completely block in +``os.fchmod``. ``os.fchmod`` may introduce extra delays if the disk gets full. +Also Gunicon may refuse to start if it can't create the files when the disk is +full. + +Currently to avoid these problems you can create a ``tmpfs`` mount (for a new +directory or for ``/tmp``) and pass its path to ``--worker-tmp-dir``. First, +check whether your ``/tmp`` is disk-backed or RAM-backed:: + + $ df /tmp + Filesystem 1K-blocks Used Available Use% Mounted on + /dev/xvda1 ... ... ... ... / + +No luck. Let's create a new ``tmpfs`` mount:: + + sudo cp /etc/fstab /etc/fstab.orig + sudo mkdir /mem + echo 'tmpfs /mem tmpfs defaults,size=64m,mode=1777,noatime,comment=for-gunicorn 0 0' | sudo tee -a /etc/fstab + sudo mount /mem + +Check the result:: + + $ df /mem + Filesystem 1K-blocks Used Available Use% Mounted on + tmpfs 65536 0 65536 0% /mem + +Now you can set ``--worker-tmp-dir /mem``. diff --git a/gunicorn/config.py b/gunicorn/config.py index 8369d974..c836fcfe 100644 --- a/gunicorn/config.py +++ b/gunicorn/config.py @@ -981,6 +981,11 @@ class WorkerTmpDir(Setting): A directory to use for the worker heartbeat temporary file. If not set, the default temporary directory will be used. + + Warning: the current heartbeat system involves calling ``os.fchmod`` on + temporary file handlers and may block a worker for arbitrary time if the + directory is on a disk-backed filesystem. See the :ref:`faq` for more + detailed information and a solution for avoiding this problem. """