docs: a warning for disk-backed --worker-tmp-dir #1305 (#1329)

* docs: a warning for disk-backed --worker-tmp-dir #1305

* docs: move os.fchmod issue solution to FAQ

* docs: improve worker_tmp_dir setting docs

docs/source/faq.rst

@@ -160,3 +160,38 @@ How do I fix Django reporting an ``ImproperlyConfigured`` error?
 
 With asynchronous workers, creating URLs with the ``reverse`` function of
 ``django.core.urlresolvers`` may fail. Use ``reverse_lazy`` instead.
+
+How do I avoid Gunicorn excessively blocking in ``os.fchmod``?
+--------------------------------------------------------------
+
+The current heartbeat system involves calling ``os.fchmod`` on temporary file 
+handlers and may block a worker for arbitrary time if the directory is on a 
+disk-backed filesystem. For example, by default ``/tmp`` is not mounted as 
+``tmpfs`` in Ubuntu; in AWS an EBS root instance volume may sometimes hang for 
+half a minute and during this time Gunicorn workers may completely block in 
+``os.fchmod``. ``os.fchmod`` may introduce extra delays if the disk gets full. 
+Also Gunicon may refuse to start if it can't create the files when the disk is 
+full.
+
+Currently to avoid these problems you can create a ``tmpfs`` mount (for a new 
+directory or for ``/tmp``) and pass its path to ``--worker-tmp-dir``. First, 
+check whether your ``/tmp`` is disk-backed or RAM-backed::
+
+    $ df /tmp
+    Filesystem     1K-blocks    Used Available Use% Mounted on
+    /dev/xvda1           ...     ...       ...  ... /
+
+No luck. Let's create a new ``tmpfs`` mount::
+
+    sudo cp /etc/fstab /etc/fstab.orig
+    sudo mkdir /mem
+    echo 'tmpfs       /mem tmpfs defaults,size=64m,mode=1777,noatime,comment=for-gunicorn 0 0' | sudo tee -a /etc/fstab
+    sudo mount /mem
+
+Check the result::
+
+    $ df /mem
+    Filesystem     1K-blocks  Used Available Use% Mounted on
+    tmpfs              65536     0     65536   0% /mem
+
+Now you can set ``--worker-tmp-dir /mem``.

gunicorn/config.py

@@ -981,6 +981,11 @@ class WorkerTmpDir(Setting):
         A directory to use for the worker heartbeat temporary file.
 
         If not set, the default temporary directory will be used.
+
+        Warning: the current heartbeat system involves calling ``os.fchmod`` on
+        temporary file handlers and may block a worker for arbitrary time if the
+        directory is on a disk-backed filesystem. See the :ref:`faq` for more
+        detailed information and a solution for avoiding this problem.
         """