gunicorn killing workers repeatedly with no traffic - python-3.x

Question

I have a Flask app running on top of gunicorn which works fine on centos8 and rhel8. I'm trying to bring the app up on an rhel8 derivative which is hardened for FIPS/military compliance. This new os constantly sees workers timing out with no REST traffic offered.
The keepalive files are not having their timestamps updated by the worker threads which leads the arbiter to kill them off. I've added enough debug code to know that much.
I've studied the code and see that gunicorn uses a notify mechanism which relies on updating the permissions on temporary files. I've been looking around for these files but can't seem to find them. I've looked in /tmp (which I believe to be the default) and even set --worker-tmp-dir /tmp/gunicorn with no luck. Both are empty.
Any advice on how to proceed here will be appreciated.
[root#host-172-23-14-219 ~]# ls -ltr /tmp/gunicorn/
total 0
[root#host-172-23-14-219 ~]# ls -latr /tmp/gunicorn/
total 0
These are the messages I see in the debug output. The repeat every 30 seconds which is the default value for timeout. If I change the timeout, the time before being killed increases.
[2021-01-15 16:33:21 +0300] [9120] [CRITICAL] WORKER TIMEOUT (pid:14483)
[2021-01-15 16:33:21 +0300] [9120] [CRITICAL] WORKER TIMEOUT (pid:14485)
[2021-01-15 16:33:21 +0300] [9120] [CRITICAL] WORKER TIMEOUT (pid:14488)
[2021-01-15 16:33:21 +0300] [14483] [INFO] Worker exiting (pid: 14483)
[2021-01-15 16:33:21 +0300] [14485] [INFO] Worker exiting (pid: 14485)
[2021-01-15 16:33:21 +0300] [14488] [INFO] Worker exiting (pid: 14488)
[2021-01-15 16:33:21 +0300] [14782] [INFO] Booting worker with pid: 14782
[2021-01-15 16:33:21 +0300] [9120] [DEBUG] 1 workers
[2021-01-15 16:33:21 +0300] [14783] [INFO] Booting worker with pid: 14783
[2021-01-15 16:33:21 +0300] [14785] [INFO] Booting worker with pid: 14785
[2021-01-15 16:33:21 +0300] [9120] [DEBUG] 3 workers
Complete trace running gunicorn from the command line.
[root#dbaas1 restapi]# /usr/bin/python3 /usr/local/bin/gunicorn --log-level debug --log-syslog --workers 3 --bind 127.0.0.1:5000 -m 007 wsgi:app
[2021-01-18 22:11:23 +0300] [2942060] [DEBUG] Current configuration:
config: None
bind: ['127.0.0.1:5000']
backlog: 2048
workers: 3
worker_class: sync
threads: 1
worker_connections: 1000
max_requests: 0
max_requests_jitter: 0
timeout: 30
graceful_timeout: 30
keepalive: 2
limit_request_line: 4094
limit_request_fields: 100
limit_request_field_size: 8190
reload: False
reload_engine: auto
reload_extra_files: []
spew: False
check_config: False
preload_app: False
sendfile: None
reuse_port: False
chdir: /opt/rbbn/dbaas/src/serviceDiscovery/restapi
daemon: False
raw_env: []
pidfile: None
worker_tmp_dir: None
user: 0
group: 0
umask: 7
initgroups: False
tmp_upload_dir: None
secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
forwarded_allow_ips: ['127.0.0.1']
accesslog: None
disable_redirect_access_to_syslog: False
access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
errorlog: -
loglevel: debug
capture_output: False
logger_class: gunicorn.glogging.Logger
logconfig: None
logconfig_dict: {}
syslog_addr: udp://localhost:514
syslog: True
syslog_prefix: None
syslog_facility: user
enable_stdio_inheritance: False
statsd_host: None
dogstatsd_tags:
statsd_prefix:
proc_name: None
default_proc_name: wsgi:app
pythonpath: None
paste: None
on_starting: <function OnStarting.on_starting at 0x7f13b046be18>
on_reload: <function OnReload.on_reload at 0x7f13b046bf28>
when_ready: <function WhenReady.when_ready at 0x7f13b04860d0>
pre_fork: <function Prefork.pre_fork at 0x7f13b04861e0>
post_fork: <function Postfork.post_fork at 0x7f13b04862f0>
post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f13b0486400>
worker_int: <function WorkerInt.worker_int at 0x7f13b0486510>
worker_abort: <function WorkerAbort.worker_abort at 0x7f13b0486620>
pre_exec: <function PreExec.pre_exec at 0x7f13b0486730>
pre_request: <function PreRequest.pre_request at 0x7f13b0486840>
post_request: <function PostRequest.post_request at 0x7f13b04868c8>
child_exit: <function ChildExit.child_exit at 0x7f13b04869d8>
worker_exit: <function WorkerExit.worker_exit at 0x7f13b0486ae8>
nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f13b0486bf8>
on_exit: <function OnExit.on_exit at 0x7f13b0486d08>
proxy_protocol: False
proxy_allow_ips: ['127.0.0.1']
keyfile: None
certfile: None
ssl_version: 2
cert_reqs: 0
ca_certs: None
suppress_ragged_eofs: True
do_handshake_on_connect: False
ciphers: None
raw_paste_global_conf: []
strip_header_spaces: False
[2021-01-18 22:11:23 +0300] [2942060] [INFO] Starting gunicorn 20.0.4
[2021-01-18 22:11:23 +0300] [2942060] [DEBUG] Arbiter booted
[2021-01-18 22:11:23 +0300] [2942060] [INFO] Listening at: http://127.0.0.1:5000 (2942060)
[2021-01-18 22:11:23 +0300] [2942060] [INFO] Using worker: sync
[2021-01-18 22:11:23 +0300] [2942066] [INFO] Booting worker with pid: 2942066
[2021-01-18 22:11:23 +0300] [2942068] [INFO] Booting worker with pid: 2942068
[2021-01-18 22:11:23 +0300] [2942069] [INFO] Booting worker with pid: 2942069
[2021-01-18 22:11:23 +0300] [2942060] [DEBUG] 3 workers
[2021-01-18 22:11:53 +0300] [2942060] [CRITICAL] WORKER TIMEOUT (pid:2942066)
[2021-01-18 22:11:53 +0300] [2942060] [CRITICAL] WORKER TIMEOUT (pid:2942068)
[2021-01-18 22:11:53 +0300] [2942060] [CRITICAL] WORKER TIMEOUT (pid:2942069)
[2021-01-18 22:11:53 +0300] [2942066] [INFO] Worker exiting (pid: 2942066)
[2021-01-18 22:11:53 +0300] [2942068] [INFO] Worker exiting (pid: 2942068)
[2021-01-18 22:11:53 +0300] [2942069] [INFO] Worker exiting (pid: 2942069)
[2021-01-18 22:11:53 +0300] [2942357] [INFO] Booting worker with pid: 2942357
[2021-01-18 22:11:53 +0300] [2942060] [DEBUG] 1 workers
[2021-01-18 22:11:53 +0300] [2942359] [INFO] Booting worker with pid: 2942359
[2021-01-18 22:11:53 +0300] [2942360] [INFO] Booting worker with pid: 2942360
[2021-01-18 22:11:53 +0300] [2942060] [DEBUG] 3 workers

Answer 1

Running into a similar problem - everything works fine on Ubuntu 18.04, but fails consistently with similar errors / logging on Ubuntu 22.04.
Try increasing timeout higher than 30 in gunicorn config / settings.
Changing the timeout to 90 was sufficient for me to get things working again - my best guess is that the initial load-in of the Flask app takes longer on some machines and so it's hitting this limit.