Daemonize celery with systemd on centos 7 - python-3.x

Question

I'm tying to run celery 4.3.0 as a daemon using systemd but it gave me this error:
It starts workers but stop them quickly. However, I can run a worker manually by typing a command like:
celery -A bin.proj.celery_app worker1
celery.service - Celery Service
Loaded: loaded (/etc/systemd/system/celery.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Fri 2020-04-10 08:39:59 CEST; 35s ago
Process: 19276 ExecStop=/bin/sh -c ${CELERY_BIN} multi stopwait ${CELERYD_NODES} --
pidfile=${CELERYD_PID_FILE} (code=exited, status=0/SUCCESS)
Process: 19245 ExecStart=/bin/sh -c ${CELERY_BIN} multi start ${CELERYD_NODES} -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} ${CELERYD_OPTS} (code=exited, status=0/SUCCESS)
Main PID: 19245 (code=exited, status=0/SUCCESS)
Apr 10 08:39:58 prod1 sh[19276]: > Stopping nodes...
Apr 10 08:39:58 prod1 sh[19276]: > worker1#prod1: TERM -> 19254
Apr 10 08:39:58 prod1 sh[19276]: > worker2#prod1: TERM -> 19264
Apr 10 08:39:58 prod1 sh[19276]: > Waiting for 2 nodes -> 19254, 19264......
Apr 10 08:39:58 prod1 sh[19276]: > worker1#prod1: OK
Apr 10 08:39:58 prod1 sh[19276]: > worker1#prod1: DOWN
Apr 10 08:39:58 prod1 sh[19276]: > Waiting for 2 nodes -> None, None....
Apr 10 08:39:58 prod1 sh[19276]: > worker2#prod1: OK
Apr 10 08:39:58 prod1 sh[19276]: > worker2#prod1: DOWN
Apr 10 08:39:58 prod1 sh[19276]: > Waiting for 1 node -> None...
Here is the celery.service:
[Unit]
Description=Celery Service
After=network.target
[Service]
type=forking
User=ssc
Group=ssc
EnvironmentFile=/etc/conf/celery
WorkingDirectory=/home/ssc/Mr.Robot/
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} --pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} restart ${CELERYD_NODES} -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} ${CELERYD_OPTS}'
[Install]
WantedBy=multi-user.target
Here is the config file associated:
# Name of nodes to start
# here we have a single node
#CELERYD_NODES="w1"
# or we could have three nodes:
CELERYD_NODES="worker1 worker2 worker3"
# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"
# App instance to use
# comment out this line if you don't use an app
CELERY_APP="bin.proj.celery_app"
# or fully qualified:
#CELERY_APP="proj.tasks:app"
# How to call manage.py
CELERYD_MULTI="multi"
# Extra command-line arguments to the worker
CELERYD_OPTS="--loglevel=INFO --time-limit=300 --concurrency=3"
# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
# and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
# you may wish to add these options for Celery Beat
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"
CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"
And the app file:
import sys,os
sys.path.append(os.getcwd())
from celery import Celery
from bin.proj.utils import create_rotating_log
# initialize logger
global logger
logger = create_rotating_log('./log/activity.log')
# initialize celery
try:
app = Celery('celery_app',
broker="".join(['redis://:pass#localhost:6379/10']),
backend="".join(['redis://:pass#localhost:6379/11']),
include=['bin.proj.tasks'])
#logger.info('Celery session OK')
except Exception as e:
logger.error('Celery session KO')
logger.error(e)
if __name__ == '__main__':
app.start()
The user is 'ssc', this user is not part of root group, and I gave all rights needed to 'ssc' user on the project.
Any help would be much appreciated, I already send too much time on it, thanks

Answer 1

I had a similar issue daemonizing celery and the problem was related on the rights on pidfile and logfile.
The directories /var/run/celery and /var/log/celery should belong to the user running the service, here user scc. The problem is that when running as daemon the output is closed and thus hiding error messages.
If you want to debug the service and find the issue, just run the full command using the 'scc' account. It should tell you about any access right error or anything else.
/bin/sh -c '/usr/local/bin/celery multi start worker1 worker2 worker3 -A bin.proj.celery_app --pidfile=/var/run/celery/%n.pid --logfile=/var/log/celery/%n%I.log --loglevel=INFO --time-limit=300 --concurrency=3'