I have a render application written in Python that uses the blender library (bpy) and renders itself using Eevee. It implements the execution of tasks using Celery. The application runs on Linux ubuntu 20.04, the nvidia-grid 510.47.03-grid-aws drivers were installed on the server and a virtual display was launched for the rendering process.
While running Celery manually from the application's virtual environment with the command: celery -A tasks worker -Q feed -l info --concurrency=3, rendering takes 15-17 seconds.
Next, I set up services for systemd to start Celery automatically. But in this way, the rendering process of exactly the same scene takes 34-36 seconds.
I tried to set the maximum priority for the processor and RAM and still do not understand why the rendering, started automatically, takes twice as long as the same one, started manually.
The systemd configuration files look like this:
/etc/conf.d/celery
CELERYD_NODES="render_server"
CELERY_BIN="/home/ubuntu/workflow/renddly-backend-instance/venv/bin/celery"
CELERY_APP="tasks"
CELERYD_MULTI="multi"
CELERYD_OPTS="-Q feed --time-limit=300 --concurrency=3"
# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
/etc/systemd/system/celery.service
[Unit]
Description=Celery service for rendering
After=network.target
After=display.service
[Service]
Type=forking
EnvironmentFile=/etc/conf.d/celery
Environment=DISPLAY=:1
CPUSchedulingPriority=99
RestrictRealtime=false
OOMScoreAdjust=-1000
LimitMEMLOCK=infinity
WorkingDirectory=/home/ubuntu/workflow/renddly-backend-instance
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
[Install]
WantedBy=multi-user.target
Related
I am using the nohup command with Python and Flask for background process. After I close the terminal it is working fine but after 1 or 2 days the process stops. Can someone tell me how to keep the background process running? I am using below command:
screen
space
nohup python -m flask run --cert local.crt --key local.key --host=0.0.0.0 --port=443 &
ctrl+a+d
Let's assume all your Flask code resides in the folder /home/abc_user/flask_app.
Steps
Create a file flask-server.service in /etc/systemd/system.
[Unit]
Description=Flask server
After=network.target
[Service]
User=abc_user
Group=abc_user
WorkingDirectory=/home/abc_user/flask_app
ExecStart=python -m flask run --cert local.crt --key local.key --host=0.0.0.0 --port=443
Restart=always
[Install]
WantedBy=multi-user.target
Run sudo systemctl daemon-reload.
Start the service using systemctl start flask-server.service.
Check that it has started by systemctl status flask-server.service. Status should say "running".
If you want your flask server to auto-start after reboots, run systemctl enable flask-server.service
Some common operations
Check current status - systemctl status flask-server.service
Start the service - systemctl start flask-server.service
Stop the service - systemctl stop flask-server.service
Check logs - journalctl -u flask-server.service
Stream logs - journalctl -f -u flask-server.service
Check logs in past 1 hour - journalctl -u flask-server.service --since "1 hour ago"
Try nohup python -m flask run --cert local.crt --key local.key --host=0.0.0.0 --port=443 >/dev/null 2>&1&
Use nohup , you should redirct you print to /dev/null or log, Otherwise it will be create a file nohup.out occupy disk space.
Most times we use gunicorn and supervisor to manager flask application.
Did you maybe shut down the computer the flask server is running on ?
If so, the problem will be solved by either not shutting down your computer or starting the flask server again after shutting down !
nohup is a POSIX command to ignore the HUP (hangup) signal. The HUP signal is, by convention, the way a terminal warns dependent processes of logout.
Output that would normally go to the terminal goes to a file called nohup.out if it has not already been redirected.
See nohup.out for searching errors in ./ or executed directory. It is no nohup error. Look nohup.out and google error and will refresh question.
I've been trying to set up celery as a daemon for three days now and I am still struggling with it. Every time I start the service and check the status, it shows this,
The documentation had Type=forking, and when using it, the service fails to start. Status shows,
● celery.service - Celery Service
Loaded: loaded (/etc/systemd/system/celery.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2020-07-23 05:36:18 UTC; 7s ago
Process: 4256 ExecStart=/bin/sh -c ${CELERY_BIN} multi start ${CELERYD_NODES} -A ${CELERY_APP} --pidfile=${CELER>
Main PID: 4272 (code=exited, status=1/FAILURE)
... systemd[1]: Starting Celery Service...
... sh[4257]: celery multi v4.4.6 (cliffs)
... sh[4257]: > Starting nodes...
... sh[4257]: > w1#dexter-server: OK
... systemd[1]: Started Celery Service.
... systemd[1]: celery.service: Main process exited, code=exited, status=1/FAILURE
... systemd[1]: celery.service: Failed with result 'exit-code'.
Nothing in the logs. And when I remove Type, I get this,
● celery.service - Celery Service
Loaded: loaded (/etc/systemd/system/celery.service; disabled; vendor preset: enabled)
Active: inactive (dead)
... systemd[1]: Started Celery Service.
... sh[2683]: celery multi v4.4.6 (cliffs)
... sh[2683]: > Starting nodes...
... sh[2683]: > w1#dexter-server: OK
... sh[2690]: celery multi v4.4.6 (cliffs)
... sh[2690]: > w1#dexter-server: DOWN
... systemd[1]: celery.service: Succeeded.
Nothing in the log as well. Here are my current files.
/etc/systemd/system/celery.service
[Unit]
Description=Celery Service
After=network.target
[Service]
User=dexter
Group=dexter
RuntimeDirectory=celery
RuntimeDirectoryMode=0755
EnvironmentFile=/etc/conf.d/celery
WorkingDirectory=/var/www/example.com/myapp
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
[Install]
WantedBy=multi-user.target
/etc/tmpfiles.d/celery.conf
d /var/run/celery 0755 dexter dexter -
d /var/log/celery 0755 dexter dexter -
/etc/conf.d/celery
# Name of nodes to start
# here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"
# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/example.com/venv/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"
# App instance to use
# comment out this line if you don't use an app
CELERY_APP="myapp.celery:app"
# or fully qualified:
#CELERY_APP="proj.tasks:app"
# How to call manage.py
CELERYD_MULTI="multi"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
# and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
# you may wish to add these options for Celery Beat
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"
CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"
CELERY_CREATE_DIRS=1
I have no idea how to configure beyond this or where to look for errors at this point. Could someone help me identify the issue and solve this? I have a very basic understanding of concepts like forking, so please go easy on me.
I solved this by replacing the block in celery.service with the following.
ExecStart='${CELERY_BIN} ${CELERy_NODES} \
-A ${CELERY_APP} worker --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop='${CELERY_BIN} stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'
ExecReload='${CELERY_BIN} restart ${CELERYD_NODES} \
-A ${CELERY_APP} worker --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
I got help from #celery IRC.
So I have a script setup that I want to run as a service. The script runs 8 curl commands continuously forever:
#!/bin/bash
threads=8
check_interval=15
batch_url=https://myurl.com
for ((i = 0 ; i < threads ; i++)); do
sleep $i
while true; do
sleep 0.5
has_items=$(curl --silent -m 15.0 --connect-timeout 1.5 $batch_url/$i)
if [ "$has_items" == "false" ]; then
sleep ${check_interval}
continue
fi
done &
done
I have a service file setup in systemd that looks like:
[Unit]
Description=Run the batch forever
After=network.target
[Service]
Type=simple
Restart=always
RestartSec=1
ExecStart=/bin/bash /usr/bin/my_script_location
[Install]
WantedBy=multi-user.target
If I simply run the script it works fine. It loops forever pinging URLs /0 through /7 forever.
However, if I set it up as a service, the service will restart after some time.
I believe the issue is related to having the while loop run in the background, but not sure how to make the service work with this. It's possible that the service approach is the wrong method. I simply need this script to always remain running.
I have several celery tasks that execute via beat. In development, I used a single command to set this up, like:
celery worker -A my_tasks -n XXXXX#%h -Q for_mytasks -c 1 -E -l INFO -B -s ./config/celerybeat-schedule --pidfile ./config/celerybeat.pid
On moving to production, I inserted this into a script that activated my venv, set the PYTHONPATH, removed old beat files, cd to correct directory and then run celery. This works absolutely fine. However, in production I want to separate the worker from the beat scheduler, like:
celery worker -A my_tasks -n XXXXX#%h -Q for_mytasks -c 1 -E -l INFO -f ./logs/celeryworker.log
celery beat -A my_tasks -s ./config/celerybeat-schedule --pidfile ./config/celerybeat.pid -l INFO -f ./logs/celerybeat.log
Now this all works fine when put into the relevant bash scripts. However, I need these to be run on server start-up. I encountered several problems:
1) in crontab -e #reboot my_script will not work. I have to insert a delay to allow rabbitmq to fully start, i.e. #reboot sleep 60 && my_script. Now this seems a little 'messy' to me but I can live with it.
2) celery worker takes several seconds to finish before celery beat can be run properly. I tried all manner of cron directives to accomplish beat being run after worker has executed successfully but couldn't get the beat to run. My current solution in crontab is something like this:
#reboot sleep 60 && my_script_worker
#reboot sleep 120 && my_script_beat
So basically, ubuntu boots, waits 60 seconds and runs celery worker then waits another 60 seconds before running celery beat. This works fine but it seems even more 'messy' to me. In an ideal world I would like to flag when rabbitmq is ready to run worker, then flag when worker has executed successfully so that I can run beat.
My question is : has anybody encountered this problem and if so do they have a more elegant way of kicking off celery worker & beat on server reboot?
EDIT: 24/09/2019
Thanks to DejanLekic & Greenev
I have spent some hours converting from cron to systemd. Yes, I agree totally that this is a far more robust solution. My celery worker & beat are now started as services by systemd on reboot.
There is one tip I have for people trying this that is not mentioned in the celery documentation. The template beat command will create a 'celery beat database' file called celerybeat-schedule in your working directory. If you restart your beat service, this file will cause spurious celery tasks to be spawned that don't seem to fit with your actual celery schedule. The solution is to delete this file each time the beat service starts. I also delete the pid file, if it's there. I did this by adding 2 ExecStartPre and a -s option to the beat service :
ExecStartPre=/bin/sh -c 'rm -f ${CELERYBEAT_DB_FILE}'
ExecStartPre=/bin/sh -c 'rm -f ${CELERYBEAT_PID_FILE}'
ExecStart=/bin/sh -c '${CELERY_BIN} beat \
-A ${CELERY_APP} --pidfile=${CELERYBEAT_PID_FILE} \
-s ${CELERYBEAT_DB_FILE} \
--logfile=${CELERYBEAT_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL}'
Thanks guys.
To daemonize celery worker we are using systemd, so the worker and the beat could be getting to run as separate services and configured to start on the server reboot via just making these services enabled
What you really want is to run Celery beat process as a systemd or SysV service. It is described in depth in the Daemonization section of the Celery documentation. In fact, same goes for the worker process too.
Why? - Unlike your solution, which involves crontab with #reboot line, systemd for an example can check the health of the service and restart it if needed. All Linux services on your Linux boxes are started this way because it has been made for this particular purpose.
Many system daemon can be started using start/stop command. I was just curious how start/stop works on Linux system. Say I wrote a daemon executable, how should I configure it so that it can be controlled by start/stop in Linux.
I make a daemon in linux (ArchLinux) few years ago, and it works every day perfectly.
There are 2 ways to do this. Short way and long way:
Short Way:
Create a file in /etc/systemd/system/ called for example mydaemon.service :
/etc/systemd/system/mydaemon.service
[Unit]
Description=This is my first daemon! - Fernando Pucci
After=network.target
[Service]
User=root
WorkingDirectory=/root
Type=oneshotmc
RemainAfterExit=yes
ExecStart=/bin/echo -e "Daemon started"
ExecStop=/bin/echo -e "Daemon Stopped"
[Install]
WantedBy=multi-user.target
This service does nothing but show Daemon Started or Stopped. You can change echoes by the sentences you need.
If you need to run some script, try the Long way:
Long way
Create a file in some directory, like root folder or /usr/lib/systemd/scripts called for example
/root/mydaemon.sh
start() {
<your start sentences here
and here>
}
stop() {
<your stop sentences here
and here>
}
case $1 in
start|stop) "$1" ;;
esac
You must to make it runnable (chmod x)
(And you can execute it with start or stop parameter to test it.)
And as second step, create another file in
/usr/lib/systemd/system/mydaemon.service
[Unit]
Description=Second daemon of Fernando Pucci
After=network.target
[Service]
User=root
WorkingDirectory=/root
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -c '/root/mydaemon.sh start'
ExecStart=/bin/echo -e "MyDaemon Started"
ExecStop=/bin/bash -c '/root/mydaemon.sh stop'
ExecStop=/bin/echo -e "MyDaemon Stopped"
[Install]
WantedBy=multi-user.target
Starting and Stopping
systemctl start mydaemon
systemctl stop mydaemon
systemctl status mydaemon
systemctl enable mydaemon
systemctl disable mydaemon
You (and someone) can send me a private msg for help about that.