I'm trying to aggregate News RSS Feeds with celery but im getting this error from SQLAlchemy.
Im runing 3 celery workers like thi:
celery -A main.celery worker --loglevel=INFO -P gevent --concurrency=50 -n celeryworker1#%%h
celery -A main.celery worker --loglevel=INFO -P gevent --concurrency=50 -n celeryworker2#%%h
celery -A main.celery worker --loglevel=INFO -P gevent --concurrency=50 -n celeryworker3#%%h
but i get a lot of this message from celery worker:
[INFO/MainProcess] Redirecting
What does this mean, is somethng i should worry about ?
Related
the server where I have hosted my app usually restarts due to maintenance and when it does a function that I kept open in the background stops and I have to manually turn it on.
Here are the commands that I do in ssh
ssh -p19199 -i <my ssh key file name> <my username>#server.net
source /home/myapp/virtualenv/app/3.8/bin/activate
cd /home/myapp/app
celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log --detach
I need to start this celery app whenever there is no 'celery' function in the command ps axuww. If it is running already then it will show:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
myapp 8792 0.1 0.2 1435172 82252 ? Sl Jun27 1:27 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp 8898 0.0 0.2 1115340 92420 ? S Jun27 0:32 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp 8899 0.0 0.2 1098900 76028 ? S Jun27 0:00 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp 8900 0.0 0.2 1098904 76028 ? S Jun27 0:00 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp 8901 0.0 0.2 1098908 76040 ? S Jun27 0:00 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp 28216 0.0 0.0 10060 2928 pts/1 Ss 15:57 0:00 -bash
myapp 28345 0.0 0.0 49964 3444 pts/1 R+ 15:57 0:00 ps axuww
I need the cron job to check every 15 minutes.
You have everything figured out already, just put it inside a IF block in a shell (Bash) script, save it wherever you feel like (eg, $HOME). After the script, we'll set Cron.
I'm gonna call this script "start_celery.sh":
#!/usr/bin/env bash -ue
#
# Start celery only if it is not running yet
#
if [[ `ps auxww | grep "[c]elery"` ]]; then
source /home/myapp/virtualenv/app/3.8/bin/activate
cd /home/myapp/app
celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log --detach
fi
Remember to make it executable!
$ chmod +x start_celery.sh
A few notes:
the output from ps is being filtered by grep "[c]elery" because we don't one the grep line that would come out if we simply did grep "celery" (reference: https://www.baeldung.com/linux/grep-exclude-ps-results).
Using grep "celery" | grep -v "grep" would not work in our case because we want to use the lack of celery commands (exit code "1") to trigger the IF block.
setting "-ue" in the script fails the whole thing/script in case something goes wrong (this is a precaution I always use, and I live it here as a shared knowledge/hint. Even if we are not using variables (-u) I like to use it). Feel free to remove it. (reference https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html)
Now we have the script -- ~/start_celery.sh in our home directory, we just have to include it in our (user's) crontab jobs. To do that, we can use some help from some websites of our (awesome) geek community:
The first one is https://crontab.guru/ . If nothing else, it puts the cronjobs syntax in very simple, visual, straight words. They also offer a payed service in case you want to monitor your jobs.
In the same spirit of online monitoring (maybe of your interest, I live it here for the records and because it's free), you may find interesting https://cron-job.org/ .
Whatever the resources we use to figure out the crontab line we need, "every 15 minutes" is: */15 * * * * .
To set that up, we open our (user's) cronjob table (crontab) and set our job accordingly:
Open crontab jobs:
$ crontab -e
Set the (celery) job (inside crontab):
*/15 * * * * /home/myapp/start_celery.sh
This should work, but I didn't test the whole thing. Let me know how that goes.
I think what you need is a corn job running your script at startup as long as you don't experience any function crashes.
Start by opening a terminal window and run the command below. Don't forget to <add your ssh key> and <username>. This will generate the script.sh file for you and save it in $HOME
echo -e "ssh -p19199 -i <my ssh key file name> <my username>#server.net\n\nsource /home/myapp/virtualenv/app/3.8/bin/activate\n\ncd /home/myapp/app\n\ncelery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log --detach" >> $HOME/startup.sh
Make it executable using the following command:
sudo chmod +x $HOME/startup.sh
Now create the cron job to run the script when you restart your server. In the terminal create a new cron job.
crontab -e
This will open a new editor screen for you to write the cron job. Press the letter i on your keyboard to start the insert mode and type the following
#reboot sh $HOME/startup.sh
Exit the editor and save your changes by pressing esc key then type :wq and hit Enter
Restart your server and the cron job should run the script for you.
If you want to manage the tasks that should run when a system reboots, shutdowns are alike, you should not make use of cron. These tasks are managed by the Power Management Utilities (pm-utils). Have a look at man pm-action and this question for a somewhat detailed explanation.
Obviously, as a user it is not always possible to configure these hooks for pm-util and alternative options are considered. Having a cron-job running every x minutes is an obvious solution that would allow the user to automatically start a job.
The command you would run in cron would look something like:
pgrep -f celery 2>/dev/null || /path/to/celery_startup
Where /path/to/celery_startup is a simple bash script that contains the commands to be executed (the ones mentioned in the OP).
I have a render application written in Python that uses the blender library (bpy) and renders itself using Eevee. It implements the execution of tasks using Celery. The application runs on Linux ubuntu 20.04, the nvidia-grid 510.47.03-grid-aws drivers were installed on the server and a virtual display was launched for the rendering process.
While running Celery manually from the application's virtual environment with the command: celery -A tasks worker -Q feed -l info --concurrency=3, rendering takes 15-17 seconds.
Next, I set up services for systemd to start Celery automatically. But in this way, the rendering process of exactly the same scene takes 34-36 seconds.
I tried to set the maximum priority for the processor and RAM and still do not understand why the rendering, started automatically, takes twice as long as the same one, started manually.
The systemd configuration files look like this:
/etc/conf.d/celery
CELERYD_NODES="render_server"
CELERY_BIN="/home/ubuntu/workflow/renddly-backend-instance/venv/bin/celery"
CELERY_APP="tasks"
CELERYD_MULTI="multi"
CELERYD_OPTS="-Q feed --time-limit=300 --concurrency=3"
# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
/etc/systemd/system/celery.service
[Unit]
Description=Celery service for rendering
After=network.target
After=display.service
[Service]
Type=forking
EnvironmentFile=/etc/conf.d/celery
Environment=DISPLAY=:1
CPUSchedulingPriority=99
RestrictRealtime=false
OOMScoreAdjust=-1000
LimitMEMLOCK=infinity
WorkingDirectory=/home/ubuntu/workflow/renddly-backend-instance
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
[Install]
WantedBy=multi-user.target
I want to pass a environement variable to my supervisord subprocesses, but the variable doesn't appear in the available names for the subprocesses. The major issue is because my azure app service builds dynamically, I can't explicitly define the variable but must use the env var $APP_PATH. I want to figure out how I can pass this to the Supervisor subprocess environments. Here are my config files;
supervisord.conf
[unix_http_server]
file=/var/run/supervisor.sock ; path to your socket file
chmod=0700
[supervisord]
command=env
logfile=/home/LogFiles/supervisord/supervisord.log ; supervisord log file
logfile_maxbytes=50MB ; maximum size of logfile before rotation
logfile_backups=10 ; number of backed up logfiles
loglevel=info ; info, debug, warn, trace
pidfile=/var/run/supervisord.pid ; pidfile location
nodaemon=false ; run supervisord as a daemon
minfds=1024 ; number of startup file descriptors
minprocs=200 ; number of process descriptors
user=root ; default user
childlogdir=/home/LogFiles/supervisord/ ; where child log files will live
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use unix:// schem for a unix sockets.
[include]
files=etc/supervisor/conf.d/celeryd.conf
celerybeat.conf
[program:celerybeat]
environment=APP_PATH=$APP_PATH
; Set full path to celery program if using virtualenv
command=%(ENV_APP_PATH)s/antenv/bin/celery -A kinduwa beat --loglevel=INFO
;command=celery -A kinduwa beat -loglevel=INFO
; remove the -A myapp argument if you aren't using an app instance
user=nobody
numprocs=1
stdout_logfile=/home/LogFiles/beat.out.log
; stdout_logfile= /var/log/celery/beat.log
stderr_logfile=/home/LogFiles/beat.err.log
; stderr_logfile= /var/log/celery/beat.log
autostart=true
autorestart=true
startsecs=10
; Causes supervisor to send the termination signal (SIGTERM) to the whole process group.
stopasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999
celeryd.conf
[program:celery]
environment=APP_PATH=$APP_PATH
user=nobody
numprocs=1
stdout_logfile=/home/LogFiles/celery/worker.out.log
; stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/home/LogFiles/celery/worker.err.log
; stderr_logfile=/var/log/celery/worker.log
autostart=true
autorestart=true
startsecs=10
; Set full path to celery program if using virtualenv
;command=celery -A proj worker --loglevel=INFO
;command=celery -A kinduwa worker --loglevel=INFO
; Alternatively,
;command=celery --app=your_app.celery:app worker --loglevel=INFO -n worker.%%h
command=%(ENV_APP_PATH)s/antenv/bin/celery -A kinduwa worker --loglevel=INFO
;command=celery -A kinduwa worker --loglevel=INFO
; Or run a script
;command=celery.sh
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; Causes supervisor to send the termination signal (SIGTERM) to the whole process group.
stopasgroup=true
; Set Celery priority higher than default (999)
; so, if rabbitmq is supervised, it will start first.
priority=1000
here is my start_up script for good measure;
apt-get update
apt-get install -y supervisor
mkdir /var/run/celery/
mkdir /var/lib/celery/
mkdir /var/log/celery/
# adding and starting custom supervisor config
cp kdw_supervisord/kdw_supervisord.conf /etc/supervisor/
supervisorctl -c /etc/supervisor/kdw_supervisord.conf
service supervisor stop
service supervisor start
# adding in celery beat and worker configs
cp kdw_supervisord/kdw_celerybeat.conf /etc/supervisor/conf.d/
cp kdw_supervisord/kdw_celeryd.conf /etc/supervisor/conf.d/
supervisorctl reread
supervisorctl update
supervisorctl start all
gunicorn --bind=0.0.0.0 --timeout 600 kinduwa.wsgi
The error I get when I attempt to deploy is;
2021-05-17T15:43:18.125749054Z ERROR: CANT_REREAD: Format string '%(ENV_APP_PATH)s/antenv/bin/celery -A kinduwa beat --loglevel=INFO' for 'program:celerybeat.command' contains names ('ENV_APP_PATH') which cannot be expanded. Available names: ENV_LANG, ENV_LANGUAGE, ENV_LC_ADDRESS, ENV_LC_ALL, ENV_LC_COLLATE, ENV_LC_CTYPE, ENV_LC_IDENTIFICATION, ENV_LC_MEASUREMENT, ENV_LC_MESSAGES, ENV_LC_MONETARY, ENV_LC_NAME, ENV_LC_NUMERIC, ENV_LC_PAPER, ENV_LC_TELEPHONE, ENV_LC_TIME, ENV_PATH, ENV_PWD, ENV_TERM, group_name, here, host_node_name, process_num, program_name in section 'program:celerybeat' (file: '/etc/supervisor/conf.d/kdw_celerybeat.conf')
tl:dr; I have a dynamically set $APP_PATH env_var want to use in supervisord subprocesses, how do I do that?
I've read How do I kill background processes / jobs when my shell script exits?, but I can't get it to work.
IDK if it's Docker shenanigans or something else.
#!/bin/bash -e
base="$(dirname "$0")"
trap 'kill $(jobs -p)' SIGINT SIGTERM EXIT
docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:12 &
while ! nc -z localhost 5432; do
sleep 0.1
done
# uh-oh, error
false
When I run this, I am left with a running Docker container.
Why? How can stop the process when my script exits?
Docker is a client/server application, consisting of a thin client, docker, and server, dockerd. When you run a container, the client makes a few API calls to the server, one to create the container, another to start it, and since you didn't run it detached, it runs an attach API. When you kill the docker process, it detaches from the container, no longer showing you the logs, and kills that client portion. But the dockerd server is still running the container until process inside the container, running as pid 1 inside the container namespace, exits. You never killed that process since it's spawned from the dockerd daemon, not directly from the docker client.
To fix this, my suggestion is to run a docker stop, with the container name or id, as part of your trap handler. I wouldn't even bother running docker in the background, and instead pass -d to run detached.
Follow up, testing the script locally, it looks like killing the docker client does send a docker stop signal when you run the client attached like that. However, there's a race condition that can cause that stop to happen before the database is running. The command:
nc -z localhost 5432
is always going to succeed even before postgresql starts listening on the port because docker creates a port forward. E.g.:
$ nc -z localhost 5432 && echo it works
$ docker run -itd --rm -p 5432:5432 busybox tail -f /dev/null
c72427053124608fe18c31e5d6f3307d74a5cdce018503e9fff85dbc039b4fff
$ nc -z localhost 5432 && echo it works
it works
$ docker stop c72
c72
$ nc -z localhost 5432 && echo it works
However, if I run a sleep in the script, that forces it to wait long enough for the container to finish starting up, and the attach to finish, the container is stopped.
A better version of the script looks like the following, that waits for the database to completely start by checking the logs, and changing the trap to run a docker stop command:
#!/bin/bash -e
base="$(dirname "$0")"
trap 'kill $(jobs -p)' SIGINT SIGTERM EXIT
cid=$(docker run --rm -d -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:12)
# leaving the kill assuming you have other background processes
trap 'docker stop $cid; kill $(jobs -p)' SIGINT SIGTERM EXIT
# waiting for the db to actually start, assuming later steps need the db to be up
while ! docker logs "$cid" 2>&1 | grep -q "database system is ready to accept connections" ; do
sleep 0.1
done
# uh-oh, error
false
It was Docker shenanigans.
I needed to use the --init option to run tini shim because
A process running as PID 1 inside a container is treated specially by Linux: it ignores any signal with the default action. As a result, the process will not terminate on SIGINT or SIGTERM unless it is coded to do so.
docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:12 &
I have several celery tasks that execute via beat. In development, I used a single command to set this up, like:
celery worker -A my_tasks -n XXXXX#%h -Q for_mytasks -c 1 -E -l INFO -B -s ./config/celerybeat-schedule --pidfile ./config/celerybeat.pid
On moving to production, I inserted this into a script that activated my venv, set the PYTHONPATH, removed old beat files, cd to correct directory and then run celery. This works absolutely fine. However, in production I want to separate the worker from the beat scheduler, like:
celery worker -A my_tasks -n XXXXX#%h -Q for_mytasks -c 1 -E -l INFO -f ./logs/celeryworker.log
celery beat -A my_tasks -s ./config/celerybeat-schedule --pidfile ./config/celerybeat.pid -l INFO -f ./logs/celerybeat.log
Now this all works fine when put into the relevant bash scripts. However, I need these to be run on server start-up. I encountered several problems:
1) in crontab -e #reboot my_script will not work. I have to insert a delay to allow rabbitmq to fully start, i.e. #reboot sleep 60 && my_script. Now this seems a little 'messy' to me but I can live with it.
2) celery worker takes several seconds to finish before celery beat can be run properly. I tried all manner of cron directives to accomplish beat being run after worker has executed successfully but couldn't get the beat to run. My current solution in crontab is something like this:
#reboot sleep 60 && my_script_worker
#reboot sleep 120 && my_script_beat
So basically, ubuntu boots, waits 60 seconds and runs celery worker then waits another 60 seconds before running celery beat. This works fine but it seems even more 'messy' to me. In an ideal world I would like to flag when rabbitmq is ready to run worker, then flag when worker has executed successfully so that I can run beat.
My question is : has anybody encountered this problem and if so do they have a more elegant way of kicking off celery worker & beat on server reboot?
EDIT: 24/09/2019
Thanks to DejanLekic & Greenev
I have spent some hours converting from cron to systemd. Yes, I agree totally that this is a far more robust solution. My celery worker & beat are now started as services by systemd on reboot.
There is one tip I have for people trying this that is not mentioned in the celery documentation. The template beat command will create a 'celery beat database' file called celerybeat-schedule in your working directory. If you restart your beat service, this file will cause spurious celery tasks to be spawned that don't seem to fit with your actual celery schedule. The solution is to delete this file each time the beat service starts. I also delete the pid file, if it's there. I did this by adding 2 ExecStartPre and a -s option to the beat service :
ExecStartPre=/bin/sh -c 'rm -f ${CELERYBEAT_DB_FILE}'
ExecStartPre=/bin/sh -c 'rm -f ${CELERYBEAT_PID_FILE}'
ExecStart=/bin/sh -c '${CELERY_BIN} beat \
-A ${CELERY_APP} --pidfile=${CELERYBEAT_PID_FILE} \
-s ${CELERYBEAT_DB_FILE} \
--logfile=${CELERYBEAT_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL}'
Thanks guys.
To daemonize celery worker we are using systemd, so the worker and the beat could be getting to run as separate services and configured to start on the server reboot via just making these services enabled
What you really want is to run Celery beat process as a systemd or SysV service. It is described in depth in the Daemonization section of the Celery documentation. In fact, same goes for the worker process too.
Why? - Unlike your solution, which involves crontab with #reboot line, systemd for an example can check the health of the service and restart it if needed. All Linux services on your Linux boxes are started this way because it has been made for this particular purpose.