I'm trying to run some crawler with Linux crontab.
This should go to the Python environment with
pyenv shell jake-crawler
Here is my crontab -e
*/10 * * * * /home/ammt/apps/crawler/scripts/bat_start.sh
This will run every 10 minutes. This command line works fine when I type
(jake-crawler) [jake#KIBA_OM crawler]$ /home/jake/apps/crawler/scripts/bat_start.sh
[DEBUG|run.py:30] 2017-09-24 19:55:49,980 > BATCH_SN:1, COLL_SN:1, 1955 equal 0908 = False
Inside of bat_start.sh I have init.sh which changes the environment to Python.
Here is my init.sh
#!/usr/bin/env bash
export PATH="${HOME}/.pyenv/scripts:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
pyenv shell jake-crawler
This has no problem when I personally run it from command line. But when cron run it by itself, it cannot find the pyenv command.
I think that you can specify which user should run that script in cron configuration file.
So, if that script is working with your user, then define it in your cron configuration filr.
See this answer for example... https://stackoverflow.com/a/8475757/3827004.
There are two things that diferentiate when you launch an application from the terminal, and when you do from a crontab file:
the environment is not the same, at least if you don't execute your .profile script from your cron job.
You don't have access to a terminal. Cron jobs don't use a terminal, so you will not be able, for example to open /dev/tty. You will have to be very careful on how redirections are handled, as you have them all directed to your tty when running on an interactive session, but all of them will be redirected possibly to a pipe, when run from cron(8).
This makes your environment quite different and is normally a source of errors. Read crontab(1) man page for details.
Related
I want to run the below via Crontab job and not working but when put them in sh file and run the sh manually it works fine.
Sh file path: /opt/etc/rt4/test.sh and the content as below:
wsgetmail --config=account01.json
wsgetmail --config=account02.json
Running manually:
sh /opt/etc/rt4/test.sh it works fine.
Crontab:
*/1 * * * * /opt/etc/rt4/test.sh
Crontab runs this file but those commands are not working.
I have other Crontab jobs and they are working fine as intended.
The crontab and terminal are two different environments, the wsgetmail perl module command is recognizable for terminal but to make it recognizable for corntab we have to add the full path to the module (wsgetmail) in the shell script.
in this case test.sh should looks like this:
#!/bin/bash
/usr/local/bin/wsgetmail --config=account01.json
/usr/local/bin/wsgetmail --config=account02.json
Running manually: sh /opt/etc/rt4/test.sh it works fine.
Crontab: */1 * * * * /opt/etc/rt4/test.sh
Those are not the same thing, as 1st line shebang, and chmod a+x test.sh, will affect the behavior.
Either remove "sh" when running manually, or prepend it to the cron command.
Run $ id, and determine if that's different
from how the cron command runs, perhaps by
having cron log id output.
Running as yourself manually,
versus as root (uid=0) under cron,
can change the behavior of that command.
Numerous other things are different
under cron, such as lack of a pty.
Take a look at $ env | sort manually.
Then run it under crond, and note the huge difference.
Pay special attention to PATH.
It is likely to be much shorter under cron,
and that can lead to "command not found"
diagnostics.
But you chose not to share any diagnostic
error messages with us,
so coming up with a definitive diagnosis
of this amounts to a mind reading exercise.
I'm trying to start a bash script(test.sh) from a second bash script that runs as a cronjob(startTest.sh) on Ubuntu 14.04.
Cron is running and both scripts work perfectly if called from command line.
startTest.sh looks like this:
#!bin/bash
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/local/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/home/username/path/to/script
bash /home/username/path/to/script/test.sh
test.sh looks like this:
#!/bin/bash
touch it_works.txt
My crontab entry looks like this
* * * * * /usr/local/bin/startTest.sh
Best practice is generally not to use relative paths (unless you do an explicit cd) in scripts run as cron jobs.
crond is probably not running from whatever directory you expect it to. Depending on what user this cron job runs as, the script either does not have permission to create it_works.txt in crond's current working directory, or it is creating the file and you're looking in the wrong place.
I have a script with this as the shebang #!/usr/bin/env node.
When cron runs my script, I get this error /usr/bin/env: node: No such file or directory.
When i run the script as my user the script runs fine, just not as cron. I'm guessing it's because node is not on the PATH of the user that runs cron?
How can I get this shebang to work with cron?
$ which node gives me
/home/myuser/.nvm/v0.11.14/bin/node
Cron jobs run in a restricted environment. In an interactive shell, your $PATH is probably set in your $HOME/.bash_profile or $HOME/.bashrc. Cron jobs are executed in an environment that hasn't sourced those files, so your user-specific $PATH settings will not be available.
You can see what that environment looks like by temporarily creating a crontab entry like:
* * * * * printenv > crontab-environment
You can explicitly set $PATH in your crontab, either in the command itself:
* * * * * PATH=$PATH:/home/myuser/.nvm/v0.11.14/bin some_command
or in a separate line in your crontab:
PATH = /usr/bin:/bin:/home/myuser/.nvm/v0.11.14/bin
You can't (directly) use the usual PATH=$PATH:/new/dir syntax to append a directory to your $PATH in an environment setting line, because variable references are not replaced in such a line. They are processed in crontab command lines.
man 5 crontab for details.
Another option is to use an explicit full path in the script itself, changing
#!/usr/bin/env node
to
#!/home/myuser/.nvm/v0.11.14/bin/node
You'll need to customize this for each system where node is installed in a different place. Or you can arrange for node (or nodejs?) to be installed in a consistent place.
I am using Linux Centos to schedule a job.
I have created a shell script file called Im_daily_loads.sh to run the job at 12:42PM everyday.
with the following comands:
#!/bin/sh
42 12 * * * cd $pdi; ./kitchen.sh -file="/opt/kff/software/pdi/5.0.1.A/data- integration/projects/IML/code/stg/IML_Load_Frm_SRC_To_PSA.kjb" -level=Basic > -logfile="/opt/kff/software/pdi/5.0.1.A/data-integration/projects/IML/log/iml_daily_loads.err.log"
Then loaded the file into crontab by using the issuing the following command crontab Im_daily_loads.sh, but my job is not running.
What would be the problem?
Why not just use
crontab -e
as the user you plan to execute the job as, enter the job, save and exit the editor?
Also, it looks like you need to define $pdi in your script. How is crontab supposed to know where your script is located?
first , run a very simple job to be shure crontab works at all.
for example
set > /tmp/crontab_works.log 2>&1
it will write down all variables. so you will see not all variables available in crontab
EDIT 2 27/06/2013: The problem was a silly mistake unrelated to venv and cron.
Running venv from cron with the same user that created the venv works great using the activate desscribed below.
EDIT 25/06/2013: Since nohup.out is unchanged after the cron runs, I suspect the problem is in the use of virtualenv.
The cron is set with the same user than the one running the script from the command line.
I have written a script to activate venv then run a python script. It runs perfectly from the command line, when I do
nohup /home/heyheyhey/run.sh &
However, for some black magic reasons, the python part does not run from cron:
0 4 * * * /home/heyheyhey/run.sh &
Content of run.sh:
#! /bin/bash
cd /home/heyheyhey
. /home/heyheyhey/.pythonbrew/venvs/Python-2.7.2/venvname/bin/activate 2> error.log
python /home/heyheyhey/top.py 2> error.log
bzip2 -c "Exporter.csv" > "extraction.csv.bz2"
The run.sh executes since the output compressed file is created.
However the python script does not work since the Exporter.csv is not updated and I do not see activity into the log file.
Thanks for your help!
The environment of a cron job is often different from the environment you see
when you're logged into an interactive shell. In particular, you might want to
check whether the python interpreter is on the $PATH for the cron job. If your
python program reads any environment variables, you should check those too, to
ensure they're set as expected under cron.