Currently I have a python script that uses a virtual environment called myenv. The command to run the script in terminal is the following:
cd env && source activate env && bin/env-dev python ~/Documents/test/ingest.py
When I tried setting this up in crontab, I kept getting an error with "command not found."
In cron tab I have tried a few different options:
1) Using a bash (the final piece is to write the errors to a text file). This has actually worked for other scripts in the file path that do not need to source activate myenv. So the problem exists when I add in source activate myenv.
* * * * * bash -c 'cd env && source activate env && bin/env-dev python ~/Documents/test/ingest.py' >> ~/test_cron.txt 2>&1
screenshot of terminal with the bash command
2) No bash
* * * * * cd env && source activate env && bin/env-dev python ~/Documents/test/ingest.py
3) I have also tried adding in a PATH=/Users/mattzerweck/miniconda3/envs/myenv at the top of the crontab but that has not successfully executed the bash.
Does any one have a solution for this? I haven't seemed to find a solution that has worked online and need this script to run automatically every morning at 5am. (currently every minute for testing purposes)
Found an answer that worked. Has to set a path and shell up top before the commands. Cron wasnt pulling my correct file path.
Related
I want my server to execute a node script every minute. The program executes perfectly if I execute the file manually (./main.js), so I'm pretty sure it's not the problem. But when I hand it over to cron to execute, nothing happens.
Here's the line from the cron file.
*/1 * * * * /home/bryce/scripts/wudu/main.js
And here's a sample log:
Oct 11 15:21:01 server CROND[2564]: (root) CMD (/home/bryce/scripts/wudu/main.js)
The executable: home/bryce/scripts/wudu/main.js
#!/usr/bin/env node
var program = require('commander');
var v = require('./cli/validation');
var a = require('./cli/actions');
program
.version('0.0.1')
.option('-u, --url', 'Register url')
.option('-s, --selector', 'Register selector')
.option('-p, --pagination', 'Register pagination')
.option('-i, --index', 'Pass an index, to destroy')
.parse(process.argv);
var args = process.argv.slice(2),
mode = v.mode(args[0]),
options = v.hasArgs(mode, program);
a.init(mode, options);
Any idea why I'm getting radio silence? Somewhere else I should be looking to debug?
UPDATE:
I believe the problem has to do with my relative filepaths, and main.js being executed from outside its own directory.
So now, I've placed exe.sh in the wudu directory. It looks like this:
#!/bin/bash
cd ${0%/*}
./main.js mail
exit
Now, I've set cron to execute this file every minute. I tried executing this file from other folders, and it works as expected. But again, cron isn't picking it up.
Wrapping the execution in a shell script, it's likely the execution of the script in cron doesn't have the same environment set as when you run from the command line.
Try prefacing the execution of the shell script in cron with setting NODE_PATH & PATH
(if you need these values, on a command line type: echo $NODE_PATH and echo $PATH)
So, your cron entry would look like:
*/1 * * * * NODE_PATH=/usr/local/lib/node_modules PATH=/opt/local/bin:ABC:XYZ /home/bryce/scripts/wudu/exe.sh
Just make sure to substitute the actual values for NODE_PATH & PATH with what you get from the echo commands that you first did.
I had the exact same problem, so I ended creating a passthrough script that would load my environment variables, then execute node like so:
##!/bin/bash
# Source bash profile to load env variables
# or any other that you want to set explicitly
. ~/.bash_profile
# tunnel that allows us to execute cron jobs
node $1 $2
To use just add it to your crontab
* * * * * <user> <passthrough_script> <arg1> <arg2>
You could try running the bash file directly from the cron?
i.e.
0 4 * * * source ~/.bashrc && cd /projects/my-project && node index.js
I am facing an issue of crontab does not run the python script in Raspbian GNU/Linux 9.4 (stretch) installed on a Raspberry Pi3. I have done lots of research on this topic e.g. by following the troubleshooting guide in https://askubuntu.com/questions/23009/why-crontab-scripts-are-not-working/24527#24527 , but none of them solved my issue.
The python script that I want to run is located in
/home/pi/Documents/Fork_BookManager.py
I have made sure that anyone can execute, read and change content to the above file.
I can run this file through terminal by writing
usr/bin/python3 /home/pi/Documents/Fork_BookManager.py
I know the file is running, because the Fork_BookManager.py is using selenium to open a web browser, and I see this web browser being opened. There is no error observed when running the Fork_BookManager.py file from terminal.
In terminal executed
crontab -e
Using nano as editor, I first verified that crontab acutally worked by
* * * * * env > /home/pi/Documents/env.output
After a minute there is a env.output file in Documents folder with below parameters.
LANGUAGE=da_DK.UTF-8
HOME=/home/pi
LOGNAME=pi
PATH=/usr/bin:/bin
LANG=da_DK.UTF-8
SHELL=/bin/sh
LC_ALL=da_DK.UTF-8
PWD=/home/pi
I noticed that the path from crontab is different from the path when env is called from terminal as I will have the below path if I call env from terminal
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games
So I know that crontab -e was working since I otherwise would not have got the env.output file created in Documents folder. I proceed by calling crontab -e from terminal and copied the path from env into the crontab like below
PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games
* * * * * /usr/bin/python3 /home/pi/Documents/Fork_BookManager.py
Saved the script and waited. Nothing happend.
To summerize I have:
Made sure that the Fork_BookManager.py is executable
Verified that crontab is working
Updated path from env into the crontab script
which are three largest cause of experiencing difficulties in making crontab to work according to this guide
https://askubuntu.com/questions/23009/why-crontab-scripts-are-not-working/24527#24527
Still it is not working. What have I missed? Is there a better practice to achieve what I want, namely to execute the python script once a minute without doing it directly from the python script itself such as a while loop with a time.sleep(60)
ADDITIONAL INFORMATION
I even tried to expand the path in crontab script by
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games:/usr/bin:/home/pi/Documents
in an attempt to specify in which folder the python3 and Fork_BookManager.py is located. Still no luck.
I'm trying to run some crawler with Linux crontab.
This should go to the Python environment with
pyenv shell jake-crawler
Here is my crontab -e
*/10 * * * * /home/ammt/apps/crawler/scripts/bat_start.sh
This will run every 10 minutes. This command line works fine when I type
(jake-crawler) [jake#KIBA_OM crawler]$ /home/jake/apps/crawler/scripts/bat_start.sh
[DEBUG|run.py:30] 2017-09-24 19:55:49,980 > BATCH_SN:1, COLL_SN:1, 1955 equal 0908 = False
Inside of bat_start.sh I have init.sh which changes the environment to Python.
Here is my init.sh
#!/usr/bin/env bash
export PATH="${HOME}/.pyenv/scripts:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
pyenv shell jake-crawler
This has no problem when I personally run it from command line. But when cron run it by itself, it cannot find the pyenv command.
I think that you can specify which user should run that script in cron configuration file.
So, if that script is working with your user, then define it in your cron configuration filr.
See this answer for example... https://stackoverflow.com/a/8475757/3827004.
There are two things that diferentiate when you launch an application from the terminal, and when you do from a crontab file:
the environment is not the same, at least if you don't execute your .profile script from your cron job.
You don't have access to a terminal. Cron jobs don't use a terminal, so you will not be able, for example to open /dev/tty. You will have to be very careful on how redirections are handled, as you have them all directed to your tty when running on an interactive session, but all of them will be redirected possibly to a pipe, when run from cron(8).
This makes your environment quite different and is normally a source of errors. Read crontab(1) man page for details.
I'm having issues getting my crontab to run I have the following line added to my crontab -e but it won't start. The command runs fine if I run it manually.
0 */3 * * * cd /home/sam/p/ && /usr/bin/python3.5 start.py
Not getting any error messages and can't see the process when I run top or grep for it.
Usually this happens because the cron environment is different from your own. Make sure your start.py script uses full paths to any referenced files or external scripts. Make sure that your start.py script does not rely on environment variables that you have in your shell but it may not. Try piping the cron output to a mail command so you can see what it is doing, like so:
0 */3 * * * cd /home/sam/p/ && /usr/bin/python3.5 start.py | mail -s "cron output" myself#example.com
An easier way to troubleshoot this is to write a wrapper shell script and send the output to a log file.
Create file python_start_cron.sh with contents
#!/bin/bash
cd /home/sam/p/ && /usr/bin/python3.5 start.py
Set the execute bit on this script script and make sure the script works manually
Modify the cronjob as shown below
0 */3 * * * python_start_cron.sh >/tmp/python_start_cron.log 2>&1
After cron executes, check the contents of the log file to ascertain the cause of the problem.
EDIT 2 27/06/2013: The problem was a silly mistake unrelated to venv and cron.
Running venv from cron with the same user that created the venv works great using the activate desscribed below.
EDIT 25/06/2013: Since nohup.out is unchanged after the cron runs, I suspect the problem is in the use of virtualenv.
The cron is set with the same user than the one running the script from the command line.
I have written a script to activate venv then run a python script. It runs perfectly from the command line, when I do
nohup /home/heyheyhey/run.sh &
However, for some black magic reasons, the python part does not run from cron:
0 4 * * * /home/heyheyhey/run.sh &
Content of run.sh:
#! /bin/bash
cd /home/heyheyhey
. /home/heyheyhey/.pythonbrew/venvs/Python-2.7.2/venvname/bin/activate 2> error.log
python /home/heyheyhey/top.py 2> error.log
bzip2 -c "Exporter.csv" > "extraction.csv.bz2"
The run.sh executes since the output compressed file is created.
However the python script does not work since the Exporter.csv is not updated and I do not see activity into the log file.
Thanks for your help!
The environment of a cron job is often different from the environment you see
when you're logged into an interactive shell. In particular, you might want to
check whether the python interpreter is on the $PATH for the cron job. If your
python program reads any environment variables, you should check those too, to
ensure they're set as expected under cron.