How to run Scrapy-spider from crontab? - linux

I'm trying to set a task to run a script from such a sh-file (run_a.sh)
#!/bin/bash
cd /home/userdir
source venvProject/bin/activate
cd /home/userdir/scrapy_project
PATH=$PATH:/usr/local/bin
export PATH
scrapy crawl my_spider
My crontab string looks like this:
*/5 * * * * sh /home/userdir/run_a.sh
but it don't works, whats wrong?

Related

Run yarn script into crontab

I made a script in TypeScript that download data from some api and store inside a mongo DB.
If i run yarn start from the app folder it works well.
I would like to put this command in a cron job that will be executed every 5 minutes.
I try it with some sintax in crontab but ti doesn't work.
I try to put the call in a run.sh script but it doesn't work too.
*/5 * * * * cd /opt/app-folder/src/ && /home/username/.nvm/versions/node/v16.15.1/bin/ts-node main.ts
*/5 * * * * cd /opt/app-folder && /usr/bin/yarn start > /home/username/app-name-out.txt
*/5 * * * * /home/username/run.sh > /home/username/app-name-out.txt
*/5 * * * * /home/username/.nvm/versions/node/v16.15.1/bin/ts-node /opt/app-folder/src/main.ts > /home/username/app-name-out.txt
*/5 * * * * cd /opt/app-folder/src/ && /home/username/.nvm/versions/node/v16.15.1/bin/ts-node main.ts > /home/username/app-name-out.txt
Can someone help me to execute the main.ts every 5 minutes?
Thanks
I get rid of this problem.
There was 2 problems, the first related to the output redirection.
I fixed by redirect stdout in a file and stderr in another one.
The second was related the the $PATH of crontab: it was /usr/bin:/bin.
To fix it I log into my user where script works and I print my $PATH with echo $PATH.
I copied the value and I set it before the crontab line in crontab file.
This is what it looks like:
# Set the same path of user username to have the correct path in script
PATH=/home/username/.nvm/versions/node/v16.15.1/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin
# Execute oracle every 5 minutes
*/5 * * * * /bin/sh /home/username/run.sh >> /home/username/app-name-info.txt 2>> /home/username/app-name-error.txt
Now it works.

Cronjob won't trigger .sh script

I have made this Cronjob that should run some tests. My script works, but cronjob won't trigger it.
Cronjob looks like this:
*/1 * * * * /bin/sh cd ~/Desktop/abc.sh
I want it to run every minute, just for testing purposes.
And my script is:
while read LINE; do curl -o /dev/null --silent --head --write-out "%{http_code} $LINE\n" "$LINE"; done < todo | tee test_results.txt
I can't even find the solution on google or youtube.
If you added #!/bin/bash to your script, then your cronjob should look like:
* * * * * ~/Desktop/abc.sh
Or
* * * * * /home/USER/Desktop/abc.sh
In the first case you have to run cronjob from the same user where of the Desktop folder.

Why is my Crontab not running at a specific time?

I have the following cron running that works:
*/2 * * * * cd /toThePath && /usr/bin/env python3 /toThePath test.py
when I have this:
0 16 * * * cd /toThePath && /usr/bin/env python3 /toThePath test.py
It does not send at 16h00. I have tried 0 16 */1 */1 */1.
My script does include #!/usr/bin/env python3 in the beginning
Try running this in terminal.
date
Does the time displayed match your time zone?

Crontab not running a binary file

My crontab file has following:
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
* * * * * /bin/bash /root/bin/run1.sh
run1.sh has
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
export DISPLAY=:0.0
cd /root/madhu_test/bpstest/
touch bhasina.txt
./bpsh-linux-x86-3.1 appTest_saurav.tcl apptest.ini.96.70.NSSTuned NSS-Tuned
./bpsh-linux-x86-3.1 appTest_saurav.tcl apptest.ini.96.70.S2C CriMajMinS2C
But crontab is unable to run the binary file bpsh-linux-x86-3.1. However touch works.

cron job in crontab not working

I have added the following entry:
*/1 * * * * /home/coddict/myapp-dev/spoolemailsender
and the shell that I am trying to execute (the file spoolemailsender) has the following:
#!/bin/sh
php app/console swiftmailer:spool:send --env=dev
Why isn't this script running every 1 minute? Do I need another command to get this cron job running?
You forgot to put user to execute cron job:
*/1 * * * * root /home/coddict/myapp-dev/./spoolemailsender
or
*/1 * * * * root sh /home/coddict/myapp-dev/spoolemailsender
root for example.
Assuming spoolemailsender is executable script and you don't need to do ./spoolemailsender or sh spoolemailsender

Resources