I have been trying to run my crawler using cron but nothing happens and the csv file that my crawler generates does not get updated. In short nothing happens. Here is my below code.
My scrapy project path:
root or home directory/insider/insider/spiders --> This folder contains the insider.py file i.e crawler script.
example.sh
#!/bin/bash
cd/root/insider
PATH =$PATH:/usr/local/bin
export PATH
scrapy crawl insider
crontab file
*/2 * * * * cd /root && example.sh
Related
I followed some other posts in stackoverflow and successfully setup cron with RVM using rvm cron setup and injected some ENV to the crontab file.
When I was troubleshooting why the dotenv gem is not working, I realised the following:
I placed my test.rb in file path /home/myuser/ruby/test.rb and had my crontab file as shown below:
* * * * * ruby /home/myuser/ruby/test.rb >> /home/myuser/ruby/output.log
and when I puts the output of the test.rb with Dir.pwd. The output states that the rb is run in the /home/myuser/ directory instead of /home/myuser/ruby directory.
While I had a hotfix by manually changing the path. But I wonder why it is the case.
By default, cron tasks of a user are executed from the user's home directory. In order to execute the script from proper directory, you have to "cd" to it.
Consider changing your crontab to:
* * * * * cd /home/myuser/ruby && ruby ./test.rb >> /home/myuser/ruby/output.log
Good luck!
According to #Pawel Dawczak who left the answer in the comment.
the solution is to rewrite the statement in crontab as
* * * * * cd /home/myuser/ruby && ruby test.rb >> /home/myuser/ruby/output.log
Thanks!
Simple Question:
I want a cron job to run a script every minute. This script (script.sh) generates a .zip file of all the files in the directory.
I have written the script in a file called script.sh:
if [ -z "$(ls -A /var/www/html/convo_files)" ]; then
exit
else
zip -rj zipped.zip /var/www/html/convo_files/*
fi
Successfully creates .zip file when I run it with command ./script.sh
My crontab script is:
*/1 * * * * /var/www/html/convo_files/script.sh
Gives message in mail (which looks correct):
updating: crap.txt (deflated 89%)
updating: script.sh (deflated 36%)
Cronjob file is within same directory as script.sh btw, yet no .zip file is created with cronjob. Really not sure how to solve this dilemma.
Have you tried out the following cron job ?
*/1 * * * * /path/to/your/script.sh
This should execute your script every minute.
If you have problems executing the script make sure you have the "executable flag" set (chmod +x /path/to/your/script.sh)
If you still have errors you should take a look at your syslog. On most Linux systems the file /var/log/syslog is the currect one but this can differ from system to system.
Command in crontab file needed to be changed:
* * * * * cd /var/www/html/convo_files && /var/www/html/convo_files/script.sh
Problem was working directory wasn't set right (https://unix.stackexchange.com/questions/38951/what-is-the-working-directory-when-cron-executes-a-job )
For beginners like myself, make sure to look at your mail directory/log files to see your errors!
Note, to go into your mail directory:
cd /var/spool/mail
tail root
Thank you to everyone who answered and helped!
I'm trying to execute a PhantomJS JavaScript file via cron job.
Desired Output:
I just want the file itself to run and execute the javascript code.
Here is what I've tried:
* * * * * http://example.com/file.js
I've also tried:
* * * * * wget http://example.com/file.js
this doesn't seem to be doing anything. Am I missing some action verbs before the url path?
Am I missing some action verbs before the url path?
Of course — you're either trying to run a script file (which is not a program to be run *), or you're trying to download a script, which will do just that, only download the file.
The script doesn't run by itself, it must executed by a program therefore you should first run a program and then give it the path to your script as an argument:
* * * * * /usr/bin/phantomjs /path/to/script.js
________
* However by using interpreter directive you could make a script runnable. Let's assume that phantomjs is located at /usr/bin/phantomjs. Then if at the beginning of the script you add
#!/usr/bin/phantomjs
And then make script executable:
chmod +x /path/to/script.js
Then you could just run the script from commandline (or cron directive) by itself:
/path/to/script.js
I'm trying to create a cronjob, I've followed multiple tutorials and searched Stackoverflow but I think I am making a misstake because it doesn't work.
Under Ubuntu 14 this is in my users account Cronjob file
* 16 * * * cd work/homefolder/homefolder && scrapy crawl scrape1
with the use of:
grep CRON /var/log/syslog
I can see the cron job is run but it doesn't execute the command. I also
can not see what is going wrong.
Just typing it in the terminal does work
cd
cd work/homefolder/homefolder && scrapy crawl scrape1
The whole path to the folder is
/home/NAMEOFUSER/work/homefolder/homefolder
Does anybody know why it doesn't work or where I can find what is going wrong?
Thank you
You need to include the absolute path, not the relative path in the cronjob file.
Try * 16 * * * cd /home/NAMEOFUSER/work/homefolder/homefolder && scrapy crawl scrape1
But I'd rather have a script ready that does the work for you (including changing directories) and call that directly:
* 16 * * * /home/NAMEOFUSER/work/homefolder/homefolder/scrape.sh
The contents of the scrape.sh can look like this:
!#/bin/sh
cd /home/NAMEOFUSER/work/homefolder/homefolder && scrapy crawl scrape1
Make sure to update your permissions, so that the cron demon has access.
Furthermore, the preferred way of editing cronjobs on Ubuntu is running crontab -e
I have written a cron job in my Server as below
35 * * * * /home/sites/domain.com/public_html/admin/filename.php
I am trying to call a file in http://domain.com/admin/filename.php.
In Cron job i navigated to every folder by the way its stored in directory in server.But the way to reach the same file in browser is as below
http://domain.com/admin/filename.php
Now the cron filename.php is not running.I checked the file permission every thing is perfect.
I want to know is there a problem in the path i specified
Thanks in advance
When you run php file in cron. You must config as:
35 * * * * /usr/bin/php /home/sites/domain.com/public_html/admin/filename.php
and you want run daemon in server. You can setup as:
nohup php /home/sites/domain.com/public_html/admin/filename.php &