How do we run Sqoop jobs by keeping them in a shell script, and calling them on a particular frequency through crontab?
You can try these steps to execute the job via cron.
1) Create a script file and type the Sqoop command in it.
vim my_sqoop_job.sh
2) Make the script executable.
chmod 755 my_sqoop_job.sh
3) Edit crontab.
crontab -e
4) Add the path and job (below example runs every 30 min)
(Ensure the PATH contains the location for Sqoop's bin directory)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/hdp/current/sqoop-client/bin
*/30 * * * * /location/of/my_sqoop_job.sh >> /location/of/logs/my_sqoop_job_run.log 2>&1
The job will run every 30 minutes and logs will be available in my_sqoop_job_run.log
Related
Here is my shell script, myscript.sh located in ~/bin
cd ../environment
. env/bin/activate
python3 office.py
The script office.py updates the database. I've tested and works with no issue. I used this command ./myscript.sh
Here is cronjob */5 * * * * cd ~/bin/myscript.sh added to crontab -e
When i check database, no changes. The cronjob isn't running? How do i solve?
You are not running the script but just trying to change directories, which will fail as myscript.sh is not a directory. You need to first cd ~/bin as you are using relative paths in your script and then run the script. Use this line:
*/5 * * * * cd ~/bin && ./myscript.sh
Also you may wanna check the syslog to check for cronjobs.
grep CRON /var/log/syslog
Have a look at this thread for more information on logging cronjobs.
I have a cron job that runs the script every 30 minutes. The problem is each time it runs the cron it creates a file in the root directory. It'll create files like this:
wp-cron.php?doing_wp_cron.1
wp-cron.php?doing_wp_cron.2
wp-cron.php?doing_wp_cron.3
This is my cron:
*/30 * * * * wget http://yourdomain.com/wp-cron.php?doing_wp_cron 2>&1 > /dev/null
How can I make it auto delete after it finishes running the cron job or make it not create the file?
wget primarily is for downloading files, it might be better to use curl
curl http://yourdomain.com/wp-cron.php?doing_wp_cron
I need a shell script that would run two .sh files which are in the directory: /opt/tomcat-latest/bin
#!/bin/sh
cd /opt/tomcat-latest/bin
./shutdown.sh
./startup.sh
Would this code achieve my goal? If so how do I make it on a cron job that runs every 2 hours? I have a Linux Centos VPS and DirectAdmin admin panel.
I think the easiest solution would be to run directly the commands with its full path from cron, instead of using a sh script.
Something like this in the crontab would work :
* */2 * * * /opt/tomcat-latest/bin/shutdown.sh && /opt/tomcat-latest/bin/startup.sh
That would run every 2 hours
you can edit crontab with crontab -e and check the crontab syntax here : https://fr.wikipedia.org/wiki/Crontab
I'm running RHEL and I'm trying to set up a cron job to run a shell script every 5 minutes.
Following the directions here: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ch-Automating_System_Tasks.html#s2-configuring-cron-jobs
I have service crond start and chkconfig crond on. Then I edited /etc/crontab and added:
*/5 * * * * my-user /path/to/shell.sh
I did a chmod +x shell.sh. And I made sure to add a new line character at the end.
I'm expecting it to run every 5 minutes but it never executes.
What am I doing wrong?
Simply try to add the cronjob entry and check the script is working fine or not by taking the viewable output in the script.
echo "test time - $(date)" > script.sh
chmod +x script.sh
crontab -e
Then enter the cronjob as below,
*/5 * * * * sh /path/to/script.sh > /path/to/log.file
Check if the log is writing correctly. If its fine, better cross check the script that you are trying to execute via cron. Otherwise it will be a cron issue.
I have build a bash script that gets .tar.gz files from IMDb and writes to a log file, the script works when run on its own as I can see the folder with the files present, but when I run the script via cron it doesn't work. Would this be due to permissions? I have edited the sudo crontab file, but I'm not sure what else I need to do.
Try this solution:
Cronjob is a file that contains your job:
cat cronjob
* * * * * bash /path/to/script.sh >> /path/to/log.txt
Then you should set executable permission and start cron service:
chmod +x cronjob
/etc/init.d/crond start #redhat based servers like centos
/etc/init.d/cron start #debian based servers like ubuntu
After that you should tell cron service to run cronjob file:
crontab cronjob
Your script should download a file.
If your script doesn't run you should run it from good path[full path], so your cronjob file would be something like this:
* * * * * /bin/bash /path/to/script.sh >> /path/to/log.txt