It does not seem that file I put to /etc/cron.hourly does not seem to work
It's called cron_hourly_homepage, so it does not seem that its filename issue. Ownership seems alright as well: -rwxr-xr-x
run-parts --test /etc/cron.hourly sees this file, run-parts --report /etc/cron.hourly can run it as well
Would like to get it fixed, so no crontab recommendations are necessary :D
● cron.service - Regular background program processing daemon
Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-12-27 20:20:40 GMT; 1 day 23h ago
Docs: man:cron(8)
Main PID: 401 (cron)
Tasks: 1 (limit: 4915)
CPU: 9.213s
CGroup: /system.slice/cron.service
└─401 /usr/sbin/cron -f
Dec 29 17:19:01 raspberrypi CRON[9255]: (CRON) info (No MTA installed, discarding output)
Dec 29 17:19:01 raspberrypi CRON[9255]: pam_unix(cron:session): session closed for user root
Dec 29 18:17:01 raspberrypi CRON[9427]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Dec 29 18:17:01 raspberrypi CRON[9428]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
It is common to forget cron scripts are running without a shell and no environment context.
Please see this answer
Related
I have a simple python script that runs a sql query and converts the query results into csv (and parquet) leveraging pandas.to_csv(path/to/csv-file) (and pandas.to_parquet(path/to/parquet-file)
I'm using Docker Desktop in Windows 10. I have WSL installed -which I believe Docker Desktop somehow connects to that.
My python script 'main.py`:
import vertica_python as vp
import pandas as pd
from datetime import datetime as dt
connection_settings = {
'host': 'host.host',
'port': 5433,
'user': "user",
'password': "password",
'database': 'db'
}
def pull_data_vertica():
sql_file = "sample.sql"
with open(sql_file) as f:
query = f.read()
connection = vp.connect(**connection_settings)
temp = pd.read_sql(query, connection)
connection.close()
_pre = "/app/data_from_vertical_"
_dt = dt.now().strftime('%Y-%m-%d-%H-%M')
csv_out = _pre + _dt + '.csv'
temp.to_csv(csv_out)
parquet_out = _pre + _dt + '.parquet'
temp.to_parquet(parquet_out)
return csv_out, parquet_out
if __name__ == "__main__":
a, b = pull_data_vertica()
The Dcokerfile is as following:
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y build-essential python3.8 python3-pip python3-dev cron vim busybox
WORKDIR /app
RUN pip3 -q install pip --upgrade
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
RUN rm requirements.txt
COPY crontab /etc/cron.d/crontab
RUN chmod 0644 /etc/cron.d/crontab
COPY script/main.py main.py
COPY script/sample.sql sample.sql
RUN chmod 0644 /app/main.py
RUN /usr/bin/crontab /etc/cron.d/crontab
CMD ["cron", "-f"]
here is my crontab:
* * * * * /usr/bin/python3 /app/main.py
I docker build the image, then docker run it in detached mode, and by docker exe ... bash I get into the container bash.
In there, I can manually execute my main.py and everything works. However, the cron job seems to be failing to produce the csv file, i.e., no file is written. I also don't have access to any log file to check if there is any issue with the cron job.
notes:
I see cron is the main process in the container (using top).
I did try to put the corntab content in a .sh script and execute that but all is the same with above: the .sh alone runs perfectly and files are written, however in the cronjob nothing happens.
I added shebang #!/usr/bin/env python3 on top of main.py script.
Update:
I installed busybox and modified the last line of Dockerfile to the following:
CMD busybox syslogd && cron -f
then in the container, I found the content /var/log/messages as following:
Oct 28 02:11:59 0ad085300c89 syslog.info syslogd started: BusyBox v1.30.1
Oct 28 02:11:59 0ad085300c89 cron.info cron[9]: (CRON) INFO (pidfile fd = 3)
Oct 28 02:11:59 0ad085300c89 cron.err cron[9]: Error: bad username; while reading /etc/cron.d/crontab
Oct 28 02:11:59 0ad085300c89 cron.info cron[9]: (*system*crontab) ERROR (Syntax error, this crontab file will be ignored)
Oct 28 02:11:59 0ad085300c89 cron.info cron[9]: (CRON) INFO (Running #reboot jobs)
Oct 28 02:12:01 0ad085300c89 authpriv.err CRON[10]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:12:01 0ad085300c89 authpriv.info CRON[10]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:12:01 0ad085300c89 cron.info CRON[11]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:12:02 0ad085300c89 cron.info CRON[10]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:12:02 0ad085300c89 authpriv.info CRON[10]: pam_unix(cron:session): session closed for user root
Oct 28 02:12:31 0ad085300c89 cron.info crontab[37]: (root) LIST (root)
Oct 28 02:13:01 0ad085300c89 authpriv.err CRON[40]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:13:01 0ad085300c89 authpriv.info CRON[40]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:13:01 0ad085300c89 cron.info CRON[41]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:13:01 0ad085300c89 cron.info CRON[40]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:13:01 0ad085300c89 authpriv.info CRON[40]: pam_unix(cron:session): session closed for user root
Oct 28 02:14:01 0ad085300c89 authpriv.err CRON[59]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:14:01 0ad085300c89 authpriv.info CRON[59]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:14:01 0ad085300c89 cron.info CRON[60]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:14:01 0ad085300c89 cron.info CRON[59]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:14:01 0ad085300c89 authpriv.info CRON[59]: pam_unix(cron:session): session closed for user root
Oct 28 02:15:01 0ad085300c89 authpriv.err CRON[81]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:15:01 0ad085300c89 authpriv.info CRON[81]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:15:01 0ad085300c89 cron.info CRON[82]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:15:02 0ad085300c89 cron.info CRON[81]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:15:02 0ad085300c89 authpriv.info CRON[81]: pam_unix(cron:session): session closed for user root
I am not sure I clearly get what cron is upset for but hope it helps.
Update 2:
Thanks to #jabbson 's comment I updfate the following:
absolute path to output file: /app/... instead of ./..
crontab is * * * * * root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1
here is the new /var/log/messages content:
Oct 28 03:10:36 898c63c3a4e6 syslog.info syslogd started: BusyBox v1.30.1
Oct 28 03:10:36 898c63c3a4e6 cron.info cron[10]: (CRON) INFO (pidfile fd = 3)
Oct 28 03:10:36 898c63c3a4e6 cron.info cron[10]: (CRON) INFO (Running #reboot jobs)
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[20]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[21]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[22]: (root) CMD (/usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[23]: (root) CMD (root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[20]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[21]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[21]: pam_unix(cron:session): session closed for user root
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[20]: pam_unix(cron:session): session closed for user root
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[25]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[26]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[27]: (root) CMD (/usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[28]: (root) CMD (root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[25]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[26]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[25]: pam_unix(cron:session): session closed for user root
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[26]: pam_unix(cron:session): session closed for user root
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[32]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[33]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[34]: (root) CMD (root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[35]: (root) CMD (/usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[33]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[32]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[32]: pam_unix(cron:session): session closed for user root
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[33]: pam_unix(cron:session): session closed for user root
AND NOTHING!!!!!! no log file in /app
Tried the same in Mac OS and it works in there. So this is not related to dataframe.to_csv (or dataframe.to_parquet) but I assume related to Docker Desktop on Windows and Docker Daemon.
I appreciate if you could kindly assist me with this issue.
Thanks,
I'm having a hard time understanding some cron behavior that I was wondering if someone might be able to shed some light on.
I have a server running Centos 6.6. If I run "sudo cat /etc/crontab", the following output is displayed:
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/
# For details see man 4 crontabs
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR
sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
As you can see, there are no actual jobs configured in the system crontab file. However, jobs are indeed getting run from the /etc/cron.d/, /etc/cron.daily/, and /etc/cron.hourly/ directories as you can see from the /var/log/cron file:
Jul 29 03:38:01 <servername> run-parts(/etc/cron.daily)[26524]: finished logrotate
Jul 29 03:38:01 <servername> run-parts(/etc/cron.daily)[26494]: starting update_phishing_sites
Jul 29 03:42:04 <servername> run-parts(/etc/cron.daily)[26565]: finished update_phishing_sites
Jul 29 03:42:04 <servername> run-parts(/etc/cron.daily)[26494]: starting update_spamassassin
Jul 29 03:45:02 <servername> run-parts(/etc/cron.daily)[26587]: finished update_spamassassin
Jul 29 03:45:02 <servername> anacron[26044]: Job `cron.daily' terminated (mailing output)
Jul 29 03:45:02 <servername> anacron[26044]: Normal exit (1 job run)
Jul 29 04:01:01 <servername> CROND[26719]: (root) CMD (run-parts /etc/cron.hourly)
Jul 29 04:01:01 <servername> run-parts(/etc/cron.hourly)[26719]: starting 0anacron
Jul 29 04:01:01 <servername> run-parts(/etc/cron.hourly)[26728]: finished 0anacron
Jul 29 04:01:01 <servername> run-parts(/etc/cron.hourly)[26719]: starting check_MailScanner
Jul 29 04:01:02 <servername> run-parts(/etc/cron.hourly)[26752]: finished check_MailScanner
Jul 29 04:01:02 <servername> run-parts(/etc/cron.hourly)[26719]: starting processing_messages_alert
Jul 29 04:01:02 <servername> run-parts(/etc/cron.hourly)[26766]: finished processing_messages_alert
Jul 29 04:01:02 <servername> run-parts(/etc/cron.hourly)[26719]: starting update_bad_phishing_sites
Jul 29 04:08:37 <servername> run-parts(/etc/cron.hourly)[26863]: finished update_bad_phishing_sites
Jul 29 04:08:37 <servername> run-parts(/etc/cron.hourly)[26719]: starting update_virus_scanners
Jul 29 04:14:52 <servername> run-parts(/etc/cron.hourly)[27187]: finished update_virus_scanners
My question is, what/where/how are those jobs being kicked off? I've seen examples online where the /etc/crontab file looks something like this:
# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
If my /etc/crontab file contained these entries then I would understand how the jobs are getting kicked off. However as stated above, my /etc/crontab file does not contain these entries.
Chris
I should have just spent a few more minutes looking into things before posting the question because I found out how the jobs were being kicked off. The man page for crond on my server states the following:
/etc/crontab
system crontab. Nowadays the file is empty by default. Originally it was usually used to run daily, weekly, monthly jobs. By default these jobs
are now run through anacron which reads /etc/anacrontab configuration file. See anacrontab(5) for more details.
If I look at the /etc/anacrontab file, it contains the following:
# /etc/anacrontab: configuration file for anacron
# See anacron(8) and anacrontab(5) for details.
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22
#period in days delay in minutes job-identifier command
1 5 cron.daily nice run-parts /etc/cron.daily
7 25 cron.weekly nice run-parts /etc/cron.weekly
#monthly 45 cron.monthly nice run-parts /etc/cron.monthly
So, that's how they are being kicked off. :)
I started with cronjobs a while ago, but up until yesterday I've run into a problem I can't figure/find out.
#reboot me /etc/application/start-script.sh
I have Raspbian Jessie (minimal) installed on a Raspberry Pi Zero. One of the users has a cronjob command #reboot. When I check "sudo /etc/init.d/cron status", I can see the cronjob is picked up after a reboot and executed. The only thing is that any output is dropped, the "No MTA installed"-message, (care?).
#!/bin/bash
# My start script
logfile=/home/me/logfile.log
echo "Starting program..." >> $logfile
application
echo "Program started!" >> $logfile
As you can see, it should create a log file, and it does this after a reboot when the script is called as a cronjob. This script works perfectly fine when you manualy execute it, it writes the output to the logfile AND starts the program.
The problem is: the program is not launched when the .sh script is called as a cronjob.
Why is only the application not started when the script is executed???
"sudo /etc/init.d/cron status" output
Mar 17 22:14:45 pizza-pi systemd[1]: Starting Regular background program processing daemon...
Mar 17 22:14:45 pizza-pi systemd[1]: Started Regular background program processing daemon.
Mar 17 22:14:45 pizza-pi cron[292]: (CRON) INFO (pidfile fd = 3)
Mar 17 22:14:45 pizza-pi cron[292]: (CRON) INFO (Running #reboot jobs)
Mar 17 22:14:45 pizza-pi CRON[296]: pam_unix(cron:session): session opened for user me by (uid=0)
Mar 17 22:14:45 pizza-pi CRON[318]: (me) CMD (etc/application/start-script.sh)
Mar 17 22:14:45 pizza-pi CRON[296]: (CRON) info (No MTA installed, discarding output)
Mar 17 22:14:45 pizza-pi pam_unix(cron:session): session closed for user me
Edit the /etc/rc.local file and add the following line in /etc/init.d/cron/start be sure that it should before exit 0.
Follow this link https://rahulmahale.wordpress.com/2014/09/03/solved-running-cron-job-at-reboot-on-raspberry-pi-in-debianwheezy-and-raspbian/
Hope answer is useful for you
I have added an crontab like this:
*/15 * * * * /home/test/demo.py
However, it runs in two times. And, we can see that two logs are logged in /var/log/cron:
Jun 30 20:00:01 demo1.ops.dev CROND[29181]: (root) CMD (/home/test/a.py)
Jun 30 20:00:01 demo1.ops.dev CROND[29180]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jun 30 20:00:01 demo1.ops.dev CROND[29189]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jun 30 20:00:01 demo1.ops.dev CROND[29190]: (root) CMD (/home/test/a.py)
Answered by myself,
I restart the crond daemon. It can not resolve the problem,
I restart the machine. It can not be scheduled properly.
Even though I set the PATH in /etc/crontab as
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/lib/mailman/cron:/usr/lib/mailman/bin
commands found in /usr/lib/mailman/cron are still not found, Thus issuing a mail to root saying
/bin/sh: mailman: command not found
I've debugged the problem, setting up a cron entry
* * * * * /bin/echo "`/bin/date`: $PATH" >> /tmp/crontest.log 2>&1
using crontab -e which actually do write the PATH to /tmp/crontest.log, confirming that the path entered in /etc/crontab is not what cron think it should be.
Fri Feb 14 10:22:01 CET 2014: /usr/bin:/bin
I've also tried to solve it by re-start cron using (both) service crond restart and service crond stop;sleep 5;service crond start (which does the same, but to make absolutely certain that it has been restarted), but this doesn't change anything.
The /etc/crontab file is readable by everyone (permissions is 644 root root)
-rw-r--r-- 1 root root 500 10 feb 10:36 /etc/crontab
/var/log/cron does not show anything about the problem, just what's started and when I restarted the cron
grep -v CMD /var/log/cron
.
.
.
Feb 14 09:45:34 p1kitlst01l crond[12214]: (CRON) INFO (running with inotify support)
Feb 14 09:45:34 p1kitlst01l crond[12214]: (CRON) INFO (#reboot jobs will be run at computer's startup.)
Feb 14 09:48:07 p1kitlst01l crontab[12331]: (root) BEGIN EDIT (root)
Feb 14 09:48:45 p1kitlst01l crontab[12331]: (root) REPLACE (root)
Feb 14 09:48:45 p1kitlst01l crontab[12331]: (root) END EDIT (root)
Feb 14 09:49:01 p1kitlst01l crond[12214]: (root) RELOAD (/var/spool/cron/root)
Feb 14 10:01:01 p1kitlst01l run-parts(/etc/cron.hourly)[13010]: starting 0anacron
Feb 14 10:01:01 p1kitlst01l run-parts(/etc/cron.hourly)[13027]: finished 0anacron
Feb 14 10:01:01 p1kitlst01l run-parts(/etc/cron.hourly)[13010]: starting mcelog.cron
Feb 14 10:01:01 p1kitlst01l run-parts(/etc/cron.hourly)[13039]: finished mcelog.cron
Feb 14 10:19:16 p1kitlst01l crontab[13840]: (root) BEGIN EDIT (root)
Feb 14 10:19:23 p1kitlst01l crontab[13840]: (root) END EDIT (root)
Feb 14 10:27:17 p1kitlst01l crond[14170]: (CRON) STARTUP (1.4.4)
Feb 14 10:27:17 p1kitlst01l crond[14170]: (CRON) INFO (running with inotify support)
Feb 14 10:27:17 p1kitlst01l crond[14170]: (CRON) INFO (#reboot jobs will be run at computer's startup.)
Any suggestions to what I have to look in to?
Henrik
What I think is ,you have two ways to go (maybe others provide more).
1) set the absolute path for mailman in your script.
2) source the .profile, then run the script, such as:
0 1 * * * . ~/.profile; bash your_script.sh