i have many crons running in the same time,
but just one of them runs multiple time (should be once an hour)
but it runs 10-12 times, and then stopes.
the logs look unussual like this:
Jul 14 11:01:02 ip-10-76-186-138 crond[829]: (root) CMD (run-parts /etc/cron.hourly)
Jul 14 11:04:02 ip-10-76-186-138 crontab[2971]: (root) BEGIN EDIT (root)
Jul 14 11:05:00 ip-10-76-186-138 crontab[2971]: (root) REPLACE (root)
Jul 14 11:05:00 ip-10-76-186-138 crontab[2971]: (root) END EDIT (root)
Jul 14 11:05:01 ip-10-76-186-138 crond[1211]: (root) RELOAD (cron/root)
Jul 14 11:05:05 ip-10-76-186-138 crontab[3514]: (root) BEGIN EDIT (root)
Jul 14 11:06:51 ip-10-76-186-138 crontab[3962]: (root) BEGIN EDIT (root)
Jul 14 11:07:04 ip-10-76-186-138 crontab[3962]: (root) REPLACE (root)
Jul 14 11:07:04 ip-10-76-186-138 crontab[3962]: (root) END EDIT (root)
Jul 14 11:07:09 ip-10-76-186-138 crontab[4014]: (root) BEGIN EDIT (root)
Jul 14 11:08:01 ip-10-76-186-138 crond[1211]: (root) RELOAD (cron/root)
Jul 14 11:10:01 ip-10-76-186-138 crond[4606]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Related
I have a simple python script that runs a sql query and converts the query results into csv (and parquet) leveraging pandas.to_csv(path/to/csv-file) (and pandas.to_parquet(path/to/parquet-file)
I'm using Docker Desktop in Windows 10. I have WSL installed -which I believe Docker Desktop somehow connects to that.
My python script 'main.py`:
import vertica_python as vp
import pandas as pd
from datetime import datetime as dt
connection_settings = {
'host': 'host.host',
'port': 5433,
'user': "user",
'password': "password",
'database': 'db'
}
def pull_data_vertica():
sql_file = "sample.sql"
with open(sql_file) as f:
query = f.read()
connection = vp.connect(**connection_settings)
temp = pd.read_sql(query, connection)
connection.close()
_pre = "/app/data_from_vertical_"
_dt = dt.now().strftime('%Y-%m-%d-%H-%M')
csv_out = _pre + _dt + '.csv'
temp.to_csv(csv_out)
parquet_out = _pre + _dt + '.parquet'
temp.to_parquet(parquet_out)
return csv_out, parquet_out
if __name__ == "__main__":
a, b = pull_data_vertica()
The Dcokerfile is as following:
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y build-essential python3.8 python3-pip python3-dev cron vim busybox
WORKDIR /app
RUN pip3 -q install pip --upgrade
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
RUN rm requirements.txt
COPY crontab /etc/cron.d/crontab
RUN chmod 0644 /etc/cron.d/crontab
COPY script/main.py main.py
COPY script/sample.sql sample.sql
RUN chmod 0644 /app/main.py
RUN /usr/bin/crontab /etc/cron.d/crontab
CMD ["cron", "-f"]
here is my crontab:
* * * * * /usr/bin/python3 /app/main.py
I docker build the image, then docker run it in detached mode, and by docker exe ... bash I get into the container bash.
In there, I can manually execute my main.py and everything works. However, the cron job seems to be failing to produce the csv file, i.e., no file is written. I also don't have access to any log file to check if there is any issue with the cron job.
notes:
I see cron is the main process in the container (using top).
I did try to put the corntab content in a .sh script and execute that but all is the same with above: the .sh alone runs perfectly and files are written, however in the cronjob nothing happens.
I added shebang #!/usr/bin/env python3 on top of main.py script.
Update:
I installed busybox and modified the last line of Dockerfile to the following:
CMD busybox syslogd && cron -f
then in the container, I found the content /var/log/messages as following:
Oct 28 02:11:59 0ad085300c89 syslog.info syslogd started: BusyBox v1.30.1
Oct 28 02:11:59 0ad085300c89 cron.info cron[9]: (CRON) INFO (pidfile fd = 3)
Oct 28 02:11:59 0ad085300c89 cron.err cron[9]: Error: bad username; while reading /etc/cron.d/crontab
Oct 28 02:11:59 0ad085300c89 cron.info cron[9]: (*system*crontab) ERROR (Syntax error, this crontab file will be ignored)
Oct 28 02:11:59 0ad085300c89 cron.info cron[9]: (CRON) INFO (Running #reboot jobs)
Oct 28 02:12:01 0ad085300c89 authpriv.err CRON[10]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:12:01 0ad085300c89 authpriv.info CRON[10]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:12:01 0ad085300c89 cron.info CRON[11]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:12:02 0ad085300c89 cron.info CRON[10]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:12:02 0ad085300c89 authpriv.info CRON[10]: pam_unix(cron:session): session closed for user root
Oct 28 02:12:31 0ad085300c89 cron.info crontab[37]: (root) LIST (root)
Oct 28 02:13:01 0ad085300c89 authpriv.err CRON[40]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:13:01 0ad085300c89 authpriv.info CRON[40]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:13:01 0ad085300c89 cron.info CRON[41]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:13:01 0ad085300c89 cron.info CRON[40]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:13:01 0ad085300c89 authpriv.info CRON[40]: pam_unix(cron:session): session closed for user root
Oct 28 02:14:01 0ad085300c89 authpriv.err CRON[59]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:14:01 0ad085300c89 authpriv.info CRON[59]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:14:01 0ad085300c89 cron.info CRON[60]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:14:01 0ad085300c89 cron.info CRON[59]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:14:01 0ad085300c89 authpriv.info CRON[59]: pam_unix(cron:session): session closed for user root
Oct 28 02:15:01 0ad085300c89 authpriv.err CRON[81]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 28 02:15:01 0ad085300c89 authpriv.info CRON[81]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 02:15:01 0ad085300c89 cron.info CRON[82]: (root) CMD (/usr/bin/python3 /app/main.py pull^M)
Oct 28 02:15:02 0ad085300c89 cron.info CRON[81]: (CRON) info (No MTA installed, discarding output)
Oct 28 02:15:02 0ad085300c89 authpriv.info CRON[81]: pam_unix(cron:session): session closed for user root
I am not sure I clearly get what cron is upset for but hope it helps.
Update 2:
Thanks to #jabbson 's comment I updfate the following:
absolute path to output file: /app/... instead of ./..
crontab is * * * * * root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1
here is the new /var/log/messages content:
Oct 28 03:10:36 898c63c3a4e6 syslog.info syslogd started: BusyBox v1.30.1
Oct 28 03:10:36 898c63c3a4e6 cron.info cron[10]: (CRON) INFO (pidfile fd = 3)
Oct 28 03:10:36 898c63c3a4e6 cron.info cron[10]: (CRON) INFO (Running #reboot jobs)
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[20]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[21]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[22]: (root) CMD (/usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[23]: (root) CMD (root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[20]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:11:01 898c63c3a4e6 cron.info CRON[21]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[21]: pam_unix(cron:session): session closed for user root
Oct 28 03:11:01 898c63c3a4e6 authpriv.info CRON[20]: pam_unix(cron:session): session closed for user root
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[25]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[26]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[27]: (root) CMD (/usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[28]: (root) CMD (root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[25]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:12:01 898c63c3a4e6 cron.info CRON[26]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[25]: pam_unix(cron:session): session closed for user root
Oct 28 03:12:01 898c63c3a4e6 authpriv.info CRON[26]: pam_unix(cron:session): session closed for user root
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[32]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[33]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[34]: (root) CMD (root /usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[35]: (root) CMD (/usr/bin/python3 /app/main.py pull >> /app/exec.log 2>&1^M)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[33]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:13:01 898c63c3a4e6 cron.info CRON[32]: (CRON) info (No MTA installed, discarding output)
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[32]: pam_unix(cron:session): session closed for user root
Oct 28 03:13:01 898c63c3a4e6 authpriv.info CRON[33]: pam_unix(cron:session): session closed for user root
AND NOTHING!!!!!! no log file in /app
Tried the same in Mac OS and it works in there. So this is not related to dataframe.to_csv (or dataframe.to_parquet) but I assume related to Docker Desktop on Windows and Docker Daemon.
I appreciate if you could kindly assist me with this issue.
Thanks,
"launcher.sh" is invoked upon every reboot of the system (done through crontab)
In the launcher.sh, a main python script, a memory monitoring python script are invoked.
It works perfectly fine. Even upon reboot also, both the python scripts are getting executed successfully.
One day, by mistake, the NUC system (in which the scripts are running) power button was pressed accidentally(March 16, 12:10 pm). After that, the system gets rebooted; Only the memory monitoring python script was getting executed; Not the main python script.
(i) Can someone help us to understand the reason for this observation?
FYI: Later on, the system is restarted intensionally; Since then, it works fine. (Both the scripts are getting executed)
Pulled out the "last" and "last reboot" results from that system. Please find below:
'''
last reboot
reboot system boot 4.15.0-1094-oem Sun Mar 21 06:01 still running
reboot system boot 4.15.0-1094-oem Thu Mar 18 14:05 - 06:01 (2+15:56)
reboot system boot 4.15.0-1094-oem Mon Mar 15 11:42 - 06:01 (5+18:18)
reboot system boot 4.15.0-1094-oem Sun Mar 14 06:01 - 06:01 (6+23:59)
reboot system boot 4.15.0-1094-oem Thu Mar 11 16:55 - 06:01 (2+13:06)
reboot system boot 4.15.0-1094-oem Mon Mar 8 10:39 - 16:54 (3+06:15)
reboot system boot 4.15.0-1094-oem Sun Mar 7 06:01 - 10:37 (1+04:35)
reboot system boot 4.15.0-1094-oem Sat Mar 6 10:27 - 06:01 (19:33)
reboot system boot 4.15.0-1094-oem Mon Mar 1 14:40 - 10:25 (4+19:45)
reboot system boot 4.15.0-1094-oem Mon Mar 1 11:14 - 14:39 (03:25)
last
linuxdev :0 :0 Sun Mar 21 06:02 still logged in
reboot system boot 4.15.0-1094-oem Sun Mar 21 06:01 still running
linuxdev :0 :0 Thu Mar 18 14:05 - 06:01 (2+15:55)
reboot system boot 4.15.0-1094-oem Thu Mar 18 14:05 - 06:01 (2+15:56)
linuxdev :0 :0 Tue Mar 16 12:11 - crash (2+01:53)
linuxdev :0 :0 Mon Mar 15 11:42 - 12:10 (1+00:27)
reboot system boot 4.15.0-1094-oem Mon Mar 15 11:42 - 06:01 (5+18:18)
linuxdev :0 :0 Sun Mar 14 06:02 - crash (1+05:39)
reboot system boot 4.15.0-1094-oem Sun Mar 14 06:01 - 06:01 (6+23:59)
linuxdev :0 :0 Thu Mar 11 16:55 - down (2+13:05)
reboot system boot 4.15.0-1094-oem Thu Mar 11 16:55 - 06:01 (2+13:06)
linuxdev :0 :0 Mon Mar 8 10:40 - down (3+06:14)
reboot system boot 4.15.0-1094-oem Mon Mar 8 10:39 - 16:54 (3+06:15)
linuxdev :0 :0 Sun Mar 7 06:02 - down (1+04:34)
reboot system boot 4.15.0-1094-oem Sun Mar 7 06:01 - 10:37 (1+04:35)
linuxdev :0 :0 Sat Mar 6 10:28 - down (19:32)
reboot system boot 4.15.0-1094-oem Sat Mar 6 10:27 - 06:01 (19:33)
linuxdev :0 :0 Mon Mar 1 14:40 - down (4+19:45)
reboot system boot 4.15.0-1094-oem Mon Mar 1 14:40 - 10:25 (4+19:45)
linuxdev :0 :0 Mon Mar 1 11:14 - down (03:24)
reboot system boot 4.15.0-1094-oem Mon Mar 1 11:14 - 14:39 (03:25)
'''
A follow up question, (ii) when will we observe "crash" in "last" command?
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 9 years ago.
Improve this question
]# crontab -l
#ROOT CRONTAB test
#22 15 * * * root my_date=`date +\%Y-\%m-\%0e_\%H:\%M:\%S`; /usr/bin/pg_dumpall -U postgres > /home/apache/tactic_backup/postgresDb_${my_date}
53 13 * * * root /home/apache/tactic_custom_tools/backup.sh
]# more /home/apache/tactic_custom_tools/backup.sh
#!/bin/bash
/usr/bin/pg_dumpall -U postgres > /home/apache/tactic_backup/postgresDb/`date +\%Y-\%m-\%0e_\%H:\%M:\%S
It doesn't create anything under /home/apache/tactic_backup/postgresDb/
But when i run the script or command as root, it works and does the required. But via crontab it doesn't The crontab above is for root. Even in logs it shows that it ran(or so)
Jan 28 13:40:01 bjweb01 crond[13935]: (root) RELOAD (/var/spool/cron/root)
Jan 28 13:40:01 bjweb01 CROND[34555]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 28 13:40:01 bjweb01 CROND[34556]: (root) CMD (root /home/apache/tactic_custom_tools/backup.sh)
Jan 28 13:46:47 bjweb01 crontab[34743]: (root) LIST (root)
Jan 28 13:49:53 bjweb01 crontab[34814]: (root) BEGIN EDIT (root)
Jan 28 13:50:01 bjweb01 CROND[34818]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 28 13:52:11 bjweb01 crontab[34814]: (root) REPLACE (root)
Jan 28 13:52:11 bjweb01 crontab[34814]: (root) END EDIT (root)
Jan 28 13:53:01 bjweb01 crond[13935]: (root) RELOAD (/var/spool/cron/root)
Jan 28 13:53:01 bjweb01 CROND[34887]: (root) CMD (root /home/apache/tactic_custom_tools/backup.sh)
Any idea what i am missing?
My documentation states that the mentioning of the user (root in this case) after the time restrictions is only valid in the system crontab which is stored at /etc/crontab.
You are showing us users' crontabs (shown with crontab -l) which differ in that aspect; the user must not be given there, even if the user is root.
Even though I set the PATH in /etc/crontab as
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/lib/mailman/cron:/usr/lib/mailman/bin
commands found in /usr/lib/mailman/cron are still not found, Thus issuing a mail to root saying
/bin/sh: mailman: command not found
I've debugged the problem, setting up a cron entry
* * * * * /bin/echo "`/bin/date`: $PATH" >> /tmp/crontest.log 2>&1
using crontab -e which actually do write the PATH to /tmp/crontest.log, confirming that the path entered in /etc/crontab is not what cron think it should be.
Fri Feb 14 10:22:01 CET 2014: /usr/bin:/bin
I've also tried to solve it by re-start cron using (both) service crond restart and service crond stop;sleep 5;service crond start (which does the same, but to make absolutely certain that it has been restarted), but this doesn't change anything.
The /etc/crontab file is readable by everyone (permissions is 644 root root)
-rw-r--r-- 1 root root 500 10 feb 10:36 /etc/crontab
/var/log/cron does not show anything about the problem, just what's started and when I restarted the cron
grep -v CMD /var/log/cron
.
.
.
Feb 14 09:45:34 p1kitlst01l crond[12214]: (CRON) INFO (running with inotify support)
Feb 14 09:45:34 p1kitlst01l crond[12214]: (CRON) INFO (#reboot jobs will be run at computer's startup.)
Feb 14 09:48:07 p1kitlst01l crontab[12331]: (root) BEGIN EDIT (root)
Feb 14 09:48:45 p1kitlst01l crontab[12331]: (root) REPLACE (root)
Feb 14 09:48:45 p1kitlst01l crontab[12331]: (root) END EDIT (root)
Feb 14 09:49:01 p1kitlst01l crond[12214]: (root) RELOAD (/var/spool/cron/root)
Feb 14 10:01:01 p1kitlst01l run-parts(/etc/cron.hourly)[13010]: starting 0anacron
Feb 14 10:01:01 p1kitlst01l run-parts(/etc/cron.hourly)[13027]: finished 0anacron
Feb 14 10:01:01 p1kitlst01l run-parts(/etc/cron.hourly)[13010]: starting mcelog.cron
Feb 14 10:01:01 p1kitlst01l run-parts(/etc/cron.hourly)[13039]: finished mcelog.cron
Feb 14 10:19:16 p1kitlst01l crontab[13840]: (root) BEGIN EDIT (root)
Feb 14 10:19:23 p1kitlst01l crontab[13840]: (root) END EDIT (root)
Feb 14 10:27:17 p1kitlst01l crond[14170]: (CRON) STARTUP (1.4.4)
Feb 14 10:27:17 p1kitlst01l crond[14170]: (CRON) INFO (running with inotify support)
Feb 14 10:27:17 p1kitlst01l crond[14170]: (CRON) INFO (#reboot jobs will be run at computer's startup.)
Any suggestions to what I have to look in to?
Henrik
What I think is ,you have two ways to go (maybe others provide more).
1) set the absolute path for mailman in your script.
2) source the .profile, then run the script, such as:
0 1 * * * . ~/.profile; bash your_script.sh
I have a problem with executing crontab jobs. I guess I have everything configured properly but when i put:*/2 * * * * /var/www/site/executescript.sh in crontab -e My script is not being executed at all.. Only thing i get in cron.log is:
Feb 15 10:22:35 server crontab[2222]: (root) BEGIN EDIT (root)
Feb 15 10:22:45 server crontab[2222]: (root) REPLACE (root)
Feb 15 10:22:45 server crontab[2222]: (root) END EDIT (root)
Feb 15 10:26:24 server crontab[2329]: (root) BEGIN EDIT (root)
Feb 15 10:27:17 server crontab[2329]: (root) REPLACE (root)
Feb 15 10:27:17 server crontab[2329]: (root) END EDIT (root)
Feb 15 10:29:34 server crontab[2415]: (root) BEGIN EDIT (root)
Feb 15 10:29:53 server crontab[2415]: (root) REPLACE (root)
Feb 15 10:29:53 server crontab[2415]: (root) END EDIT (root)
What mean that cron doesn't even try to execute my script. Do anyone knew what is this related to? Does it mean there is an error in my script ( i don't see one..) Is there a place where i can check more accurate cron logs?
Version of my cron is v5.0
You need to define the binary executing the script. Try with this:
*/2 * * * * /bin/sh /var/www/site/executescript.sh in