I am following this post. I want to run a python script, in the background, after logging out of ssh, witht the output stored into a specific file. Namely, I wish to use the following bash command:
nohup python3 main.py --dataset CorrSR/testTraining/small --train --input_height=256 --output_height=256 --epoch=2 | at 1:25 PM Mon > logs/background_run_small.txt &
I am not sure regarding the order of the command. is | before >? The command runs with no errors, though a process is immediately opened with
4285 pts/5 Sl 0:02 /usr/bin/python3 -u /usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/start_ipython_kernel.p
and also the output file is immediately created. Is that normal? how do I know the program waits for the designated time to run?
Your command line executes main.py immediately.
What you probably want is:
echo 'nohup python3 main.py --dataset CorrSR/testTraining/small --train --input_height=256 --output_height=256 --epoch=2 > logs/background_run_small.txt' | at "1:25 PM Mon"
Related
I'm puzzled by this problem I'm having on Ubuntu 20.04 where cron is able to run a bash script but the overall outcome is different then when using the shell command.
I've look through all questions I could in here and on Google but couldn't find anyone that had the same problem.
Background:
I'm using Pushgateway to store metrics I'm generating through a bash script, and afterwards it's being imported automatically to Prometheus.
The end goal is to export a list of running processes, their CPU%, Mem% etc, similar to top command.
This is the bash script:
#!/bin/bash
z=$(top -n 1 -bi)
while read -r z
do
var=$var$(awk 'FNR>7{print "cpu_usage{process=\""$12"\", pid=\""$1"\"}", $9z} FNR>7{print "memory_usage{process=\""$12"\", pid=\""$1"\"}", $10z}')
done <<< "$z"
curl -X POST -H "Content-Type: text/plain" --data "$var
" http://localhost:9091/metrics/job/top/instance/machine
I used to have a version that used ps aux but then I found out that it only shows the average CPU% per process.
As you can see, the command I'm running is top -n 1 -bi which gives me a snapshot of active processes and their metrcis.
I'm using awk to format the data, and FNR>7 because I need to ignore the first 7 lines which is the summery presented by top.
The bash scrip is registered on /bin, /usr/bin and /usr/local/bin.
When checking http://localhost:9091/metrics, which is supposed to show me the information gathered, I'm getting this some of information when running the scrip using shell:
cpu_usage{instance="machine",job="top",pid="114468",process="php-fpm74"} 17.6
cpu_usage{instance="machine",job="top",pid="114483",process="php-fpm74"} 11.8
cpu_usage{instance="machine",job="top",pid="126305",process="ffmpeg"} 64.7
And this is the same information when cron is running the same script:
cpu_usage{instance="machine",job="top",pid="114483",process="php-fpm+"} 5
cpu_usage{instance="machine",job="top",pid="126305",process="ffmpeg"} 60
cpu_usage{instance="machine",job="top",pid="128777",process="php"} 15
So, for some reason, when I run it from cron it cuts the process name after 7 places.
I initially though it was related to the FNR>7 but even after changing it to 8 or 9 (and using exec bash to re-register the command) it gives the same results, also when I run it manually it works just fine.
Any help would be appreciated!!
I have a bash script that executes in loop a python script:
while true; do python3 script.py && break; done
launched with nohup:
nohup ./run_script.sh &
Why if I run the command:
tail -f nohup.out
I don't see the output of the python script?
I correctly see the script in my running process:
pi 2757 1.5 3.4 37268 33064 ? S 15:06 0:18 python3 script.py
pi 2819 0.0 0.0 1908 388 ? S 15:07 0:00 /bin/sh ./run_script.sh
pi 2820 1.6 3.5 37268 33096 ? S 15:07 0:18 python3 script.py
If I directly launch the python script with nohup, I see the output but I need to relaunch the script everytime it fails, so I need to lauch it with the bash script
Maybe I'm missing some concept concerning nohup usage.
I resolved modifing the script in this way:
while true; do python3 -u script.py && break; done
The output file nohup.out during execution remains empty until the execution is finished. This happens because of output buffering. Adding the -u flag I can avoid output buffering.
For other details see this blog page https://janakiev.com/blog/python-background/.
When I check the process from the bash, it display:
In [42]: !ps
PID TTY TIME CMD
417 ttys000 0:00.49 -bash
7783 ttys000 0:06.50 /Users/me/anaconda3/bin/python /Users/me/anaconda3/bin/ipython
It seems that pid 7783 run two commands simultaneous,
Could please provide any hints help understand it?
It is running only one command but with one argument:
/Users/me/anaconda3/bin/python /Users/me/anaconda3/bin/ipython
^ command ^ argument
Python scripts are not directly executable. An interpreter is required to actually run them. Similarly, in your case the command is the python interpreter, and the argument is the ipython script.
When you directly execute the script, the operating system peeks inside to see if it has a shebang. This is a line starting with #! (actually the byte sequence 0x2321) followed by the path to the program meant to run the file. For example, on my system the ipython script points at the python3.7 interpreter:
$ head -1 $(which ipython3)
#!/usr/local/opt/python/bin/python3.7
Calling the script automatically expands to calling the shebang program with the script. Thus, you never see the actual script being run by itself - only the interpreter running the script.
$ ipython3 -c '!ps' | grep ipython3
5764 ttys004 0:00.37 /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python /Users/miyagi/Library/Python/3.7/bin/ipython3 -c !ps
I am trying to run a python3 script to check email for a certain condition, every day at a certain time.
I can see crontab invokes the commands, but the scripts does not give me the result I need, I.e. doesn't seem to run. I can see the cron execution in the syslog:
Aug 3 16:25:01 raspberrypi /USR/SBIN/CRON[4597]: (pi) CMD (cd /home/pi/pythonscripts )
Aug 3 16:25:01 raspberrypi /USR/SBIN/CRON[4598]: (pi) CMD (sudo python3 dfj_gmail_test_v1g.py > /home/pi/pythonscripts/cronlog.log)
Aug 3 16:25:02 raspberrypi /USR/SBIN/CRON[4595]: (CRON) info (No MTA installed, discarding output)
The python script need to run, check gmail and if a certain email is found with a certain subject, it should turn on a LED through GPIO on my raspberry, it more a test for me than anything else. If I run the script from the command line itself (using the lines as in my crontab,) the script does the check and light the led. so the script itself is good. Any ideas on how to get more info on why the python script doesnt work through cron?
If need I can add the python 3 code
It seems you have two different cron jobs, one doing
cd /home/pi/pythonscripts
and the other
sudo python3 dfj_gmail_test_v1g.py > /home/pi/pythonscripts/cronlog.log
Those are independent jobs, the first one won't affect the second one, so the working directory for the second job will not be changed and the script will not be found. If you want this to work, you need to use a single command for both, e.g:
cd /home/pi/pythonscripts && sudo python3 dfj_gmail_test_v1g.py > /home/pi/pythonscripts/cronlog.log
Or give the fully qualified path to the python file if the working directory doesn't really matter:
sudo python3 /home/pi/pythonscripts/dfj_gmail_test_v1g.py
Also, instad of using sudo from a crontab, consider using the root user's contab directly. And you should redirect stderr and stdout to your logfile, that will give your more detailled output in case of an error, e.g.:
... > /home/pi/pythonscripts/cronlog.log 2>&1
Or install a MTA, then the output of cront jobs will be sent to the owner as email.
I would like to run a specific script at a certain time (only once!). If I run it normally like this:
marc#Marc-Linux:~/tennis_betting_strategy1/wrappers$ Rscript write_csv2.R
It does work. I however would like to program it in a cronjob to run at 10:50 and therefore did the following:
50 10 11 05 * Rscript ~/csv_file/write_csv.R
This does not seem to work however. Any thoughts where I go wrong? These are the details of the cron package im
using:
PID COMMAND
1015 cron
My system time also checks out:
marc#Marc-Linux:~/tennis_betting_strategy1/wrappers$ date
wo mei 11 10:56:46 CEST 2016
There is a special tool for running commands only once - at.
With at you can schedule a command like this:
at 09:05 am today
at> enter you commands...
Note, you'll need the atd daemon running.
Your crontab entry looks okay, however. I'd suggest checking if the cron daemon is running(exact daemon name depends on the cron package; it could be cron, crond, or vixie-cron, for instance). One way to check if the daemon is running is to use the ps command, e.g.:
$ ps -C cron -o pid,args
PID COMMAND
306 /usr/sbin/cron
Some advices.
Read more about the PATH variable. Notice that it is set differently in interactive shells (see your ~/.bashrc) and in cron or at jobs. See also this about Rscript.
Replace your command by a shell script, e.g. in ~/bin/myrscriptjob.sh
That myrscriptjob.sh file should start with #!/bin/sh
Be sure to make that shell script executable:
chmod u+x ~/bin/myrscriptjob.sh
Add some logging in your shell script, near the start; either use logger(1) or at least some date(1) command suitably redirected, or even both:
#!/bin/sh
# file myrscriptjob.sh
/bin/date +"myrscriptjob starting %c %n" > /tmp/myrscriptjob.start
/usr/bin/logger -t myrscript job starting $$
/usr/local/bin/Rscript $HOME/csv_file/write_csv.R
in the last line above, replace /usr/local/bin/Rscript by the output of which Rscript done in some interactive terminal.
Notice that you should not use ~ (but replace them with $HOME when appropriate) in shell scripts.
Finally, use at to run your script once. If you want to run it periodically in a crontab job, give the absolute path, e.g.
5 09 11 05 * $HOME/bin/myrscriptjob.sh
and check in /tmp/myrscriptjob.start and in your system log if it has started successfully.
BTW, in your myrscriptjob.sh script, you might replace the first line #!/bin/sh with #!/bin/sh -vx (then the shell is verbose about execution, and cron or at will send you some email). See dash(1), bash(1), execve(2)
Use full path (started from /) for both Rscript and write_csv2.R. Check sample command as follows
/tmp/myscript/Rscript /tmp/myfile/write_csv2.R
Ensure you have execution permission of Rscript and write permission in folder where write_csv2.R will be created(/tmp/myfile)