When I check the process from the bash, it display:
In [42]: !ps
PID TTY TIME CMD
417 ttys000 0:00.49 -bash
7783 ttys000 0:06.50 /Users/me/anaconda3/bin/python /Users/me/anaconda3/bin/ipython
It seems that pid 7783 run two commands simultaneous,
Could please provide any hints help understand it?
It is running only one command but with one argument:
/Users/me/anaconda3/bin/python /Users/me/anaconda3/bin/ipython
^ command ^ argument
Python scripts are not directly executable. An interpreter is required to actually run them. Similarly, in your case the command is the python interpreter, and the argument is the ipython script.
When you directly execute the script, the operating system peeks inside to see if it has a shebang. This is a line starting with #! (actually the byte sequence 0x2321) followed by the path to the program meant to run the file. For example, on my system the ipython script points at the python3.7 interpreter:
$ head -1 $(which ipython3)
#!/usr/local/opt/python/bin/python3.7
Calling the script automatically expands to calling the shebang program with the script. Thus, you never see the actual script being run by itself - only the interpreter running the script.
$ ipython3 -c '!ps' | grep ipython3
5764 ttys004 0:00.37 /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python /Users/miyagi/Library/Python/3.7/bin/ipython3 -c !ps
Related
I found ps or pgrep cannot find running script without "#!/bin/bash"
Here is a sample.sh:
while true
do
echo $(date)
done
start the script (ubuntu 18.04, Linux version 4.15.0-101-generic):
$echo $BASH
/bin/bash
./sample.sh
open another terminal, ps only find the command grep
$ps -aux |grep sample.sh
16887 0.0 0.0 16184 1008 pts/4 S+ 07:12 0:00 grep --color=auto sample
pgrep find nothing
$pgrep sample
$
But if I add "#!/bin/bash" to sample.sh, everything works now:
#!/bin/bash <-----add this line
while true
do
echo $(date)
done
I am wondering why.
Let's start with the second of your cases, namely where you do have #!/bin/bash, because it is actually the easier one to deal with first.
With the #!/bin/bash
When you execute a script which starts with #!/path/to/interpreter, the Linux kernel will understand this syntax and will invoke the specified interpreter for you in the same way as if you had explicitly added /path/to/interpreter to the start of the command line. So in the case of your script starting with #!/bin/bash, if you look using ps ux, you will see the command line /bin/bash ./sample.sh.
Without the #!/bin/bash
Now turning to the other one where the #!/bin/bash is missing. This case is more subtle.
A file which is neither a compiled executable nor a file starting with the #! line cannot be executed by the Linux kernel at all. Here is an example of trying to run the sample.sh without the #!/bin/bash line from a python script:
>>> import subprocess
>>> p = subprocess.Popen("./sample.sh")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error
And to show that this is not just a python issue, here is a demonstration of exactly the same thing, but from a C program. Here is the C code:
#include <stdio.h>
#include <unistd.h>
int main() {
execl("./sample.sh", "sample.sh", NULL);
/* exec failed if reached */
perror("exec failed");
return 1;
}
and here is the output:
exec failed: Exec format error
So what is happening here case when you run the script is that because you are invoking it from a bash shell, bash is giving some fault tolerance by running the commands directly after the attempt to "exec" the script has failed.
What is happening in more detail is that:
bash forks a subshell,
inside the subshell it straight away does a call to the Linux kernel to "exec" your executable, and if successful, that would end that (subshell) process and replace it with a process running the executable
however, the exec is not successful, and this means that the subshell is still running
at that point the subshell just reads the commands inside your script and starts executing them directly.
The overall effect is very similar to the #!/bin/bash case, but because the subshell was just started by forking your original bash process, it has the same command line, i.e. just bash, without any command line argument. If you look for this subshell in the output of ps uxf (a tree-like view of your processes), you will see it just as
bash
\_ bash
whereas in the #!/bin/bash case you get:
bash
\_ /bin/bash ./sample.sh
I am running some shell commands with os.system that needs to be run as root.
I tried-
os.system("sudo su")
os.system("other commands")
and also-
home_dir = os.system("sudo su")
os.system("other commands")
But both the above scripts just become root and then stop executing, so the rest of my commands aren't executed.
I'm running Python 3.6.9 on an Ubuntu 18.04 VM.
The root privileges gained by sudo only apply to the command that is run through sudo, and do not raise the privileges of the caller (in this case, your python script). So your first command os.system("sudo su") would run an interactive root shell, but after you have exited from that and then your python code does the subsequent call to os.system("other commands"), these will run under its ordinary user privileges.
You could run each command one at a time via sudo:
os.system("sudo some_command")
os.system("sudo some_other_command")
Note that each command will be separately logged by sudo in the system log, and that even if there are several commands, sudo shouldn't ask for a password more than once within a short time interval.
Or if you need to do a sequence of steps like changing directories that might not be possible in the caller (for example, if the directory is not accessible by the non-root user that is running the python script), then you could do for example:
os.system("sudo sh -c 'cd some_dir && some_other_command'")
(Just for info, && is similar to ; but the other command is only run if the cd succeeded, so it is safer, although this point relates to shell syntax rather than python.)
If there are a lot of commands, of course you also have the option of just making a separate "helper" shell-script and running the entire script through sudo.
os.system("sudo sh /path/to/myscript.sh")
Finally to note, if you are running your python script in a non-interactive environment, you may need to tell sudo not to prompt for a password, at least for the relevant invoking user and target commands. For details, do man sudoers and look for examples involving NOPASSWD.
I am following this post. I want to run a python script, in the background, after logging out of ssh, witht the output stored into a specific file. Namely, I wish to use the following bash command:
nohup python3 main.py --dataset CorrSR/testTraining/small --train --input_height=256 --output_height=256 --epoch=2 | at 1:25 PM Mon > logs/background_run_small.txt &
I am not sure regarding the order of the command. is | before >? The command runs with no errors, though a process is immediately opened with
4285 pts/5 Sl 0:02 /usr/bin/python3 -u /usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/start_ipython_kernel.p
and also the output file is immediately created. Is that normal? how do I know the program waits for the designated time to run?
Your command line executes main.py immediately.
What you probably want is:
echo 'nohup python3 main.py --dataset CorrSR/testTraining/small --train --input_height=256 --output_height=256 --epoch=2 > logs/background_run_small.txt' | at "1:25 PM Mon"
I would like to run a specific script at a certain time (only once!). If I run it normally like this:
marc#Marc-Linux:~/tennis_betting_strategy1/wrappers$ Rscript write_csv2.R
It does work. I however would like to program it in a cronjob to run at 10:50 and therefore did the following:
50 10 11 05 * Rscript ~/csv_file/write_csv.R
This does not seem to work however. Any thoughts where I go wrong? These are the details of the cron package im
using:
PID COMMAND
1015 cron
My system time also checks out:
marc#Marc-Linux:~/tennis_betting_strategy1/wrappers$ date
wo mei 11 10:56:46 CEST 2016
There is a special tool for running commands only once - at.
With at you can schedule a command like this:
at 09:05 am today
at> enter you commands...
Note, you'll need the atd daemon running.
Your crontab entry looks okay, however. I'd suggest checking if the cron daemon is running(exact daemon name depends on the cron package; it could be cron, crond, or vixie-cron, for instance). One way to check if the daemon is running is to use the ps command, e.g.:
$ ps -C cron -o pid,args
PID COMMAND
306 /usr/sbin/cron
Some advices.
Read more about the PATH variable. Notice that it is set differently in interactive shells (see your ~/.bashrc) and in cron or at jobs. See also this about Rscript.
Replace your command by a shell script, e.g. in ~/bin/myrscriptjob.sh
That myrscriptjob.sh file should start with #!/bin/sh
Be sure to make that shell script executable:
chmod u+x ~/bin/myrscriptjob.sh
Add some logging in your shell script, near the start; either use logger(1) or at least some date(1) command suitably redirected, or even both:
#!/bin/sh
# file myrscriptjob.sh
/bin/date +"myrscriptjob starting %c %n" > /tmp/myrscriptjob.start
/usr/bin/logger -t myrscript job starting $$
/usr/local/bin/Rscript $HOME/csv_file/write_csv.R
in the last line above, replace /usr/local/bin/Rscript by the output of which Rscript done in some interactive terminal.
Notice that you should not use ~ (but replace them with $HOME when appropriate) in shell scripts.
Finally, use at to run your script once. If you want to run it periodically in a crontab job, give the absolute path, e.g.
5 09 11 05 * $HOME/bin/myrscriptjob.sh
and check in /tmp/myrscriptjob.start and in your system log if it has started successfully.
BTW, in your myrscriptjob.sh script, you might replace the first line #!/bin/sh with #!/bin/sh -vx (then the shell is verbose about execution, and cron or at will send you some email). See dash(1), bash(1), execve(2)
Use full path (started from /) for both Rscript and write_csv2.R. Check sample command as follows
/tmp/myscript/Rscript /tmp/myfile/write_csv2.R
Ensure you have execution permission of Rscript and write permission in folder where write_csv2.R will be created(/tmp/myfile)
Alright guys, so I try to install rvm in a docker container based on ubuntu:14.04. During the process, I discovered that some people do something like this to ensure docker commands are also run with the bash:
RUN ln -fs /bin/bash /bin/sh
Now The weirdness happens and I hope someone of you can explain it to me:
→ docker run -it --rm d81ff50de1ce /bin/bash
root#e93a877ab3dc:/# ls -lah /bin
....
lrwxrwxrwx 1 root root 9 Mar 1 16:15 sh -> /bin/bash
lrwxrwxrwx 1 root root 9 Mar 1 16:15 sh.distrib -> /bin/bash
...
root#e93a877ab3dc:/# /bin/sh
sh-4.3# echo $0
/bin/sh
Can someone explain what's going on here? I know I could just prefix my commands in the dockerfile w/ bash -c, but I would like to understand what is happening here and if possible still ditch the bash -c prefix in the dockerfile.
Thanks a lot,
Robin
It's because bash has a compatibility mode where it tries to emulate sh if it is started via the name sh, as the manpage says:
If bash is invoked with the name sh, it tries to mimic the startup
behavior of historical versions of sh as closely as possible, while
conforming to the POSIX standard as well. When invoked as an
interactive login shell, or a non-interactive shell with the --login
option, it first attempts to read and execute commands from
/etc/profile and ~/.profile, in that order. The --noprofile option
may be used to inhibit this behavior. When invoked as an interactive
shell with the name sh, bash looks for the variable ENV, expands its
value if it is defined, and uses the expanded value as the name of a
file to read and execute. Since a shell invoked as sh does not
attempt to read and execute commands from any other startup files, the
--rcfile option has no effect. A non-interactive shell invoked with the name sh does not attempt to read any other startup files. When
invoked as sh, bash enters posix mode after the startup files are
read.