Adapting a Complex System Command to subprocess.run() - python-3.x

I am writing a Python3 script, which must run this system command on my Ubuntu 20.04.5 LTS machine:
sudo su - otherUser -c "scp -P 1234 otherUser#10.10.10.10:/path/to/file/x.txt /tmp/x.txt"
When I manually paste this command into the server's command line, the command works just fine. Now I need my Python script to execute it. Thanks to posts like this, I believe that the subprocess.run() command should work here.
Here's my test code:
import subprocess
def main():
subprocess.run(["ls", "-l"]) # For testing purposes
print("===================================================================================")
cmd = ["sudo", "su - otherUser", "-c", "\"scp -P 1234 otherUser#10.10.10.10:/path/to/file/x.txt /tmp/x.txt\""]
print(cmd) # see the command first
subprocess.run(cmd)
if __name__ == "__main__":
main()
Output is:
me#ubuntu01$ python3 testSubprocess.py
total 4
-rw-r----- 1 demo demo 576 Jan 18 21:08 testSubprocess.py
===================================================================================
['sudo', 'su - otherUser', '-c', '"scp -P 1234 otherUser#10.10.10.10:/path/to/file/x.txt /tmp/x.txt"']
sudo: su - otherUser: command not found
me#ubuntu01$
As you can tell, I'm really confused about how to parse the command so that subprocess.run() can understand it. The documentation (here and here) isn't that helpful for a beginner like me. The format of subprocess.run() is:
subprocess.run(args, *, ...lots of other stuff...)
...where args is:
args is required for all calls and should be a string, or a sequence
of program arguments. Providing a sequence of arguments is generally
preferred, as it allows the module to take care of any required
escaping and quoting of arguments (e.g. to permit spaces in file
names). If passing a single string, either shell must be True (see
below) or else the string must simply name the program to be executed
without specifying any arguments.
I don't fully follow that, but I do understand that subprocess.run() will take a list of strings that should be the command-to-be-executed. Okay. Can someone explain how to slice up my original command?

Solution found by trial-and-error. The original code defined the command like this:
cmd = ["sudo", "su - otherUser", "-c", "\"scp -P 1234 otherUser#10.10.10.10:/path/to/file/x.txt /tmp/x.txt\""]
...and the solution that worked:
cmd = ["sudo", "su", "otherUser", "-c", "scp -P 1234 otherUser#10.10.10.10:/path/to/file/x.txt /tmp/x.txt"]
i.e.:
Broke up "su - otherUser" into "su", "otherUser"
Removed the double quotes around the SCP command

Related

How to parse json data correctly using jq to set to var inside shell script [duplicate]

I do this in a script:
read direc <<< $(basename `pwd`)
and I get:
Syntax error: redirection unexpected
in an ubuntu machine
/bin/bash --version
GNU bash, version 4.0.33(1)-release (x86_64-pc-linux-gnu)
while I do not get this error in another suse machine:
/bin/bash --version
GNU bash, version 3.2.39(1)-release (x86_64-suse-linux-gnu)
Copyright (C) 2007 Free Software Foundation, Inc.
Why the error?
Does your script reference /bin/bash or /bin/sh in its hash bang line? The default system shell in Ubuntu is dash, not bash, so if you have #!/bin/sh then your script will be using a different shell than you expect. Dash does not have the <<< redirection operator.
Make sure the shebang line is:
#!/bin/bash
or
#!/usr/bin/env bash
And run the script with:
$ ./script.sh
Do not run it with an explicit sh as that will ignore the shebang:
$ sh ./script.sh # Don't do this!
If you're using the following to run your script:
sudo sh ./script.sh
Then you'll want to use the following instead:
sudo bash ./script.sh
The reason for this is that Bash is not the default shell for Ubuntu. So, if you use "sh" then it will just use the default shell; which is actually Dash. This will happen regardless if you have #!/bin/bash at the top of your script. As a result, you will need to explicitly specify to use bash as shown above, and your script should run at expected.
Dash doesn't support redirects the same as Bash.
Docker:
I was getting this problem from my Dockerfile as I had:
RUN bash < <(curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer)
However, according to this issue, it was solved:
The exec form makes it possible to avoid shell string munging, and
to RUN commands using a base image that does not contain /bin/sh.
Note
To use a different shell, other than /bin/sh, use the exec form
passing in the desired shell. For example,
RUN ["/bin/bash", "-c", "echo hello"]
Solution:
RUN ["/bin/bash", "-c", "bash < <(curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer)"]
Notice the quotes around each parameter.
You can get the output of that command and put it in a variable. then use heredoc. for example:
nc -l -p 80 <<< "tested like a charm";
can be written like:
nc -l -p 80 <<EOF
tested like a charm
EOF
and like this (this is what you want):
text="tested like a charm"
nc -l -p 80 <<EOF
$text
EOF
Practical example in busybox under docker container:
kasra#ubuntu:~$ docker run --rm -it busybox
/ # nc -l -p 80 <<< "tested like a charm";
sh: syntax error: unexpected redirection
/ # nc -l -p 80 <<EOL
> tested like a charm
> EOL
^Cpunt! => socket listening, no errors. ^Cpunt! is result of CTRL+C signal.
/ # text="tested like a charm"
/ # nc -l -p 80 <<EOF
> $text
> EOF
^Cpunt!
do it the simpler way,
direc=$(basename `pwd`)
Or use the shell
$ direc=${PWD##*/}
Another reason to the error may be if you are running a cron job that updates a subversion working copy and then has attempted to run a versioned script that was in a conflicted state after the update...
On my machine, if I run a script directly, the default is bash.
If I run it with sudo, the default is sh.
That’s why I was hitting this problem when I used sudo.
In my case error is because i have put ">>" twice
mongodump --db=$DB_NAME --collection=$col --out=$BACKUP_LOCATION/$DB_NAME-$BACKUP_DATE >> >> $LOG_PATH
i just correct it as
mongodump --db=$DB_NAME --collection=$col --out=$BACKUP_LOCATION/$DB_NAME-$BACKUP_DATE >> $LOG_PATH
Before running the script, you should check first line of the shell script for the interpreter.
Eg:
if scripts starts with /bin/bash , run the script using the below command
"bash script_name.sh"
if script starts with /bin/sh, run the script using the below command
"sh script_name.sh"
./sample.sh - This will detect the interpreter from the first line of the script and run.
Different Linux distributions having different shells as default.

Can docker entrypoint in shell form use runtime command args?

Here is a sample dockerfile with a shell-form entrypoint:
FROM ubuntu
ENTRYPOINT /bin/echo "(shell) ENTRYPOINT#image ($0)"
Below, are some of the outputs that I see from various runs:
$ docker run -it sc-test:v4
(shell) ENTRYPOINT#image (/bin/sh)
$ docker run -it sc-test:v4 /bin/echo
(shell) ENTRYPOINT#image (/bin/echo)
$ docker run -it sc-test:v4 /bin/cat
(shell) ENTRYPOINT#image (/bin/cat)
$ docker run -it sc-test:v4 /bin/dog
(shell) ENTRYPOINT#image (/bin/dog)
$ docker run -it sc-test:v4 "/bin/dog ($0)"
(shell) ENTRYPOINT#image (/bin/dog (-bash))
Based on docker documentation here, we can see that the command args are ignored.
However, the value of $0 changes with the args provided. Can someone explain why this happens? Thanks!
The table in that part of the Docker documentation isn't technically correct: Docker doesn't actually drop or ignore the command part when there's a shell-form entrypoint. What actually happens here (and your example demonstrates) is:
If either the ENTRYPOINT or CMD (or both) is the shell form, it's wrapped in ["/bin/sh", "-c", "..."].
The ENTRYPOINT and CMD lists are concatenated to form a single command list.
Let's take your third example. This is, in Dockerfile syntax,
ENTRYPOINT /bin/echo "(shell) ENTRYPOINT#image ($0)"
CMD ["/bin/cat"]
and the resulting combined command is (in JSON array syntax, expanded for clarity)
[
"/bin/sh",
"-c",
"/bin/echo \"(shell) ENTRYPOINT#image ($0)\"",
"/bin/cat"
]
So, what does sh -c do if you give it multiple arguments? The POSIX spec for the sh command documents the syntax as
sh -c command_string [command_name [argument...]]
and further documents
-c: Read commands from the command_string operand. Set the value of special parameter 0 [...] from the value of the command_name operand and the positional parameters ($1, $2, and so on) in sequence from the remaining argument operands. No commands shall be read from the standard input.
That's what you're seeing in your examples. If ENTRYPOINT is a bare string and CMD is a JSON array, then within the ENTRYPOINT string command, the arguments in CMD can be used as $0, $1, and so on. If both are bare strings, both get wrapped in sh -c, and you'll get something like:
ENTRYPOINT /bin/echo "$0 is /bin/sh, $1 is -c, and then $2"
CMD the rest of the line
In your first example, the command part is empty, and in this case (still from the POSIX sh documentation)
If command_name is not specified, special parameter 0 shall be set to [...] normally a pathname used to execute the sh utility.
Your last example is slightly more subtle:
docker run -it sc-test:v4 "/bin/dog ($0)"
Since the string is double-quoted, your local shell expands the $0 reference in it, which is how bash gets in there; then since it's a single (quoted) word, it becomes the single command_name argument to sh -c.
There's two more normal patterns for using ENTRYPOINT and CMD together. The pattern I prefer has CMD be a full shell command, and ENTRYPOINT does some first-time setup and then runs a command like exec "$#" to run that command. There's also a "container as command" pattern where ENTRYPOINT has a complete command (perhaps with involved JVM arguments) and CMD additional options. In these cases the ENTRYPOINT must be JSON-array syntax:
ENTRYPOINT ["/script-that-exec-dollar-at.sh"]
CMD ["the_command", "--option"]
Since the ENTRYPOINT string doesn't directly reference $0, $1, etc. the CMD arguments are effectively ignored by the sh -c wrapper. If you had ENTRYPOINT script.sh it would be invoked by sh -c as a subprocess with no arguments, and the CMD would get lost.
It's probably clearer for the Docker documentation to say "if ENTRYPOINT is a string then CMD is ignored" than to try to explain the subtleties of this.

Single PID run two commands

When I check the process from the bash, it display:
In [42]: !ps
PID TTY TIME CMD
417 ttys000 0:00.49 -bash
7783 ttys000 0:06.50 /Users/me/anaconda3/bin/python /Users/me/anaconda3/bin/ipython
It seems that pid 7783 run two commands simultaneous,
Could please provide any hints help understand it?
It is running only one command but with one argument:
/Users/me/anaconda3/bin/python /Users/me/anaconda3/bin/ipython
^ command ^ argument
Python scripts are not directly executable. An interpreter is required to actually run them. Similarly, in your case the command is the python interpreter, and the argument is the ipython script.
When you directly execute the script, the operating system peeks inside to see if it has a shebang. This is a line starting with #! (actually the byte sequence 0x2321) followed by the path to the program meant to run the file. For example, on my system the ipython script points at the python3.7 interpreter:
$ head -1 $(which ipython3)
#!/usr/local/opt/python/bin/python3.7
Calling the script automatically expands to calling the shebang program with the script. Thus, you never see the actual script being run by itself - only the interpreter running the script.
$ ipython3 -c '!ps' | grep ipython3
5764 ttys004 0:00.37 /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python /Users/miyagi/Library/Python/3.7/bin/ipython3 -c !ps

How to run command during Docker build which requires a tty?

I have some script I need to run during a Docker build which requires a tty (which Docker does not provide during a build). Under the hood the script uses the read command. With a tty, I can do things like (echo yes; echo no) | myscript.sh.
Without it I get strange errors I don't completely understand. So is there any way to use this script during the build (given that its not mine to modify?)
EDIT: Here's a more definite example of the error:
FROM ubuntu:14.04
RUN echo yes | read
which fails with:
Step 0 : FROM ubuntu:14.04
---> 826544226fdc
Step 1 : RUN echo yes | read
---> Running in 4d49fd03b38b
/bin/sh: 1: read: arg count
The command '/bin/sh -c echo yes | read' returned a non-zero code: 2
RUN <command> in Dockerfile reference:
shell form, the command is run in a shell, which by default is /bin/sh -c on Linux or cmd /S /C on Windows
let's see what exactly /bin/sh is in ubuntu:14.04:
$ docker run -it --rm ubuntu:14.04 bash
root#7bdcaf403396:/# ls -n /bin/sh
lrwxrwxrwx 1 0 0 4 Feb 19 2014 /bin/sh -> dash
/bin/sh is a symbolic link of dash, see read function in dash:
$ man dash
...
read [-p prompt] [-r] variable [...]
The prompt is printed if the -p option is specified and the standard input is a terminal. Then a line
is read from the standard input. The trailing newline is deleted from the line and the line is split as
described in the section on word splitting above, and the pieces are assigned to the variables in order.
At least one variable must be specified. If there are more pieces than variables, the remaining pieces
(along with the characters in IFS that separated them) are assigned to the last variable. If there are
more variables than pieces, the remaining variables are assigned the null string. The read builtin will
indicate success unless EOF is encountered on input, in which case failure is returned.
By default, unless the -r option is specified, the backslash ``\'' acts as an escape character, causing
the following character to be treated literally. If a backslash is followed by a newline, the backslash
and the newline will be deleted.
...
read function in dash:
At least one variable must be specified.
let's see read function in bash:
$ man bash
...
read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] [-t timeout] [-u fd] [name...]
If no names are supplied, the line read is assigned to the variable REPLY. The return code is zero,
unless end-of-file is encountered, read times out (in which case the return code is greater than
128), or an invalid file descriptor is supplied as the argument to -u.
...
So I guess your script myscript.sh is start with #!/bin/bash or something else but not /bin/sh.
Also, you can change your Dockerfile like below:
FROM ubuntu:14.04
RUN echo yes | read ENV_NAME
Links:
https://docs.docker.com/engine/reference/builder/
http://linux.die.net/man/1/dash
http://linux.die.net/man/1/bash
Short answer : You can't do it straightly because docker build or either buildx didn't implement [/dev/tty, /dev/console]. But there is a hacky solution where you can achieve what you need but I highly discourage using it since it break the concept of CI. That's why docker didn't implement it.
Hacky solution
FROM ubuntu:14.04
RUN echo yes | read #tty requirement command
As mentioned in docker reference document the RUN consist of two stage, first is execution of command and the second is commit to the image as a new layer. So you can do the stages manually on your own where we will provide tty to first stage(execution) and then commit the result.
Code:
cd
cat >> tty_wrapper.sh << EOF
echo yes | read ## Your command which needs tty
rm /home/tty_wrapper.sh
EOF
docker run --interactive --tty --detach --privileged --name name1 ubuntu:14.04
docker cp tty_wrapper.sh name1:/home/
docker exec name1 bash -c "cd /home && chmod +x tty_wrapper.sh && ./tty_wrapper.sh "
docker commit name1 your:tag
Your new image is ready.
Here is a description about the code.
At first we make a bash script which wrap our tty to it and then remove itself after fist execute. Then we run a container with provided tty option(you can remove privileged if you don't need). Next step we copy wrapped bash script inside container and do the execution & commit stage on our own.
You don't need a tty for feeding your data to your script . just doing something like (echo yes; echo no) | myscript.sh as you suggested will do. also please make sure you copy your file first before trying to execute it . something like COPY myscript.sh myscript.sh
Most likely you don't need a tty. As the comment on the question shows, even the example provided is a situation where the read command was not properly called. A tty would turn the build into an interactive terminal process, which doesn't translate well to automated builds that may be run from tools without terminals.
If you need a tty, then there's the C library call to openpty that you would use when forking a process that includes a pseudo tty. You may be able to solve your problem with a tool like expect, but it's been so long that I don't remember if it creates a ptty or not. Alternatively, if your application can't be built automatically, you can manually perform the steps in a running container, and then docker commit the resulting container to make an image.
I'd recommend against any of those and to work out the procedure to build your application and install it in a non-interactive fashion. Depending on the application, it may be easier to modify the installer itself.

Linux set date&time via bash script

I'm trying to set a fake date via bash script
I'm using the following commands:
#!/bin/bash
echo 'myPass' | sudo -s 'date -s "1 NOV 2011 09:00:00"'
But I'm getting command error.
What is the right way to do it ?
sudo does not read the password from standard input by default, but from the terminal itself, so you cannot pipe your password into sudo this way. You need to use the -S option to read from standard input.
echo "myPass" | sudo -S date -s "1 NOV 2011 09:00:00"
(note that you don't need to use the -s (lowercase) option; sudo can run date directly without starting an intervening shell).
Exposing your password like this, however, is a security risk. It would be better to configure sudo to allow you (or anyone who is intended to run this script) to run
this particular date command without a password.
For sudo, the first parameter without a dash is the command to execute, the following parameters are the arguments to give to that command. If you wrap the command and its arguments together in quotes (e.g. "echo foo"), then sudo tries to execute the command "echo foo" instead of the command "echo" with parameter "foo". Hence, you need to omit the outermost quotes:
sudo date -s "1 NOV 2011 09:00:00"

Resources