In my Case, I want to upload all of pdf file to google cloud storage.
So I use google-sdk with pytho subprocess for practice instead of "google.cloud.storage API".
But there is an error below:
Code:
from subprocess import Popen
def subprocess_cmd(command):
print(f'$>: {command}')
process = Popen(command,
stdout=subprocess.PIPE,
executable='/bin/bash',
shell=True)
proc_stdout = process.communicate()[0].strip()
print(proc_stdout.decode("utf-8"))
Exec Function:
command = "gsutil -m cp -r ./source/*(.pdf|.PDF) gs://<bucket_name>"
subprocess_cmd(command)
Error:
/bin/bash: -c: line 0: syntax error near unexpected token `('
/bin/bash: -c: line 0: `gsutil -m cp -r ./source/*(.pdf|.PDF) gs://<bucket_name>'
You should use this line instead:
command = "gsutil -m cp -r ./source/{*.pdf,*.PDF} gs://<bucket_name>"
See the { } (curly brackets) section in this document about bash wildcards. You can also see this document about gsutil wildcards.
Related
I am trying to get grep to work using pipes and subprocess. I've double-checked the cat and I know it's working, but for some reason the grep isn't returning anything, even though when I run it through the terminal, it works just fine. I'm wondering if I have the command constructed correctly, as it doesn't give the desired output, and I can't figure out why.
I'm trying to retrieve a few specific lines of data from a file I've already retrieved from a server. I've been having a lot of issues with getting grep to work and perhaps I do not simply understand how it works.
p1 = subprocess.Popen(["cat", "result.txt"], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "tshaper"], stdin=p1.stdout,
stdout=subprocess.PIPE)
o = p1.communicate()
print(o)
p1.stdout.close()
out, err = p2.communicate()
print(out)
The output for the file I have when I run this command (cat result.txt | grep "tshaper") on the terminal:
tshaper.1.devname=eth0
tshaper.1.input.burst=0
tshaper.1.input.rate=25000
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
tshaper.1.output.rate=25000
tshaper.1.output.status=enabled
tshaper.1.status=enabled
tshaper.status=disabled
My results running the command in the script:
(b'', b'')
where the tuple is the stdout, stderr respectively of the p2 process.
EDIT:
I changed the command based on the Popen documentation to
p1 = subprocess.Popen(['result.txt', 'cat'], shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, cwd=os.getcwd())
to the p1 subprocess statement. While I was able to get the output in stderr, it didn't really change anything, saying
(b'', b'cat: 1: cat: result.txt: not found\n')
FYI: You got this error: (b'', b'cat: 1: cat: result.txt: not found\n') because you have changed the seq. of commands in your Popen method: ['result.txt', 'cat'] (Based on your question).
I have written a working solution which provides the expected output.
Python3.6.6 has been used for it.
result.txt file:
I have changed some lines to test the grep command.
tshaper.1.devname=eth0
ashaper.1.input.burst=0
bshaper.1.input.rate=25000
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
cshaper.1.output.rate=25000
tshaper.1.output.status=enabled
dshaper.1.status=enabled
tshaper.status=disabled
Code:
I have made an understandable printing but it is not necessary if you only need the output of grep. (It is a bytesl-like object in Python3)
import subprocess
p1 = subprocess.Popen(['cat', 'result.txt'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p2 = subprocess.check_output(["grep", "tshaper"], stdin=p1.stdout)
print("\n".join(p2.decode("utf-8").split(" ")))
Otuput:
You can see the grep filters the lines from cat command as it is expected.
>>> python3 test.py
tshaper.1.devname=eth0
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
tshaper.1.output.status=enabled
tshaper.status=disabled
I am trying to run this using python3
system('echo %s|sudo -S %s' % (password, "date > " + file_path))
but getting this as error
sh: 2: Syntax error: "|" unexpected
A possibly unrelated problem is that you aren't properly quoting the input and output the variables in the command.
system('echo "%s"|sudo -S "%s"' % (password, "date > " + file_path))
However, for all legal filenames as filenames. I would recommend using the subprocess module instead of os.system, leaving the shell out of the process altogether:
subprocess.Popen('echo %s|sudo -S %s' % (password, "date > " + file_path), stdout=subprocess.PIPE)
Groovy code is :
def cmd = "sed -i \'1 i <?xml version=\"1.1\"?>\' test.xml"
println cmd
println cmd.execute().err.text
Output:
sed -i '1 i <?xml version="1.1"?>' test.xml
sed: -e expression #1, char 1: unknown command: `''
Here is the actual command sed -i '1 i <?xml version="1.1"?>' test.xml that runs well in cli. But Groovy execute() does not work correctly. How to fix this ?
Update 1 :
Also tried with the below cmd, butstill shows same error.
def cmd = /sed -i '1 i <?xml version="1.1"?>' test.xml/
def cmd = "sed -i '1 i <?xml version=\"1.1\"?>' test.xml"
Update Note:
To check and update an xml file,
def insertversion(String filename)
{
def lines= new File (filename).readLines()
if(!(lines.get(0)).contains('xml version'))
{
def cmd = ['sed', '-i', '1 i <?xml version="1.1"?>', filename]
cmd.execute()
}
}
In this case execute shell command as a list of cmd and parameters instead of a executing command as String:
def cmd = ['sed', '-i', '1 i <?xml version="1.1"?>', 'test.xml']
println cmd
println cmd.execute().err.text
After running script like this part <?xml version="1.1"> gets duplicated in test.xml file (same behavior as running given command from command line).
What is the difference between List.execute() and String.execute()?
If you execute shell command as a String, java.lang.Runtime.exec() method will use java.util.StringTokenizer to split your input String into an array. In your case tokenizer will create 7 tokens, you can check it by running following script:
def cmd = "sed -i \'1 i <?xml version=\"1.1\"?>\' test.xml"
def tokenizer = new StringTokenizer(cmd)
def tokens = []
while (tokenizer.hasMoreTokens()) {
tokens << tokenizer.nextToken()
}
tokens.each { println it }
It outputs:
sed
-i
'1
i
<?xml
version="1.1"?>'
test.xml
You can also verify it by running debugger with a checkpoint set in java.lang.Runtime class at line 96:
This is of course incorrect. When using a list to execute shell command we will get correct array of command line parameters:
The general rule of thumb is that if your shell command contains characters that may confuse java.util.StringTokenizer it's better to use a list to define correct list of command line parameters.
I expected both of these to behave the same in which stdout is not empty:
assert !"bash -c \"ls *.txt\"".execute().text.empty // assertion failure here
assert !['bash', '-c', 'ls *.txt'].execute().text.empty
but they do not. What are the semantic differences? For the first line I suspect Groovy is sending ["-c", "\"ls", "*.txt\""] as arguments to bash, but I'm not sure. Can anyone confirm that?
Your assumption is correct. See the return code/stderr from thatt command:
groovy:000> p = "bash -c \"ls *.txt\"".execute()
===> java.lang.UNIXProcess#54eb2b70
groovy:000> p.waitFor() // exitValue()
===> 1
groovy:000> p.errorStream.readLines()
===> [*.txt": -c: line 0: unexpected EOF while looking for matching `"', *.txt": -c: line 1: syntax error: unexpected end of file]
If you follow the source down via
https://github.com/apache/groovy/blob/GROOVY_2_4_8/src/main/org/codehaus/groovy/runtime/ProcessGroovyMethods.java#L532-L534
into the JDK's
java.lang.Runtime:exec(String command, String[] envp, File dir)
https://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#exec(java.lang.String,%20java.lang.String[],%20java.io.File)
you will find, that a StringTokenizer is used to split that string/command, thus rendering your " quoting for the shell useless. After all the quoting is only needed for the shell itself and not the ProcessBuilder.
I need some help in Bash function while passing arguments. I need to pass arguments into SQL query in Bash but I am getting error message.
#!/bin/bash --
whw='mysql -uroot -proot -habczx.net'
function foo() {
for v in "$#"; do
eval "$v"; # line 9
echo "${v}";
done
${whw} -e"select id, idCfg, idMfg, $DATE from tblMfg"
}
foo DATE="curdate()"
Below is the error message I get:
$ sh test4.sh
test4.sh: eval: line 9: syntax error near unexpected token `('
test4.sh: eval: line 9: `DATE=curdate()'
test4.sh: line 9: warning: syntax errors in . or eval will cause future versions of the shell to abort as Posix requires
DATE=curdate()
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'from tblMfg limit 4' at line 1
==
If I change in function call to below I do not get any error message:
foo DATE="2014-12-21"
==
Any help?
Thanks!
The problem is the evaluated expression. You can see that by typing it:
$ DATE=bla()
-bash: syntax error near unexpected token `('
If you put it in quotes, it will work:
$ DATE="bla()"
In order to pass the quotes to the eval, they need to be protected from evaluation at callsite:
foo 'DATE="curdate()"'
should do the trick.
BTW: this is rather dangerous to eval a string, especially if the arguments are from untrusted users, one could use foo "rm -rf /" :)
I am not sure why you need the eval, instead of $DATE you could also use $1 and then
${whw} -e"SELECT ... $1...."
foo "curdate()"