I see there are several posts on python subprocess invoking bash shell commands. But I can't find an answer to my problem unless someone has a link that I'm missing.
So here is a start of my code.
import os;
import subprocess;
subprocess.call("wget ‐O /home/oracle/Downloads/puppet-repo.rpm https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm");
When I do
wget ‐O /home/oracle/Downloads/puppet-repo.rpm https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm
straight up in terminal, it works.
But my IDE gives me FileNotFoundError: [Errno 2] No such file or directory: 'wget'
Again, I'm new to invoking os/subprocess module within python and I would appreciate any insight on how to use these modules effectively.
{UPDATE: with miindlek's answer, I get these errors. 1st - subprocess.call(["wget", "‐O", "/home/oracle/Downloads/puppet-repo.rpm", "https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm"])}
--2015-06-07 17:14:37-- http://%E2%80%90o/
Resolving ‐o... failed: Temporary failure in name resolution.
wget: unable to resolve host address “‐o”
/home/oracle/Downloads/puppet-repo.rpm: Scheme missing.
--2015-06-07 17:14:52-- https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm
{with 2nd bash method subprocess.call("wget ‐O /home/oracle/Downloads/puppet-repo.rpm https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm", shell=True)}
Resolving yum.puppetlabs.com... 198.58.114.168, 2600:3c00::f03c:91ff:fe69:6bf0
Connecting to yum.puppetlabs.com|198.58.114.168|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10184 (9.9K) [application/x-redhat-package-manager]
Saving to: “puppetlabs-release-el-6.noarch.rpm.1”
0K ......... 100% 1.86M=0.005s
2015-06-07 17:14:53 (1.86 MB/s) - “puppetlabs-release-el-6.noarch.rpm.1” saved [10184/10184]
FINISHED --2015-06-07 17:14:53--
Downloaded: 1 files, 9.9K in 0.005s (1.86 MB/s)
Process finished with exit code 0
You should split your command string into a list of arguments:
import subprocess
subprocess.call(["wget", "-O", "/home/oracle/Downloads/puppet-repo.rpm", "https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm"])
You could also use the shell option as an alternative:
import subprocess
subprocess.call("wget -O /home/oracle/Downloads/puppet-repo.rpm https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm", shell=True)
By the way, in python you don't need to add semicolons at the end of a line.
Update
The dash in option -O is a utf8 hyphen Charakter, not a dash. See for example:
>>> a = "‐" # utf8 hyphen
>>> b = "-" # dash
>>> str(a)
'\xe2\x80\x9'
>>> str(b)
'-'
You should delete your old dash and relace it by a normal one. I updated the former source code. You can also copy it from there.
This sounds primarily because your IDE is launching that python subprocess from not 'straight up in a terminal'.
This will be a reading suggestion, rather than a direct answer to only this problem.
Check your IDE; read docs about how it launches stuff.
1 - in terminal
type $ env where you tested $ wget
2 - in IDE
import os ; print(os.environ)
3 - read here about shell and Popen
https://docs.python.org/3/library/subprocess.html
Begin the learning process from there.
I would even suggest replacing
subprocess.call("wget -O /home/oracle/Downloads/puppet-repo.rpm https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm", shell=True)
With a clear declaration of what 'shell' you want to use
subprocess.Popen(['/bin/sh', '-c', 'wget' '<stuff>'])
to mitigate future IDE/shell/env assumption problems.
Related
Up front confession... I'm a python neophyte. Please forgive any ignorance on my part.
As just a way to learn more, I'm trying to write a script that will take my latest Quicken backup and with 7-Zip, create a compressed and encrypted file with a timestamp in the name. I then want to copy it two two locations.
I have succeeded up the the name part. I've created a variable for the time the script starts and I want to append this to a base file name, but I don't seem to be able to do it. When I've tried, the script runs, but nothing is ever created, no compressed file that is. If I remove what I have with regard to the variable for the filename, I do get a compressed and encrypted file as expected.
Here is what I have so far. The print commands are just there for me making sure I've done the 'today' and 'newfile' variables correctly. Also, I'm doing the testing with just an empty .docx file so that it processes quicker.
Thanks in advance for any/all help or suggestions. I really appreciate it.
from datetime import datetime
from pathlib import Path
today = datetime.now()
print(today.strftime('%Y-%m-%d %H:%M:%S'))
newfile = 'Q2020_' + (today.strftime('%m-%d-%Y_%H%M%S'))
print(newfile)
os.chdir('D:/Quicken Backups')
print(os.getcwd())
import subprocess
cmd= ['C:\\Program Files\\7-Zip\\7z.exe', 'a', '-t7z', '-mx9', '"D:\\Quicken Backups\\Quicken2020\\" + newfile ".7z"', '-ppassword', '-mhe', 'D:\\Quicken Backups\\test.docx']
subprocess.Popen(cmd, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
Have you thought about using a python package made for zipping, like zipfile?
This way you dont need to deal with opening a subprocess
import zipfile
dest_zip_folder = '2020_backup.zip' # new file name
file_to_zip = 'D:\\Quicken Backups\\test.docx'
zipfile.ZipFile(dest_zip_folder, mode='w').write(file_to_zip)
If you dont want to use that way, I see an error in your code.
You are missing a + after newfile
cmd = [ ...., .... + newfile + '.7zip', ... ]
Also give subprocess.call() a try
import subproccess
cmd = [. . . . . .]
subprocess.call(cmd)
It took a LOT of mucking around with Windows variables syntax and running the subprocess a little differently, but I finally got the thing working, outputting an encrypted file with a date/time stamp in the file name.
For troubleshooting I added the /K to the cmd command so that the window would not close upon completion. This was a HUGE help in trouble shooting this.
Thanks to everyone who even looked at my posted. I hope someday this thread will help someone else!
import os
from pathlib import Path
os.chdir('D:/Quicken Backups')
print(os.getcwd())
import subprocess
cmd = subprocess.Popen('cmd.exe /K ""C:\\Program Files\\7-Zip\\7z.exe" a -t7z -mx9 "D:\\Quicken Backups\\Quicken2020\\Q2020_%DATE:~4,2%-%DATE:~7,2%-%DATE:~-4%_%TIME:~0,2%%TIME:~3,2%%TIME:~8%.7z" -ppassword -mhe "D:\\Quicken Backups\\test.docx"')
This question already has answers here:
subprocess wildcard usage
(3 answers)
Closed 3 years ago.
I'm trying to get the output from a shell command in a Python3 program. I've been looking at the documentation and this is what I've been working with:
sortered=subprocess.run(
# ['sort', time.strftime("%Y")+'*', '|', 'uniq', '-u'], # causes error did not understand * char
#['ls', '-l'], # worked fine
['sort', '2019*'], # gives same error as before
capture_output=True
)
After running the script I get back this error:
$ myscript.py
CompletedProcess(args=['sort', '2019*'], returncode=2, stdout=b'', stderr=b"sort: cannot read: '2019*': No such file or directory\n")
To_Downloadtest1.sh has been created successfully
If I run the command normally using * it works fine.
$ sort 2019*
a file.
a file.
this one is a.
this one is b.
The script is ran from the same directory that the files starting with 2019 are in.
.
├── 2019-A.txt
├── 2019-B.txt
└── myscript.py
What should happen when I run the python script is the output from the command should be put into a variable as a string. This is not happening. The only time I get an error from subprocess.run comes from using * in the command. Otherwise, I get the stdout from subprocess.run correctly. I tried ls -l as a test and it worked correctly. How can I use * with subprocess.run?
The problem here is that you don't have a shell, and so the wildcard does not get expanded.
You can use
sortered = subprocess.run('sort 2019*', shell=True, capture_output=True)
or
import glob
sortered = subprocess.run(['sort'] + glob.glob('2019*'), capture_output=True)
or, of course,
import glob
lines = []
for file in glob.glob('2019*') as handle:
lines.extend(handle.read())
sortered = sorted(lines)
My Python 3.7.1 script generates a fasta file called
pRNA.sites.fasta
Within the same script, I call following system command:
cmd = "weblogo -A DNA < pRNA.sites.fasta > OUT.eps"
os.system(cmd)
print(cmd) #for debugging
I am getting the following error message and debugging message on the command line.
Error: Please provide a multiple sequence alignment
weblogo -A DNA < pRNA.sites.fasta > OUT.eps
"OUT.eps" file is generated but it's emtpy. On the other hand, if I run the following 'weblogo' command from the command line, It works just find. I get proper OUT.eps file.
$ weblogo -A DNA<pRNA.sites.fasta>OUT.eps
I am guessing my syntax for os.system call is wrong. Can you tell me what is wrong with it? Thanks.
Never mind. It turned out to be that I was not closing my file, "pRNA.sites.fasta" before I make system call that uses this file.
Im currently writing a python3 script that checks out a C source file by running the C code with various of input files. the compilation is done by GCC if it matters.
in some case, the C code enters into an infinite loop (I figured it out because I ran out of memory).
is there a way that I can "protect" my code like a watchdog or something that
tells me a after X minutes that I ran into an infinite loop?
I cant assume anything about the input so i cant have answers like change it or something...
#runfile is an exe file, code file is .c file, inputlist/outputlist are directories
import subprocess as sbp
import os
sbp.call('gcc -o {0} {1}'.format(runfile,codefile), shell = True)
for i in range(numoFiles):
#run the file with input i and save it to output i in outdir
os.system("./{0} <{1} >{2}".format(ID,inputList[i],outputList[i]))
Look up the "Halting Problem". It is not possible to determine whether an arbitrary program will eventually finish or if it will be stuck in a loop forever.
I figured out a way to avoid enter infinite loop by this method:
import subprocess
for i in range(numofiles):
cmd="gcc -o {0} {1}".format(runfile,codefile)
subprocess.call(cmd, shell = True)
try:
cmd="./{0} <{1} >'/dev/null'".format(Cfile,inputfile) #check if runtime>100 sec
subprocess.run(cmd,shell=True,timeout=100)
except subprocess.TimeoutExpired:
print("infinite loop")
continue
cmd="./{0} <{1} >{2}".format(Cfile,inputfile,outputfile)
subprocess.run(cmd,shell=True) #print output to txt file
I'm new to Linux, and have been trying to solve an assignment but to no avail.
I have a shell script which prints out lines of a text file in a certain manner (a line within every few seconds):
python << END
import time,random
a= open ('/home/ch/pshety/course/fielding_history.txt','r')
flag =False
for i in range(1000):
b=a.readline()
if i==402 or flag:
print(a.readline())
flag=True
time.sleep(2)
END
sh th.sh
If I run it without trying to redirect it anywhere, I get the output on the terminal. However, when I tried to redirect it into a new text file, it doesn't do anything - the text remains empty:
sh th.sh > debug.txt
I've tried looking for answers, I've stumbled upon a lot of suggestions including tee but nothing helps - the file remains empty.
What am I doing wrong?
Try this:
import time,random
a = open('/home/ch/pshety/course/fielding_history.txt', 'r')
for i in range(1000):
b = a.readline()
if i >= 402:
print(b, flush=True)
time.sleep(2)
Your Python script likely needs to flush the contents of the output buffer before you can see it.
Note: aside from the sleep() call, Unix provides other ways of accomplishing this. I would take a look at man tail and read about the -f and -n switches.
Edit: didn't realize that tail has a switch (-s) to sleep as well!