How to use Unix regex character with python3 subprocesses run? [duplicate] - python-3.x

This question already has answers here:
subprocess wildcard usage
(3 answers)
Closed 3 years ago.
I'm trying to get the output from a shell command in a Python3 program. I've been looking at the documentation and this is what I've been working with:
sortered=subprocess.run(
# ['sort', time.strftime("%Y")+'*', '|', 'uniq', '-u'], # causes error did not understand * char
#['ls', '-l'], # worked fine
['sort', '2019*'], # gives same error as before
capture_output=True
)
After running the script I get back this error:
$ myscript.py
CompletedProcess(args=['sort', '2019*'], returncode=2, stdout=b'', stderr=b"sort: cannot read: '2019*': No such file or directory\n")
To_Downloadtest1.sh has been created successfully
If I run the command normally using * it works fine.
$ sort 2019*
a file.
a file.
this one is a.
this one is b.
The script is ran from the same directory that the files starting with 2019 are in.
.
├── 2019-A.txt
├── 2019-B.txt
└── myscript.py
What should happen when I run the python script is the output from the command should be put into a variable as a string. This is not happening. The only time I get an error from subprocess.run comes from using * in the command. Otherwise, I get the stdout from subprocess.run correctly. I tried ls -l as a test and it worked correctly. How can I use * with subprocess.run?

The problem here is that you don't have a shell, and so the wildcard does not get expanded.
You can use
sortered = subprocess.run('sort 2019*', shell=True, capture_output=True)
or
import glob
sortered = subprocess.run(['sort'] + glob.glob('2019*'), capture_output=True)
or, of course,
import glob
lines = []
for file in glob.glob('2019*') as handle:
lines.extend(handle.read())
sortered = sorted(lines)

Related

how to get a variable of a python file from bash script

I have a python file, conf.py which is used to store configuration variables. conf.py is given below:
import os
step_number=100
I have a bash script runner.sh which tries to reach the variables from conf.py:
#! /bin/bash
#get step_number from conf file
step_number_=$(python ./conf.py step_number)
However, if I try to print the step_number_ with echo $step_number_, it returns empty value. Can you please help me to fix it?
$(command) is replaced with the standard output of the command. So the Python script needs to print the variable so you can substitute it this way.
import os
step_number = 100
print(step_number)

How to run a python file and scan the generated ips by nmap

I have written a python script to search through shodan, my script code returns a file containing an ip list that each line contains single ip.
here is my code:
import shodan
SHODAN_API="YOUR_SHODAN_API"
api = shodan.Shodan(SHODAN_API)
try:
# Search Using Shodan
results = api.search('EXAMPLE')
# Showing the results
print 'Results found: %s' % results['total']
for result in results['matches']:
print '%s' % result['ip_str']
''' following lines could be uncommented due to more information
Don't Uncomment if you are using scanning methods with the results '''
#print result['data']
#print ''
except shodan.APIError, e:
print 'Error: %s' % e
I was wondering if there is any way to automate the task of running my code and then scanning the ip list by external script or something that work on OSX and Linux ?
You can simply use a bash script like the following one:
#!/bin/bash
python ShodanSearch.py >> IPResult.txt
cat IPResult.txt | while read line
do
sudo nmap -n -Pn -sV -p 80,8080 -oG - $line >> NResult.txt
done
As an alternative to the solution above, you could also execute nmap using the python os module to execute shell commands within your python script, or the now preferred method is with the subprocess module, haven’t personally used the latter, but it can definitely do what you want.

How to get the command line to interpret & as a string without explicitly making it a string

Everytime I run my script and try to pass a & symbol to the end of my command line args it gives back what I believe to be some sort of process ID.
This is my script:
import sys
s = sys.argv[1:]
print(s)
And I run it like this:
$ python3 shell.py hello.sh &
[2] 3211
["hello.sh"]
The ampersand sign & tells the shell to fork itself and run the job in the background. It also prints you the number of the job and its process ID (PID).
If you need more information about job control in a shell, take a look at this blog post.
If you wand to pass it as argument to your program, you need to escape it first with a backslash:
$ python3 test.py hello.sh \&
['hello.sh', '&']

How to pass arguments/parameters when executing bash/shell script from NodeJS [duplicate]

This question already has an answer here:
Execute an shell program with node.js and pass node variable to the shell command [closed]
(1 answer)
Closed 4 years ago.
I have the following code:
exec('sh cert-check-script-delete.sh', req.body.deletedCert);
console.log(req.body.deletedCert);
The console log correctly shows the req.body.deletedCert is non-empty.
And in cert-check-script-delete.sh I have:
#!/bin/sh
certs.json="./" # Name of JSON file and directory location
echo -e $1 >> certs.json
But it's just writing an empty line to certs.json
I've also tried:
exec('sh cert-check-script-delete.sh' + req.body.deletedCert)
But neither formats work
Use execFile(), and pass your arguments out-of-band:
child_process.execFile('./cert-check-script-delete.sh', [req.body.deletedCert])
That way your string (from req.body.deletedCert) is passed as a literal argument, not parsed as code. Note that this requires that your script be successfully marked executable (chmod +x check-cert-script-delete.sh), and that it start with a valid shebang.
If you can't fix your file permissions to make your executable, at least pass the arguments out-of-band:
child_process.execFile('/bin/sh', ['./check-cert-script-delete.sh', req.body.deletedCert])

python3 subprocess in Oracle Linux (wget -o)

I see there are several posts on python subprocess invoking bash shell commands. But I can't find an answer to my problem unless someone has a link that I'm missing.
So here is a start of my code.
import os;
import subprocess;
subprocess.call("wget ‐O /home/oracle/Downloads/puppet-repo.rpm https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm");
When I do
wget ‐O /home/oracle/Downloads/puppet-repo.rpm https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm
straight up in terminal, it works.
But my IDE gives me FileNotFoundError: [Errno 2] No such file or directory: 'wget'
Again, I'm new to invoking os/subprocess module within python and I would appreciate any insight on how to use these modules effectively.
{UPDATE: with miindlek's answer, I get these errors. 1st - subprocess.call(["wget", "‐O", "/home/oracle/Downloads/puppet-repo.rpm", "https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm"])}
--2015-06-07 17:14:37-- http://%E2%80%90o/
Resolving ‐o... failed: Temporary failure in name resolution.
wget: unable to resolve host address “‐o”
/home/oracle/Downloads/puppet-repo.rpm: Scheme missing.
--2015-06-07 17:14:52-- https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm
{with 2nd bash method subprocess.call("wget ‐O /home/oracle/Downloads/puppet-repo.rpm https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm", shell=True)}
Resolving yum.puppetlabs.com... 198.58.114.168, 2600:3c00::f03c:91ff:fe69:6bf0
Connecting to yum.puppetlabs.com|198.58.114.168|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10184 (9.9K) [application/x-redhat-package-manager]
Saving to: “puppetlabs-release-el-6.noarch.rpm.1”
0K ......... 100% 1.86M=0.005s
2015-06-07 17:14:53 (1.86 MB/s) - “puppetlabs-release-el-6.noarch.rpm.1” saved [10184/10184]
FINISHED --2015-06-07 17:14:53--
Downloaded: 1 files, 9.9K in 0.005s (1.86 MB/s)
Process finished with exit code 0
You should split your command string into a list of arguments:
import subprocess
subprocess.call(["wget", "-O", "/home/oracle/Downloads/puppet-repo.rpm", "https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm"])
You could also use the shell option as an alternative:
import subprocess
subprocess.call("wget -O /home/oracle/Downloads/puppet-repo.rpm https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm", shell=True)
By the way, in python you don't need to add semicolons at the end of a line.
Update
The dash in option -O is a utf8 hyphen Charakter, not a dash. See for example:
>>> a = "‐" # utf8 hyphen
>>> b = "-" # dash
>>> str(a)
'\xe2\x80\x9'
>>> str(b)
'-'
You should delete your old dash and relace it by a normal one. I updated the former source code. You can also copy it from there.
This sounds primarily because your IDE is launching that python subprocess from not 'straight up in a terminal'.
This will be a reading suggestion, rather than a direct answer to only this problem.
Check your IDE; read docs about how it launches stuff.
1 - in terminal
type $ env where you tested $ wget
2 - in IDE
import os ; print(os.environ)
3 - read here about shell and Popen
https://docs.python.org/3/library/subprocess.html
Begin the learning process from there.
I would even suggest replacing
subprocess.call("wget -O /home/oracle/Downloads/puppet-repo.rpm https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm", shell=True)
With a clear declaration of what 'shell' you want to use
subprocess.Popen(['/bin/sh', '-c', 'wget' '<stuff>'])
to mitigate future IDE/shell/env assumption problems.

Resources