I do not find a way to create this kind of arguments structure with the argparse package:
$ python3 prog.py [A (a1 VAR | a2 | a3) | B | C c]
I would like A to be a root for a sub-commands family, in my case a blacklist (./prog blacklist [add PATH | rm | ...]), but I also need two other features : $ ./prog push FILE and finally $ ./prog --interactive.
I've already taken a serious look at the argparse documentation.
I've tried to use add_mutually_exclusive_group() to regroup a1, b1 and c1 but I'm stuck with this :
ValueError: mutually exclusive arguments must be optional
I did manage to have python3 prog.py (A a1 a2| B | C c1) using groups, but either I miss understood something, or argparse has nothing included for this case.
Any help would be appreciated. Thanks for your time !
EDIT : I've finally managed to do what I was looking for. However, I am not convinced this is very clean. It could be great to have some packages which does it better than I.
Thanks to this website, I've found a way to achieve my goal. I attach the code if ever it interests anyone.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import argparse
from sys import argv
class CLI:
"""
Class which represent the CLI's behaviour.
Every function describes a subcommands tree and
how the CLI should respond.
"""
def __init__(self):
parser = argparse.ArgumentParser(
description='',
usage='''cli <command> [<args>]'
Commands list:
cli blacklist Interact with the blacklist.
cli interactive Run an interactive mode for queries.
cli read <path> Try to extract credentials.
'''
)
parser.add_argument('command', help='Subcommand to run')
# only parsing the subcommand
args = parser.parse_args(argv[1:2])
if not hasattr(self, args.command):
print('Unrecognized command')
parser.print_help()
exit(1)
# use dispatch pattern to invoke method with same name
getattr(self, args.command)()
def blacklist(self):
"""
Parser for blacklist methods.
"""
parser = argparse.ArgumentParser(
description='',
usage='''cli blacklist <subcommand> [<args>]
Subcommands list:
add <domain> Add <domain> from the blacklist.
export <path> Export blacklist as csv to <path>.
import <path> [-e] Import <path> as csv. Erase existing blacklist if -e.
remove <domain> Remove <domain> from the blacklist.
show Print the blacklist.
'''
)
parser.add_argument('subcommand')
args = parser.parse_args(argv[2:3])
if not hasattr(self, 'bl_%s' % (args.subcommand)):
print('Unrecognized command')
parser.print_usage()
exit(1)
getattr(self, 'bl_%s' % (args.subcommand))()
def bl_add(self):
print('add `%s` to the blacklist' % (argv[3:4][0]))
def bl_export(self):
print('export')
def bl_import(self):
print('import')
def bl_remove(self):
print('remove')
def bl_show(self):
print('show')
def read(self):
if len(argv) != 3:
print('Wrong number of arguments')
print('usage: cli read <path>')
exit(1)
print('process %s' % (argv[2:3][0]))
def interactive(self):
print('run interactive mode')
CLI()
Related
This is my first time using Pytest, I have a program that is called with command line parameters, as in :
$ myprog -i value_a -o value_b
I am not sure how to use Pytest to test the output of this program. Given values of value_a and value_b, I expect a certain output that I want to test.
The Pytest examples that I see all refer to testing functions, for instance if there is a function such as:
import pytest
def add_nums(x,y):
return x + y
def test_add_nums():
ret = add_nums(2,2)
assert ret == 4
But I am not sure how to call my program using Pytest and not just test individual functions? Do I need to use os.system() and then call my program that way?
In my program I am using argparse module.
The solution is based on monkeypatch fixture. In below example myprog reads number from the file myprog_input.txt adds 2 to it and stores result in myprog_output.txt
Program under test
cat myprog.py
#!/usr/bin/python3.9
import argparse
import hashlib
def main():
parser = argparse.ArgumentParser(description='myprog')
parser.add_argument('-i')
parser.add_argument('-o')
args = parser.parse_args()
with open(args.i) as f:
input_data=int(f.read())
output_data=input_data+2
f.close()
with open(args.o,"w") as fo:
fo.write(str(output_data) + '\n')
fo.close()
with open(args.o) as fot:
bytes = fot.read().encode() # read entire file as bytes
fot.close()
readable_hash = hashlib.sha256(bytes).hexdigest();
return readable_hash
if __name__ == '__main__':
print(main())
Test
cat test_myprog.py
#!/usr/bin/python3.9
import sys
import myprog
def test_myprog(monkeypatch):
with monkeypatch.context() as m:
m.setattr(sys, 'argv', ['myprog', '-i', 'myprog_input.txt', '-o', 'myprog_output.txt'])
assert myprog.main() == 'f0b5c2c2211c8d67ed15e75e656c7862d086e9245420892a7de62cd9ec582a06'
Input file
cat myprog_input.txt
3
Running the program
myprog.py -i myprog_input.txt -o myprog_output.txt
f0b5c2c2211c8d67ed15e75e656c7862d086e9245420892a7de62cd9ec582a06
Testing the program
pytest test_myprog.py
============================================= test session starts =============================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /home/<username>/py
plugins: hypothesis-6.23.1
collected 1 item
test_myprog.py . [100%]
============================================== 1 passed in 0.04s ==============================================
I want to create a script that takes two arguments that should be consumed:
directory_path,
files -- the list of files under the directory_path argument.
I've written something like that:
#!/usr/bin/python3
import argparse
import os
import argcomplete
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("directory_path",
help="a path to some directory",
nargs=1)
# conditional choices of an argument
conditional_choices = [os.listdir(parser.parse_known_args()[0].directory_path[0])]
parser.add_argument("files",
metavar="FILES",
nargs='+',
choices=conditional_choices)
argcomplete.autocomplete(parser)
args = parser.parse_args()
print("directory_path {}".format(args.directory_path))
print("files {}".format(args.files))
So the files argument depends on the directory_path argument.
Using: Python3.8
Problems
For the above snippet, the bash-completion (built from register-python-argcomplete3) for a files argument doesn't work.
If I push enter after the valid command (with path and file) then I'm getting an error
error: argument FILES: invalid choice: ...
First is worth step in argcomplete documentation based on which I created a solution
#!/usr/bin/python3
# PYTHON_ARGCOMPLETE_OK
import argparse
import os
import argcomplete
def files_names(prefix, parsed_args, **kwargs):
absolute_pat = os.path.abspath(parsed_args.directory_path[0])
files = [file for file in os.listdir(absolute_pat) if os.path.isfile(os.path.join(absolute_pat, file))]
return files
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("directory_path",
help="a path to some directory",
nargs=1)
parser.add_argument("files",
metavar="FILES",
nargs='+').completer = files_names
argcomplete.autocomplete(parser)
args = parser.parse_args()
print("directory_path {}".format(args.directory_path))
print("files {}".format(args.files))
*usesfull snippet from argcomplete test directory
To debugging the completion you can set the _ARC_DEBUG variable in your shell to enable verbose debug output
I am writing a script in python 3 which takes space separated arguments from command line. Depending on the first argument I call different methods.
When calling the script it looks like this:
python name.py [-k] [-p pr] number filename
The user will give or [-k] or [-p pr] (not both), number and filename.
for example:
1. python name.py -k 3 filename
if the user gives -k then I want to call a function which will have as arguments the number 3(n1) and the input_filename(filename).
def function1(n1, filename):
code
2. python name.py -p 2 3 filename
if the user gives -r then I want to call a function which will have as arguments the number 2(pr), the number 3(n1) and the input_filename(filename).
def function2(pr, n1, filename):
code
How can I separate the arguments from the command line and use them as arguments for the functions (using argparse)?
import argparse
def function1(n1, filename):
print('doing 1')
def function2(pr, n1, filename):
print('doing 2')
parser = argparse.ArgumentParser()
parser.add_argument('-k', action='store_true')
parser.add_argument('-p', '--pr')
parser.add_argument('number', type=int)
parser.add_argument('filename')
args = parser.parse_args()
print(args)
# could use mutually_exclusive_group, but with this simple alternative
# this is just as good. Tweak as needed.
if args.k:
function1(args.number, args.filename)
elif args.pr is not None:
function2(args.pr, args.number, args.filename)
add in your comment (corrected):
import sys
argv = sys.argv
print(argv)
if argv[1]=="-p":
function2(int(argv[2]), int(argv[3]), argv[4])
if argv[1]=="-k":
function1(int(argv[2]), argv[3])
test
1700:~/mypy$ python3 stack61218370.py -p 2 3 filename
Namespace(filename='filename', k=False, number=3, pr='2')
doing 2
['stack61218370.py', '-p', '2', '3', 'filename']
doing 2
1702:~/mypy$ python3 stack61218370.py -k 3 filename
Namespace(filename='filename', k=True, number=3, pr=None)
doing 1
['stack61218370.py', '-k', '3', 'filename']
doing 1
I try to use the concurrent.future multithreading in Python with subprocess.run to launch an external Python script. But I have some troubles with the shell=True part of the subprocess.run().
Here is an example of the external code, let's call it test.py:
#! /usr/bin/env python3
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('-x', '--x_nb', required=True, help='the x number')
parser.add_argument('-y', '--y_nb', required=True, help='the y number')
args = parser.parse_args()
print('result is {} when {} multiplied by {}'.format(int(args.x_nb) * int(args.y_nb),
args.x_nb,
args.y_nb))
In my main python script I have:
#! /usr/bin/env python3
import subprocess
import concurrent.futures
import threading
...
args_list = []
for i in range(10):
cmd = './test.py -x {} -y 2 '.format(i)
args_list.append(cmd)
# just as an example, this line works fine
subprocess.run(args_list[0], shell=True)
# this multithreading is not working
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
executor.map(subprocess.run, args_list)
The problem here is that I can't pass the shell=True option to the executor.map.
I have already tried without success:
args_list = []
for i in range(10):
cmd = './test.py -x {} -y 2 '.format(i)
args_list.append((cmd, eval('shell=True'))
or
args_list = []
for i in range(10):
cmd = './test.py -x {} -y 2 '.format(i)
args_list.append((cmd, 'shell=True'))
Anyone has an idea on how to solve this problem?
I don't think the map method can call a function with keyword args directly but there are 2 simple solutions to your issue.
Solution 1: Use a lambda to set the extra keyword argument you want
The lambda is basically a small function that calls your real function, passing the arguments through. This is a good solution if the keyword arguments are fixed.
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
executor.map(lambda args: subprocess.run(args, shell=True), args_list)
Solution 2: Use executor.submit to submit the functions to the executor
The submit method lets you specify args and keyword args to the target function.
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
for args in args_list:
executor.submit(subprocess.run, args, shell=True)
I am trying to parametrize a test which is being generated from the
cmdline options in conftest.py.
#!/usr/bin/env python
import pytest
import test
def pytest_addoption(parser):
parser.addoption("--low", action="store", type=int, help="low")
parser.addoption("--high", action="store",type=int, help="high")
#pytest.fixture(scope="session", autouse=True)
def user(request):
return request.config.getoption("low")
#pytest.fixture(scope="session", autouse=True)
def rang(request):
return request.config.getoption("high")
#test_file.py
def data(low, high):
return list(range(low, high))
#pytest.mark.parametrize("num", data(10, 20))
def test(num):
assert num < 1000
I would like run a command like "pytest --low=10 --high=100 test_file.py". Code is working fine with #pytest.mark.parametrize("num", data(x, y)) for the range of values between x and y. I don't want to provide any values in parametrization other than low and high. If I code something like #pytest.mark.parametrize("num", data(low, high)), it throws an error. Is there any way I can get this parametrization work? I know code works when we generate list outside of a method. But I want to write a method for generating list and use that list inside parametrization.
Is there any way I can access these low and high cmdline options anywhere in the test_file.py ?
You can parametrize the test using the pytest_generate_tests hook. Withing the hook, you will have access to the command line args.
# conftest.py
def pytest_addoption(parser):
parser.addoption("--low", action="store", type=int, help="low")
parser.addoption("--high", action="store",type=int, help="high")
def pytest_generate_tests(metafunc):
if 'num' in metafunc.fixturenames:
lo = metafunc.config.getoption('low')
hi = metafunc.config.getoption('high')
metafunc.parametrize('num', range(lo, hi))
# test_file.py
def test_spam(num):
assert num
Another possibility is to access the args via pytest.config, although note that this is a deprecated feature that will be removed soon:
import pytest
def data():
lo = pytest.config.getoption('low')
hi = pytest.config.getoption('high')
return list(range(lo, hi))
#pytest.mark.parametrize('num', data())
def test_spam(num):
assert num