I am trying to write a command line application that has several modes in which it can run (similar to git having clone, pull, etc.). Each of my subcommands have their own options, but I also wanted them to share a set of required options, so I tried using a parent parser to implement this. However, it seems that inheriting a required option is causing the subparser to keep asking for it. Here is an example recreating the behavior:
import argparse
parent_parser = argparse.ArgumentParser(description="The parent parser")
parent_parser.add_argument("-p", type=int, required=True,
help="set the parent required parameter")
subparsers = parent_parser.add_subparsers(title="actions", required=True, dest='command')
parser_command1 = subparsers.add_parser("command1", parents=[parent_parser],
add_help=False,
description="The command1 parser",
help="Do command1")
parser_command1.add_argument("--option1", help="Run with option1")
parser_command2 = subparsers.add_parser("command2", parents=[parent_parser],
add_help=False,
description="The command2 parser",
help="Do command2")
args = parent_parser.parse_args()
So now if I run python test.py I get:
usage: test.py [-h] -p P {command1,command2} ...
test.py: error: the following arguments are required: -p, command
Ok, so far so good. Then if I try to specify just the -p option with python test.py -p 3 I get:
usage: test.py [-h] -p P {command1,command2} ...
test.py: error: the following arguments are required: command
But then if I run python test.py -p 3 command1
I get:
usage: test.py command1 [-h] -p P [--option1 OPTION1] {command1,command2} ...
test.py command1: error: the following arguments are required: -p, command
If I add another -p 3 it still asks to specify command again, and then if I add that again it asks for another -p 3 etc.
If I don't make the -p option required the problem is fixed, but is there a way to share required options among multiple subparsers without just copy pasting them within each subparser? Or am I going about this entirely the wrong way?
Separate the main parser and parent parser functionality:
import argparse
parent_parser = argparse.ArgumentParser(description="The parent parser", add_help=False)
parent_parser.add_argument("-p", type=int, required=True,
help="set the parent required parameter")
main_parser = argparse.ArgumentParser()
subparsers = main_parser.add_subparsers(title="actions", required=True, dest='command')
parser_command1 = subparsers.add_parser("command1", parents=[parent_parser],
description="The command1 parser",
help="Do command1")
parser_command1.add_argument("--option1", help="Run with option1")
parser_command2 = subparsers.add_parser("command2", parents=[parent_parser],
description="The command2 parser",
help="Do command2")
args = main_parser.parse_args()
The main parser and subparser both write to the same args namespace. If both define a 'p' argument, the subparser's value (or default) will overwrite any value set by the main. So it's possible to use the same 'dest' in both, but it is generally not a good idea. And each parser has to meet its own 'required' specifications. There's no 'sharing'.
The parents mechanism copies arguments from the parent to the child. So it saves typing or copy-n-paste, but little else. It's most useful when the parent is defined elsewhere and imported. Actually it copies by reference, which sometimes raises problems. In general it isn't a good idea to run both the parent and child. Use a 'dummy' parent.
Related
I am using mod_wsgi to run my Django app. As there are a plethora of options to define when the server run command is fired, I was trying to create some kind of python script to pass the options and their pre-set values.
For example:
Instead of using:
$python3 manage.py runmodwsgi --processes 3 --threads 1
I am trying to use a python script and use it as:
$python3 runme.py
where runme.py is the script file I am trying to use to start the server.
What I have done so far?
Created the script file:
import os
from django.core.management import execute_from_command_line
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<proj>.settings')
myargs = 'runmodwsgi'
list_of_args = ['', myargs]
execute_from_command_line(list_of_args)
As expected the server started with preset options, for example:
Now what I am trying to achieve is the pass values of certain options like:
--processes 3 --threads 1
and so on.
Is there a way I may pass the preset values (as I may be able to define in my script file runme.py), say something like adding to the list of arguments:
list_of_args = ['', myargs, addl_args]
I have been checking SO queries posted in addition to help available on python site, but could not get my head around to the problem.
I tried the following which is not very helpful though:
import os
import argparse # New import
from django.core.management import execute_from_command_line
# Addition of new code lines
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-p", "--processes", action="store_true")
group.add_argument("-t", "--threads", action="store_true")
parser.add_argument("pint", type=int, help="no. of processes")
parser.add_argument("tint", type=int, help="no. of threads")
args = parser.parse_args()
# End of new code
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<proj>.settings')
myargs = 'runmodwsgi'
# At this line if I add ", args" (like shown below:])
list_of_args = ['', myargs, args]
execute_from_command_line(list_of_args)
and run the command, it comes up with error:
TypeError: 'Namespace' object does not support indexing
If I simply run: python3 runme.py, I get the following error:
usage: runme.py [-h] [-p | -t] pint tint
runme.py: error: the following arguments are required: pint, tint
Whereas, using
python3 runme.py 3 1
starts the server but the options integers "3" and "1" does not have any effect (as intended for no. of processes and threads).
If I use:
python3 runme.py --processes 3 --threads 1
I get the following error:
usage: runme.py [-h] [-p | -t] pint tint
runme.py: error: argument -t/--threads: not allowed with argument -p/--processes
Tried with a single arg like:
python3 runme.py --processes 3 1
The server starts at this time but the options value/s are not affected.
How do I define the option values and pass these preset values of options to the run command?
It looks like you're just misunderstanding/misusing argparse.
import os
import sys
import argparse
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<proj>.settings')
parser = argparse.ArgumentParser()
parser.add_argument("-p", "--processes", type=int, default=1)
parser.add_argument("-t", "--threads", type=int, default=1)
args = parser.parse_args()
from django.core.management import execute_from_command_line
command = [sys.argv[0], 'runmodwsgi', '--processes', str(args.processes), '--threads', str(args.threads)]
execute_from_command_line(command)
might be closer to what you want; it will default processes and threads both to 1.
Is there a way to make an argument required to be true if another specific argument choice is present otherwise argument required is false?
For example the following code if argument -act choice select is 'copy' then the argument dp required is true otherwise required is false:
import argparse
ap = argparse.ArgumentParser()
ap.add_argument("-act", "--act", required=True, choices=['tidy','copy'],type=str.lower,
help="Action Type")
ap.add_argument("-sp", "--sp", required=True,
help="Source Path")
args = vars(ap.parse_args())
if args["act"] == 'copy':
ap.add_argument("-dp", "--dp", required=True,
help="Destination Path")
else:
ap.add_argument("-dp", "--dp", required=False,
help="Destination Path")
args = vars(ap.parse_args())
### Tidy Function
def tidy():
print("tidy Function")
### Copy Function
def copy():
print("copy Function")
### Call Functions
if args["act"] == 'tidy':
tidy()
if args["act"] == 'copy':
copy()
I am currently getting an error unrecognized arguments: -dp with the above code.
The expected result would be to move on to call function. Thanks
I would use ArgumentParser.add_subparsers to define the action type {tidy, copy} and give the command specific arguments. Using a base parser with parents allows you to define arguments that are shared by both (or all) your sub-commands.
import argparse
parser = argparse.ArgumentParser(
prog='PROG',
epilog="See '<command> --help' to read about a specific sub-command."
)
base_parser = argparse.ArgumentParser(add_help=False)
base_parser.add_argument("--sp", required=True, help="source")
subparsers = parser.add_subparsers(dest='act', help='Sub-commands')
A_parser = subparsers.add_parser('copy', help='copy command', parents=[base_parser])
A_parser.add_argument('--dp', required=True, help="dest, required")
B_parser = subparsers.add_parser('tidy', help='tidy command', parents=[base_parser])
B_parser.add_argument('--dp', required=False, help="dest, optional")
args = parser.parse_args()
if args.act == 'copy':
pass
elif args.act == 'tidy':
pass
print(args)
Which produces the following help pages, note that instead of needing to use -act the command is given as a positional parameter.
~ python args.py -h
usage: PROG [-h] {tidy,copy} ...
positional arguments:
{tidy,copy} Sub-commands
tidy tidy command
copy copy command
optional arguments:
-h, --help show this help message and exit
See '<command> --help' to read about a specific sub-command.
~ python args.py copy -h
usage: PROG copy [-h] --sp SP [--dp DP]
optional arguments:
-h, --help show this help message and exit
--sp SP source
--dp DP dest, optional
~ python args.py tidy -h
usage: PROG tidy [-h] --sp SP --dp DP
optional arguments:
-h, --help show this help message and exit
--sp SP source
--dp DP dest, required
ap.add_argument("-dp", "--dp", help="Destination Path")
args = parser.parse_args()
if args.copy is in ['copy']:
if args.dp is None:
parser.error('With copy, a dp value is required')
Since a user can't set a value to None, is None is a good test for arguments that haven't been used.
The parser has a list of defined arguments, parser._actions. Each has a required attribute. During parsing it maintains a set of seen-actions. At the end of parsing it just checks this set against the actions for which required is True, and issues the error message if there are any.
I would argue that testing after parsing is simpler than trying to set the required parameter before hand.
An alternative is to provide dp with a reasonable default. Then you won't care whether the user provides a value or not.
I can imagine defining a custom Action class for copy that would set the required attribute of dp, but overall that would be more work.
I am trying to pass arguments from a pytest testcase to a module being tested. For example, using the main.py from Python boilerplate, I can run it from the command line as:
$ python3 main.py
usage: main.py [-h] [-f] [-n NAME] [-v] [--version] arg
main.py: error: the following arguments are required: arg
$ python3 main.py xx
hello world
Namespace(arg='xx', flag=False, name=None, verbose=0)
Now I am trying to do the same with pytest, with the following test_sample.py
(NOTE: the main.py requires command line arguments. But these arguments need to be hardcoded in a specific test, they should not be command line arguments to pytest. The pytest testcase only needs to send these values as command line arguments to main.main().)
import main
def test_case01():
main.main()
# I dont know how to pass 'xx' to main.py,
# so for now I just have one test with no arguments
and running the test as:
pytest -vs test_sample.py
This fails with error messages. I tried to look at other answers for a solution but could not use them. For example, 42778124 suggests to create a separate file run.py which is not a desirable thing to do. And 48359957 and 40880259 seem to deal more with command line arguments for pytest, instead of passing command line arguments to the main code.
I dont need the pytest to take command line arguments, the arguments can be hardcoded inside a specific test. But these arguments need to be passed as arguments to the main code. Can you give me a test_sample.py, that calls main.main() with some arguments?
If you can't modify the signature of the main method, you can use the monkeypatching technique to temporarily replace the arguments with the test data. Example: imagine writing tests for the following program:
import argparse
def main():
parser = argparse.ArgumentParser(description='Greeter')
parser.add_argument('name')
args = parser.parse_args()
return f'hello {args.name}'
if __name__ == '__main__':
print(main())
When running it from the command line:
$ python greeter.py world
hello world
To test the main function with some custom data, monkeypatch sys.argv:
import sys
import greeter
def test_greeter(monkeypatch):
with monkeypatch.context() as m:
m.setattr(sys, 'argv', ['greeter', 'spam'])
assert greeter.main() == 'hello spam'
When combined with the parametrizing technique, this allows to easily test different arguments without modifying the test function:
import sys
import pytest
import greeter
#pytest.mark.parametrize('name', ['spam', 'eggs', 'bacon'])
def test_greeter(monkeypatch, name):
with monkeypatch.context() as m:
m.setattr(sys, 'argv', ['greeter', name])
assert greeter.main() == 'hello ' + name
Now you get three tests, one for each of the arguments:
$ pytest -v test_greeter.py
...
test_greeter.py::test_greeter[spam] PASSED
test_greeter.py::test_greeter[eggs] PASSED
test_greeter.py::test_greeter[bacon] PASSED
A good practice might to have this kind of code, instead of reading arguments from main method.
# main.py
def main(arg1):
return arg1
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='My awesome script')
parser.add_argument('word', help='a word')
args = parser.parse_args()
main(args.word)
This way, your main method can easily be tested in pytest
import main
def test_case01():
main.main(your_hardcoded_arg)
I am not sure you can call a python script to test except by using os module, which might be not a good practice
How does argparse work? I was told to 'hide' the password from the psycopg2 connection I am building so to be able to run the script automatically every week and being able to share it between departments. This is the beginning of the psycopg2 script where the password is asked:
#Connect to database
conn_string = "host='my_aws_postgresql_database.rds.amazonaws.com' dbname='my_database_name' user='my_username' password='my_password'"
# print the connection string we will use to connect
print ("\"Connecting to database\n ->%s\"" % (conn_string))
Now, how would I use argparse (and getpass) to hide my password? I found this script about this subject a couple of times (I would delete the print statement after getting it to work):
import argparse
import getpass
class Password(argparse.Action):
def __call__(self, parser, namespace, values, option_string):
if values is None:
values = getpass.getpass()
setattr(namespace, self.dest, values)
parser = argparse.ArgumentParser('Test password parser')
parser.add_argument('-p', action=Password, nargs='?', dest='password',
help='Enter your password')
args = parser.parse_args()
print (args.password)
I tried to add the argparse snippet above the #Connect to database code. And replaced the password part on line 2 with
conn_string =
"host='my_aws_postgresql_database.rds.amazonaws.com'
dbname='my_database_name'
user='my_username'
password='" + args + "'"
Then I tried to run the whole script with the command python3 my_script_file.py my_password -p I get asked for the password which I entered, but this rendered the following error
usage: Test password parser [-h] [-p [PASSWORD]]
Test password parser: error: unrecognized arguments: my_password
If I use python3 my_script_file.py my_password I get the same error, but I did not have to enter the password (again).
Am I close to the solution? Is this the standard way of doing this?
The problem was that I used python3 my_script_file.py my_password -p instead of the correct order python3 my_script_file.py -p my_password, see accepted answer below by #hpaulj and the comments to that answer.
This parser is gives the user 2 ways of entering the password, on the commandline, or with a separate getpass prompt:
import argparse
import getpass
class Password(argparse.Action):
def __call__(self, parser, namespace, values, option_string):
if values is None:
values = getpass.getpass()
setattr(namespace, self.dest, values)
parser = argparse.ArgumentParser('Test password parser')
parser.add_argument('-p', action=Password, nargs='?', dest='password',
help='Enter your password')
args = parser.parse_args()
print (args)
Sample runs:
0911:~/mypy$ python3 stack44571340.py
Namespace(password=None)
0912:~/mypy$ python3 stack44571340.py -p test
Namespace(password='test')
0912:~/mypy$ python3 stack44571340.py -p
Password:
Namespace(password='testing')
0912:~/mypy$ python3 stack44571340.py test
usage: Test password parser [-h] [-p [PASSWORD]]
Test password parser: error: unrecognized arguments: test
I tested without any arguments (got the default None)`; with '-p test' which uses the 'test' string; with just '-p', which asks; and without '-p', which produces the error.
I don't know why python3 my_script_file.py -p my_password produced an error; my best guess there's a typo in your parser definition (something wrong with nargs?).
It's not entirely clear how you merged this parser code into the larger script. Done right it shouldn't have changed the behavior of the parser.
The password argument would be used as:
password='" + args.password + "'"
The echo argument, is a positional one, which requires a string. In contrast the -p with nargs='?', is an optional flagged argument, which allows for the three way input I illustrated.
parser.add_argument("echo")
Thank you #CarlShiles, your answer didn't work with that long argparse/getpass snippet from above, but it made me realise that I could just echo the password in there. So I did a simple
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("echo")
args = parser.parse_args()
and then used your suggestion, password='" + args.echo + "'". And then ran the following command python3 my_script_file.py my_password. This worked just fine.
I am trying to write a groovy script which uses CliBuilder to read command line arguments.
Here's my code
def cli = new CliBuilder(usage:'groovy -s "server name" -r "file name"')
cli.s('Name of the server', required: true)
cli.r('Name of the file', required: true)
def args = ['-s', 'ServerEx', '-r', 'RakeEx']
def opt = cli.parse(args)
when I run this script, it gives me error Missing required option: r
even though I have provided the argument r.
I searched for the solution but I found reading multiple arguments for the same flags,
eg. -s arg1 arg2, but not reading multiple arguments for different flags.
Any help is really appreciated. Thank you.
Well, as it turns out, I have to specify the args:1 as a property when defining flags. I just found this from this article from Javaworld: http://www.javaworld.com/article/2073443/explicitly-specifying--args--property-with-groovy-clibuilder.html
Also, as an added bonus, now I can directly access the value provided by using opt.r. It used to show boolean true/false before adding args:1