I have a module which contains the python code and I execute it using the following command:
python script.py \
--eps 12 \
--minpts \
--train \
--predict \
--lower_case \
--input_file data.csv \
--dev_file devdata.csv \
--output_dir /output/
All I want to do is to execute the above command through a python function. Is there any way of doing it?
I don't know why everyone is having such difficulty with this question, it's perfectly clear, unfortunately it's also difficult to do what you want because python uses implicit data types, and that's uncommon. As a result all command line arguments are passed to python as strings.
I'd check this out for the details:
https://www.tutorialspoint.com/python/python_command_line_arguments.htm
but the tldr is to have inside your python script:
import sys
eps = int(sys.argv[sys.argv.index('eps')+1])
minpts = True if '-minputs' in sys.argv else False
…
Obviously this isn't ideal, or pretty but it is quick and easy.
Alternatively you can use the argparser library:
https://docs.python.org/3/library/argparse.html
For a more robust and user friendly solution. Hope this helps
A.
Edit:
I was missing the ' around eps
Command-Line Arguments
import sys
print 'version is', sys.version
The first line imports a library called sys, which is short for "system". It defines values such as sys.version, which describes which version of Python we are running.
This command tells the python interpreter installed in your machine to run program sys-version.py from the current directory.
Here's another script that does something more interesting:import sys
print 'sys.argv is', sys.argv
Related
I have a python file, conf.py which is used to store configuration variables. conf.py is given below:
import os
step_number=100
I have a bash script runner.sh which tries to reach the variables from conf.py:
#! /bin/bash
#get step_number from conf file
step_number_=$(python ./conf.py step_number)
However, if I try to print the step_number_ with echo $step_number_, it returns empty value. Can you please help me to fix it?
$(command) is replaced with the standard output of the command. So the Python script needs to print the variable so you can substitute it this way.
import os
step_number = 100
print(step_number)
I am running a working project in my new position.
I believe virtual environment was created in it as I see:
$head bm3.py
#!/usr/bin/env /opt/bm3_venv/bin/python3
bm3_venv is the name of env created with requirements.txt (using virtualenv?)
$ ls -la /usr/bin/env
-rwxr-xr-x. 1 root root 28992 Jun 30 2016 /usr/bin/env
The bm3.py is presumably using python3 for the entire project, not only from the above first line in bm3.py but also seen in some other python scripts using print('asdf') which is a python3 grammar.
However I do see in the project there are python2 grammar print 'asdf', i.e.
/data/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/bin/../lib/impala-shell/impala_shell.py is used when executing bm3.py and impala-shell.py is written in python2 grammar.
That means, in the current working project, when running bm3.py, it is using python3 but in the same running python2 is also somehow used.
How could this happen?
BTW, where can I download the original copy of impala-shell.py for the parcel of CDH-5.12.0-1.cdh5.12.0.p0.29?
Thank you very much.
UPDATE:
In the existing environment the first line of bm3.py is:
/usr/bin/env /opt/al2_venv/bin/python3
This specifies using python3 in this bm3.py
In the impala-shell.py used in the existing environment the first line is:
/usr/bin/env /usr/bin/env python
This specifies using python2 in this impala-shell.py
Now, the question becomes how does /usr/bin/env work here?
If I ran it in the existing environment, I get a list of variables settings like below:
> XDG_SESSION_ID=224064 SELINUX_ROLE_REQUESTED= TERM=xterm
> SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=192.168.103.81 50182 22
> PATH=/usr/lib64/qt-3.3/bin:/home/xxxx/perl5/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/tableau/tabcmd/bin:/home/rxie/.local/bin:/home/rxie/bin
> PWD=/home/xxxx JAVA_HOME=/usr/java/latest LANG=en_US.UTF-8
> KDEDIRS=/usr SELINUX_LEVEL_REQUESTED= HISTCONTROL=ignoredups
> KRB5CCNAME=FILE:/tmp/krb5cc_1377008653_sw88z6 SHLVL=1 HOME=/home/xxxx
> PERL_LOCAL_LIB_ROOT=:/home/xxxx/perl5 LOGNAME=xxxx
> QTLIB=/usr/lib64/qt-3.3/lib SSH_CONNECTION=192.168.103.81 50182
> 192.168.101.231 22 LESSOPEN=||/usr/bin/lesspipe.sh %s XDG_RUNTIME_DIR=/run/user/1377008653
> QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
> PERL_MM_OPT=INSTALL_BASE=/home/rxie/perl5
> _=/usr/bin/env
What is this env for and how do I use it? Thanks.
I think you are running python 2 which you can verify using python -V in the Bash, now how could it be using python3 print() is by using from __future__ import print_function in the first line of your code, which from python 2.6+ ports/makes avialble the print function of python3 to python2.
I think I have the answer now:
I believe this is by design that python allows any python script (despite its python grammar) can specify the interpreter's version in the script's first line starts with #! like
#!/usr/bin/env /opt/bm3_venv/bin/python3 in bm3.py, meaning the entire script is written in python 3; at the meantime, when impala-shell.py is used during the job running, the first line in impala-shell.py specifies the python interpreter - which is python 2.6.6 - comes with the built-in python in Cloudera's CDH.
I’m trying to figure out how to use sys.argv in Python 3.6, but can’t figure out how to make it work using the Python interpreter (I’m not even 100% sure I’m actually using the interpreter, a bit confused around the terminology with interpreter, shell, terminal etc.)
Question 1: to access the Python interpreter, can I simply type $ python into the Terminal (I’m on a Mac)? If not, how do I access it?
Seems that when I go to find what I believe to be the interpreter in my files (I’ve downloaded Python via Anaconda), I find a program called “pythonw”, and starting this launches the Terminal, with what looks to be the Python interpreter already running. Is this the interpreter? The code chunk below is what is printed in a Terminal window when I run the "pythonw" program:
Last login: Tue Aug 7 18:26:37 on ttys001
Users-MacBook-Air:~ Username$ /anaconda3/bin/pythonw ; exit;
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:14:23)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Question 2: Let’s assume that I have the Python interpreter running for the sake of argument. Assume also that I have the following script/module saved as test.py.
import sys
print('Number of arguments:', len(sys.argv), 'arguments.')
print('Argument List:', str(sys.argv))
If I simply import this module in the command line of the interpreter, I get the printout:
Number of arguments: 1 arguments.
Argument List: ['']
But how do I actually supply the module with arguments in the command line?
I've been looking around on internet, all of them showing this way of doing it, but it does not work.
Question 3: can the sys.argv only be used when arguments are written in the command line of the interpreter, or is there a way to supply a module with arguments in Spyder for instance?
Thank you for taking the time to read through it all, would make me so happy if I could get an answer to this! Been struggling for days now without being able to grasp it.
The Python interpreter is just a piece of code which translates and runs Python code. You can interact with it in different ways. The most straightforward is probably to put some Python code in a file, and pass that as the first argument to python:
bash$ cat <<\: >./myscript.py
from sys import argv
print(len(argv), argv[1:])
:
bash$ # in real life you would use an editor instead to create this file
bash$ python ./myscript.py one two three
4 ['one', 'two', 'three']
If you don't want to put the script in a file, perhaps because you just need to check something quickly, you can also pass Python a -c command-line option where the option argument is a string containing your Python code, and any non-option arguments are exposed to that code in sys.argv as before:
bash$ python -c 'from sys import argv; print(len(argv), argv[1:])' more like this
4 ['more', 'like', 'this']
(Single quotes probably make the most sense with Bash. Some other shells use other conventions to wrap a piece of longer text as a single string; in particular, Windows works differently.)
In both of these cases, the Python interpreter was started with a program to execute; it interpreted and executed that Python program, and then it quit. If you want to talk to Python more directly in an interactive Read-Eval-Print-Loop (which is commonly abbreviated REPL) that's what happens when you type just python:
bash$ python
Python 3.5.1 (default, Dec 26 2015, 18:08:53)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+2
3
>>>
As you can see, anything you type at the >>> prompt gets read, evaluated, and printed, and Python loops back to the >>> to show that it's ready to do it again. (If you type something incomplete, the prompt changes to .... It will sometimes take a bit of puzzling to figure out what's missing - it could be indentation or a closing parenthesis to go with an opening parenthesis you typed on a previous line, for example.)
There is nothing per se which prevents you from assigning a value to sys.argv yourself:
>>> import sys
>>> sys.argv = ['ick', 'poo', 'ew']
At this point, you can import the script file you created above, and it will display the arguments after the first;
>>> import myscript
3, ['poo', 'ew']
You'll notice that the code ignores the first element of sys.argv which usually contains the name of the script itself (or -c if you used python -c '...').
... but the common way to talk to a module you import is to find its main function and call it with explicit parameters. So if you have a script otherscript.py and inspect its contents, it probably contains something like the following somewhere near the end:
def main():
import sys
return internal_something(*sys.argv[1:])
if __name__ == '__main__':
main()
and so you would probably instead simply
>>> import otherscript
>>> otherscript.internal_something('ick', 'poo')
Your very first script doesn't need to have this structure, but it's a common enough arrangement that you should get used to seeing it; and in fact, one of the reasons we do this is so that we can import code without having it start running immediately. The if __name__ == '__main__' condition specifically evaluates to False when you import the file which contains this code, so you can control how it behaves under import vs when you python myscript.py directly.
Returning to your question, let's still examine how to do this from a typical IDE.
An IDE usually shields you from these things, and simply allows you to edit a file and show what happens when the IDE runs the Python interpreter on the code in the file. Of course, behind the scenes, the IDE does something quite similar to python filename.py when you hit the Execute button (or however it presents this; a function key or menu item perhaps).
A way to simulate what we did above is to edit two files in the IDE. Given myscript.py from above, the second file could be called something like iderun.py and contain the same code we submitted to the REPL above.
import sys
sys.argv = ['easter egg!', 'ick', 'poo', 'ew']
import myscript
I'm wondering if it's possible to populate sys.argv (or some other structure) with command line arguments in a jupyter/ipython notebook, similar to how it's done through a python script.
For instance, if I were to run a python script as follows:
python test.py False
Then sys.argv would contain the argument False. But if I run a jupyter notebook in a similar manner:
jupyter notebook test.ipynb False
Then the command line argument gets lost. Is there any way to access this argument from within the notebook itself?
After a lot of looking around I found very cumbersome, custom libraries, but solved it with a few lines of code which I thought was pretty slick. I used nbconvert to end up with an html report as output that contains all graphics and markdown from the notebook, but accepts command line parameters just as always through a minimal python wrapper:
The python file test_args.py (which takes command line params as normal):
import sys,os
IPYNB_FILENAME = 'test_argv.ipynb'
CONFIG_FILENAME = '.config_ipynb'
def main(argv):
with open(CONFIG_FILENAME,'w') as f:
f.write(' '.join(argv))
os.system('jupyter nbconvert --execute {:s} --to html'.format(IPYNB_FILENAME))
return None
if __name__ == '__main__':
main(sys.argv)
The notebook contains:
import sys,os,argparse
from IPython.display import HTML
CONFIG_FILE = '.config_ipynb'
if os.path.isfile(CONFIG_FILE):
with open(CONFIG_FILE) as f:
sys.argv = f.read().split()
else:
sys.argv = ['test_args.py', 'input_file', '--int_param', '12']
parser = argparse.ArgumentParser()
parser.add_argument("input_file",help="Input image, directory, or npy.")
parser.add_argument("--int_param", type=int, default=4, help="an optional integer parameter.")
args = parser.parse_args()
p = args.int_param
print(args.input_file,p)
and I can run the python notebook with arguments parsed as usual:
python test_args.py my_input_file --int_param 12
I tend to paste the block with argparse calls into the python wrapper so that command line errors are caught by the python script and -h works properly.
There are two projects I've found that do what you ask for
Papermill, will add a cell to your notebook with arguments that you pass to it on the command line. So this is quite straightforward, you define your defaults in the first cell (the should have parameters tag)
nbparameterise it is a similar concept but you don't tag your cell with defaults, it has to be first.
Here is a good resource discussing the issue: https://github.com/jupyter/help/issues/218
If the goal is to run a notebook with configurable arguments passed from commandline, I think the easiest way is to use environment variables, like this:
NB_ARGS=some_args jupyter nbconvert --execute --to html --template full some_notebook.ipynb
Then in the notebook, you can import os and use os.environ['NB_ARGS']. The variable value can be some text that contains key-value pairs or json for example.
At the top of the Jupyter cell, put a line like:
%%python - --option1 value1 --option2 value2 --etc
In your example:
%%python - True
This will run your script like in a command line with the args provided.
Example:
%%python - --option1 value1 --option2 value2 --etc
import sys
if __name__ == '__main__':
print(sys.argv)
will output:
['-', '--option1', 'value1', '--option2', 'value2', '--etc']
Hope it helps.
I think this Gist may help you : https://gist.github.com/gbishop/acf40b86a9bca2d571fa
This is an attempt at a simple argument parser for mostly key=value pairs that can be used both on the command line and in IPython notebooks. It support query parameters in notebook URLs and a Run command for notebooks.
Using args = parser.parse_args(args=[]) would work.
or for testing, you can declare it as class format.
class Args:
data = './data/penn'
model = 'LSTM'
emsize = 200
nhid = 200
args=Args()
sys.argv yields a list, so I used
sys.argv.append('hello')
in a jupyter notebook, which allowed me to append extra members and pretend like I'm passing in arguments from the command line.
I tried out the answers listed above, and came up with a different solution.
My original code was
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True, help="path to input image")
ap.add_argument("-y", "--yolo", required=True, help="base path to YOLO directory")
ap.add_argument("-c", "--confidence", type=float, default=0.5, help="minimum probability to filter weak detections")
ap.add_argument("-t", "--threshold", type=float, default=0.3, help="threshold when applying non-maxima suppression")
args = vars(ap.parse_args())
I tried to make a Class as
Class Args():
image='photo.jpg'
yolo='yolo-coco'
confidence=0.5
threshold=0.3
args=Args()
but futher code snippets were producing an error.
So I printed args after vars(ap.parse_args()) and found that it was a dictionary.
So just create a dictionary for the original args:
args={"image": 'photo.jpg', "yolo": 'yolo-coco', "confidence": 0.5,"threshold": 0.3}
A workaround is to make the jupyter notebook read the arguments from a file.
From the command line, modify the file and run the notebook.
I assume that you just want to parse some arguments to the notebooks, but it's not necessary to use the command line.
If you want to parse commands like.
python script.py --a A --b B
You can use the following code in the notebook:
cmd = '--a A --b B'
args = args = parser.parse_args(cmd)
For parse_args, you can find more information here.
A simple and naïve solution is to put the following snippet at the first line of your program:
import sys
sys.argv = "your expected command line arguments here".split()
After executing this command, packages like argparse will work well.
So, you can just run your scripts in the jupyter lab server, without opening a terminal and typing your arguments.
Currently, I have a command that looks something like the following:
my_command = Popen([activate_this_python_virtualenv_file, \
"-m", "my_command", "-l", \
directory_where_ini_file_for_my_command_is + "/" + my_ini_file_name], \
stderr=subprocess.STDOUT, stdout=subprocess.PIPE, shell=False,
universal_newlines=False, cwd=directory_where_my_module_is)
I have figured out how to access and process the output, deal with subprocess.PIPE, and make subprocess do a few other neat tricks.
However, it seems odd to me that the standard Python documentation for subprocess doesn't mention a way to just get the actual command line as subprocess.Popen puts it together from arguments to the Popen constructor.
For example, perhaps my_command.get_args() or something like that?
Is it just that getting the command line run in Popen should be easy enough?
I can just put the arguments together on my own, without accessing the command subprocess runs with Popen, but if there's a better way, I'd like to know it.
It was added in Python 3.3. According to docs:
The following attributes are also available:
Popen.args The args argument as it was passed to Popen – a sequence of
program arguments or else a single string.
New in version 3.3.
So sample code would be:
my_args_list = [] # yourlist
p = subprocess.Popen(my_args_list)
assert p.args == my_args_list