How to specify python version in a conda virtual env - python-3.x

I am running a working project in my new position.
I believe virtual environment was created in it as I see:
$head bm3.py
#!/usr/bin/env /opt/bm3_venv/bin/python3
bm3_venv is the name of env created with requirements.txt (using virtualenv?)
$ ls -la /usr/bin/env
-rwxr-xr-x. 1 root root 28992 Jun 30 2016 /usr/bin/env
The bm3.py is presumably using python3 for the entire project, not only from the above first line in bm3.py but also seen in some other python scripts using print('asdf') which is a python3 grammar.
However I do see in the project there are python2 grammar print 'asdf', i.e.
/data/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/bin/../lib/impala-shell/impala_shell.py is used when executing bm3.py and impala-shell.py is written in python2 grammar.
That means, in the current working project, when running bm3.py, it is using python3 but in the same running python2 is also somehow used.
How could this happen?
BTW, where can I download the original copy of impala-shell.py for the parcel of CDH-5.12.0-1.cdh5.12.0.p0.29?
Thank you very much.
UPDATE:
In the existing environment the first line of bm3.py is:
/usr/bin/env /opt/al2_venv/bin/python3
This specifies using python3 in this bm3.py
In the impala-shell.py used in the existing environment the first line is:
/usr/bin/env /usr/bin/env python
This specifies using python2 in this impala-shell.py
Now, the question becomes how does /usr/bin/env work here?
If I ran it in the existing environment, I get a list of variables settings like below:
> XDG_SESSION_ID=224064 SELINUX_ROLE_REQUESTED= TERM=xterm
> SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=192.168.103.81 50182 22
> PATH=/usr/lib64/qt-3.3/bin:/home/xxxx/perl5/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/tableau/tabcmd/bin:/home/rxie/.local/bin:/home/rxie/bin
> PWD=/home/xxxx JAVA_HOME=/usr/java/latest LANG=en_US.UTF-8
> KDEDIRS=/usr SELINUX_LEVEL_REQUESTED= HISTCONTROL=ignoredups
> KRB5CCNAME=FILE:/tmp/krb5cc_1377008653_sw88z6 SHLVL=1 HOME=/home/xxxx
> PERL_LOCAL_LIB_ROOT=:/home/xxxx/perl5 LOGNAME=xxxx
> QTLIB=/usr/lib64/qt-3.3/lib SSH_CONNECTION=192.168.103.81 50182
> 192.168.101.231 22 LESSOPEN=||/usr/bin/lesspipe.sh %s XDG_RUNTIME_DIR=/run/user/1377008653
> QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
> PERL_MM_OPT=INSTALL_BASE=/home/rxie/perl5
> _=/usr/bin/env
What is this env for and how do I use it? Thanks.

I think you are running python 2 which you can verify using python -V in the Bash, now how could it be using python3 print() is by using from __future__ import print_function in the first line of your code, which from python 2.6+ ports/makes avialble the print function of python3 to python2.

I think I have the answer now:
I believe this is by design that python allows any python script (despite its python grammar) can specify the interpreter's version in the script's first line starts with #! like
#!/usr/bin/env /opt/bm3_venv/bin/python3 in bm3.py, meaning the entire script is written in python 3; at the meantime, when impala-shell.py is used during the job running, the first line in impala-shell.py specifies the python interpreter - which is python 2.6.6 - comes with the built-in python in Cloudera's CDH.

Related

how to get a variable of a python file from bash script

I have a python file, conf.py which is used to store configuration variables. conf.py is given below:
import os
step_number=100
I have a bash script runner.sh which tries to reach the variables from conf.py:
#! /bin/bash
#get step_number from conf file
step_number_=$(python ./conf.py step_number)
However, if I try to print the step_number_ with echo $step_number_, it returns empty value. Can you please help me to fix it?
$(command) is replaced with the standard output of the command. So the Python script needs to print the variable so you can substitute it this way.
import os
step_number = 100
print(step_number)

How to run the python script file which contains some arguments to pass?

I have a module which contains the python code and I execute it using the following command:
python script.py \
--eps 12 \
--minpts \
--train \
--predict \
--lower_case \
--input_file data.csv \
--dev_file devdata.csv \
--output_dir /output/
All I want to do is to execute the above command through a python function. Is there any way of doing it?
I don't know why everyone is having such difficulty with this question, it's perfectly clear, unfortunately it's also difficult to do what you want because python uses implicit data types, and that's uncommon. As a result all command line arguments are passed to python as strings.
I'd check this out for the details:
https://www.tutorialspoint.com/python/python_command_line_arguments.htm
but the tldr is to have inside your python script:
import sys
eps = int(sys.argv[sys.argv.index('eps')+1])
minpts = True if '-minputs' in sys.argv else False
…
Obviously this isn't ideal, or pretty but it is quick and easy.
Alternatively you can use the argparser library:
https://docs.python.org/3/library/argparse.html
For a more robust and user friendly solution. Hope this helps
A.
Edit:
I was missing the ' around eps
Command-Line Arguments
import sys
print 'version is', sys.version
The first line imports a library called sys, which is short for "system". It defines values such as sys.version, which describes which version of Python we are running.
This command tells the python interpreter installed in your machine to run program sys-version.py from the current directory.
Here's another script that does something more interesting:import sys
print 'sys.argv is', sys.argv

Using sys.argv in Python 3 with Python interpreter

I’m trying to figure out how to use sys.argv in Python 3.6, but can’t figure out how to make it work using the Python interpreter (I’m not even 100% sure I’m actually using the interpreter, a bit confused around the terminology with interpreter, shell, terminal etc.)
Question 1: to access the Python interpreter, can I simply type $ python into the Terminal (I’m on a Mac)? If not, how do I access it?
Seems that when I go to find what I believe to be the interpreter in my files (I’ve downloaded Python via Anaconda), I find a program called “pythonw”, and starting this launches the Terminal, with what looks to be the Python interpreter already running. Is this the interpreter? The code chunk below is what is printed in a Terminal window when I run the "pythonw" program:
Last login: Tue Aug 7 18:26:37 on ttys001
Users-MacBook-Air:~ Username$ /anaconda3/bin/pythonw ; exit;
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:14:23)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Question 2: Let’s assume that I have the Python interpreter running for the sake of argument. Assume also that I have the following script/module saved as test.py.
import sys
print('Number of arguments:', len(sys.argv), 'arguments.')
print('Argument List:', str(sys.argv))
If I simply import this module in the command line of the interpreter, I get the printout:
Number of arguments: 1 arguments.
Argument List: ['']
But how do I actually supply the module with arguments in the command line?
I've been looking around on internet, all of them showing this way of doing it, but it does not work.
Question 3: can the sys.argv only be used when arguments are written in the command line of the interpreter, or is there a way to supply a module with arguments in Spyder for instance?
Thank you for taking the time to read through it all, would make me so happy if I could get an answer to this! Been struggling for days now without being able to grasp it.
The Python interpreter is just a piece of code which translates and runs Python code. You can interact with it in different ways. The most straightforward is probably to put some Python code in a file, and pass that as the first argument to python:
bash$ cat <<\: >./myscript.py
from sys import argv
print(len(argv), argv[1:])
:
bash$ # in real life you would use an editor instead to create this file
bash$ python ./myscript.py one two three
4 ['one', 'two', 'three']
If you don't want to put the script in a file, perhaps because you just need to check something quickly, you can also pass Python a -c command-line option where the option argument is a string containing your Python code, and any non-option arguments are exposed to that code in sys.argv as before:
bash$ python -c 'from sys import argv; print(len(argv), argv[1:])' more like this
4 ['more', 'like', 'this']
(Single quotes probably make the most sense with Bash. Some other shells use other conventions to wrap a piece of longer text as a single string; in particular, Windows works differently.)
In both of these cases, the Python interpreter was started with a program to execute; it interpreted and executed that Python program, and then it quit. If you want to talk to Python more directly in an interactive Read-Eval-Print-Loop (which is commonly abbreviated REPL) that's what happens when you type just python:
bash$ python
Python 3.5.1 (default, Dec 26 2015, 18:08:53)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+2
3
>>>
As you can see, anything you type at the >>> prompt gets read, evaluated, and printed, and Python loops back to the >>> to show that it's ready to do it again. (If you type something incomplete, the prompt changes to .... It will sometimes take a bit of puzzling to figure out what's missing - it could be indentation or a closing parenthesis to go with an opening parenthesis you typed on a previous line, for example.)
There is nothing per se which prevents you from assigning a value to sys.argv yourself:
>>> import sys
>>> sys.argv = ['ick', 'poo', 'ew']
At this point, you can import the script file you created above, and it will display the arguments after the first;
>>> import myscript
3, ['poo', 'ew']
You'll notice that the code ignores the first element of sys.argv which usually contains the name of the script itself (or -c if you used python -c '...').
... but the common way to talk to a module you import is to find its main function and call it with explicit parameters. So if you have a script otherscript.py and inspect its contents, it probably contains something like the following somewhere near the end:
def main():
import sys
return internal_something(*sys.argv[1:])
if __name__ == '__main__':
main()
and so you would probably instead simply
>>> import otherscript
>>> otherscript.internal_something('ick', 'poo')
Your very first script doesn't need to have this structure, but it's a common enough arrangement that you should get used to seeing it; and in fact, one of the reasons we do this is so that we can import code without having it start running immediately. The if __name__ == '__main__' condition specifically evaluates to False when you import the file which contains this code, so you can control how it behaves under import vs when you python myscript.py directly.
Returning to your question, let's still examine how to do this from a typical IDE.
An IDE usually shields you from these things, and simply allows you to edit a file and show what happens when the IDE runs the Python interpreter on the code in the file. Of course, behind the scenes, the IDE does something quite similar to python filename.py when you hit the Execute button (or however it presents this; a function key or menu item perhaps).
A way to simulate what we did above is to edit two files in the IDE. Given myscript.py from above, the second file could be called something like iderun.py and contain the same code we submitted to the REPL above.
import sys
sys.argv = ['easter egg!', 'ick', 'poo', 'ew']
import myscript

How to set/define/use sys.argv

I'm fairly new to Python, so please bear with me.
Currently, I'm using Python 3.5 in an Anaconda environment on Pycharm, and I am trying to understand how to set/define/use sys.argv so that I can automate several processes before uploading my changes onto github.
For example:
python function/function.py input_folder/input.txt output_folder/output.txt
This means that function.py will take input.txt from input_folder, apply whatever script written in function.py, and store the results into output.txt in folder output_folder.
However, when I type this into terminal, I get the following error:
python: can't open file 'function/function.py': [Errno 2] No such file or directory
Then, typing sys.argv into Python console, I receive the following:
['C:\\Program Files (x86)\\JetBrains\\PyCharm 2016.2\\helpers\\pydev\\pydevconsole.py',
'53465',
'53466']
My guess is that if I were to set sys.argv[0:1] correctly, then I should be able to apply function.py to input.txt and store the results into output.txt.
I've already tried to define these directories, but they wouldn't work. Any help would be awesome!
your issue is that python does not know where the function directory exists. If you are trying to run a script from a sub directory like so
function
|_function.py
|
input_folder
|_input.txt
|
|output_folder
|_output.txt
you must tell python that the function folder is local, so
python ./function/function.py ./input_folder/input.txt ./output_folder/output.txt
or
python $PWD/function/function.py $PWD/input_folder/input.txt $PWD/output_folder/output.txt
$PWD is a bash variable that gives the current directory

How to execute an external execution file with parameters in Python 3?

I want to execute an exe file using Python 3.4.
That is,
C:/crf_test.exe -m input.txt output.txt
When I executed this at the command line, the result was:
Go SEARCH
to O
...
But, when I executed this in Python like this:
import os
os.startfile('crf_test.exe -m model.txt test.txt')
Nothing happened (I mean appeared in the result window.)
Using os.popen() you can execute and read commands:
cmd = os.popen(r'crf_test.exe -m model.txt test.txt')
result = cmd.read()

Resources