I have the following small script in a file named fgh.py which I have been attempting to schedule as a webjob
import pandas as pd
df=pd.DataFrame({'a':[1,2,2],'b':[5,6,9]})
df['x']=df.a.sub(df.b)
print(df)
Using #Peter Pan post. I have created a virtual environment, done a pip install pandas. From the virtual environment, the script runs and executes as required.It however does not execute when loaded in Azure Webjobs. I suspect issues arise from the interface between the run,bat file and the Azure python console but have limited understanding of Azure to resolve the issue
In kudus, I have used this post to install python.
Running where python in cmd command in https://myapp.scm.azurewebsites.net/DebugConsole I get;
Additionally from https://arcgistrial.scm.azurewebsites.net/DebugConsole I get the following when I run cmd command python -V
In my run.bat file, I have tried to use either of the directories above without success.
Whether I make my run.bat file D:\home\python364x64\python.exe fgh.py or D:python364x64\python.exe fgh.py I get the following error;
I have gone a head and installed pandas and checked if successful by trying to install numpy
All this has not helped. I have been on this for a couple of days and it has to work somehow. Any help?
(Things are not quite straightforward in old Webjobs to run python task with dependencies. It has been quite some time, the world has moved on to Azure Function :))
However, since you still need to stick to Webjobs, below are the steps I followed which worked. I am using a batch file (.cmd) to run the python script due to the pre-requisites.
By default webjob supports python 2.7 at this moment. So, add python3 from 'extension' in your web app, In this case it was 3.6.4 x64 for me. This will add in path D:\home\python364x64\. How did I know? Kudus console :)
Create a requirements.txt file which contains pandas and numpy (note I had to explicitly add numpy version 1.19.3 due to an issue with latest 1.19.4 in Windows host at the time of this writing). Basically I used your fgh.py which depends on pandas which in turn depends on numpy.
pandas==1.1.4
numpy==1.19.3
Create a run.cmd file having the following content. Note 1st line is not needed. I was just checking python version.
D:\home\python364x64\python --version
D:\home\python364x64\python -m pip install --user --upgrade pip
D:\home\python364x64\python -m pip install --user certifi
D:\home\python364x64\python -m pip install --user virtualenv
D:\home\python364x64\python -m virtualenv .venv
.venv\Scripts\pip install -r requirements.txt
.venv\Scripts\python fgh.py
Zip fgh.py, run.bat and the requirements.txt files into a single zip. Below is the content of my zip.
Upload the zip for the webjob.
Run the job :)
Ignore the error "ModuleNotFoundError: No module named 'certifi'", not needed.
The key to solving the problem is that you need to create your venv environment on azure.
Step 1. Run successfully in local.
Step 2. Compress your webjob file.
Step 3. Upload webjob zip file.
Because the test environment has python1 before, I will create a webjob of python2 later.
Step 4. Log in kudu.
① cd ..
② find Python34, click it.
③ python -m venv D:\home\site\wwwroot\App_Data\jobs\continuous\python2\myenv
④ Find myenv folder.
⑤ active myenv, input .\activate.bat.
D:\home\site\wwwroot\App_Data\jobs\continuous\python2\myenv\Scripts>.\activate.bat
⑥ Back to python2 folder, and input pip install pandas.
⑦ Then input python aa.py.
Related
I am new to Python and Azure scripting...and Mac (somewhat). I am trying to create a simple python script so I can create a set of Azure resources I use in a project. I followed these directions to set up vscode and I loosely followed these directions to get started with the Azure commands. Loosely, so I could ensure I knew what was happening. I did not pull down the repo. I am getting an Import "azure.common.credentials" could not be resolved error on the following line:
from azure.common.credentials import ServicePrincipalCredentials
In setting up the vscode/python environment, I selected the 3.7.3 interpreter. I did not pull down the github repo or run the pip install -r requirements.txt, again just to make sure I could do this from scratch and understand how these things work together.
When I run pip install azure from the command line it seems to default to the python 2.7 installation on the Mac, which is deprecated but I think required on the Mac.
Do I need to do the pip install azure for 3.7.3? If so how do I do that?
I have created a virtual environment for my Python project and have installed my requirement.txt in it and every time my Python code ex executes it first activate my virtual environment and then run the .py code which is what I want.
I have an issue here - I want to do everything programmatically.
Below are the high level steps i'm looking for :
1)pipreqs requirements.txt using py code
2)check if .txt file has changed, create a flag(1,0)
3)check if virtualenv is created
2.1 - if not created, create and install 1 and run py code
2.2 - if created - check if flag=1 then re-install 1 and run py code
if flag=0 , just run py code
I'm struggling hard to achieve this. Could somebody help.
maybe
pip install -r requirements.txt
is a good starting point (if your paint point is to manually install all the required libraries). It installs all the libraries listed in requirements.txt.
If this is not enough putting this command in a bash / cmd file might be an option.
I need to provide my version of Python and packages for a project.
How can I do that?
I tried:
sudo apt-get install python3-venv
virtualenv my-env -p python3
source tutorial-env/bin/activate
This should show installed packages, but it shows:
pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.1)
pkg-resources (0.0.0)
setuptools (39.0.1)
This is also suspicious:
(tutorial-env) linux#LINUXMINT:~$ pip freeze > requirements.txt
(tutorial-env) linux#LINUXMINT:~$ cat requirements.txt
pkg-resources==0.0.0
(tutorial-env) linux#LINUXMINT:~$ python -m pip install -r requirements.txt
Requirement already satisfied: pkg-resources==0.0.0 in ./tutorial-env/lib/python3.6/site-packages (from -r requirements.txt (line 1))
And I cannot find requirements.txt in my directory.
Pass the version of python when creating your virtualenv like this:
virtualenv my-env -p python3
I think you should not use any external module for creating a virtual environment.
You can create a Virtual Environment by using venv attribute of Python in Command Line.
What is the syntax?
The syntax is pretty simple.
C:\>python -m venv path\to\where\you\want\to\create\it
It can be easily done, and you can get a reference from the example below:
C:\>python -m venv "C:\Users\Bhavyadeep\Desktop\Discord Bots\Bot" 1\Bot-1-env
Here the name Bot-1-env is the name of the folder which will be created on execution of the command, and it doesn't have to exist.
What if I am using an IDE (like VS Code), then how will I create a virtual environment?
Creating a Virtual Environment in an IDE is much easier than creating it using CMD. In CMD you need to specify the full path of the directory where the Environment has to be created whereas in an IDE you can create one using its own terminal and also there won't be any need of adding and full path to the directory.
Syntax for IDEs with their Terminals is:
C:\>python -m venv My-Env
This would simply create a Virtual Environment in the folder of the project you are working on in the IDE. If you still want to create it using full path you can do the same as above in the Terminal of IDE.
Example with Images and Code in One Step:
My target directory would be the Desktop for now to explain.
Write and Execute the command in Command Line.
I entered the line in the image and pressed Enter.
Here the name of the Folder would be Example-Venv and it doesn't exist. This command created a folder with that name and created that a Virtual Environment.
This command created a folder, and it can be seen in the picture below.
Now you can use it anywhere you want by simply making this folder as the Interpreter.
How to set interpreter?
The following links would explain:
Pycharm: https://www.jetbrains.com/help/pycharm/configuring-python-interpreter.html
VS Code: https://code.visualstudio.com/docs/python/environments#:~:text=To%20select%20a%20specific%20environment,or%20library%20versions%20as%20needed.
Spyder: https://www.spyder-ide.org/blog/release-spyder-330/#:~:text=Just%20set%20the%20path%20under,start%20in%20the%20selected%20environment.
Sublime: https://medium.com/#hariyanto.tan95/set-up-sublime-text-3-to-use-python-3-c845b742c720
I was glad to help! If you still get any problem, please feel free to ask in the comments and I would gladly help you! :)
Thank You! :)
I'm a bit new to Python and all its "deployment" related tools: pip3,setuptools, virtualenv wheel etc, so I hope my question will make sense...anyways it is like so:
I have a Python3 project which is "managed" with virtual environment using virtualenv where all the projects' dependencies are "listed" within the project's setup.py file. The contents of the setup.py files are as follows:
setup(name="MyProjectName",
version="0.1",
description="Some description",
url="someURL",
author="My Name",
author_email="someemail",
license="MIT",
packages=find_packages(),
include_package_data=True,
install_requires=["robotframework", "paramiko"])
As you can see, the only 3rd party packages the project uses (explicitly) are robotframework & paramiko.
Now when I'm deploying the project, I do the following actions (in that order):
Create a virtual environment with the command:
virtualenv -p python3 virtualEnvFolderName
Switching "into" the virtual environment like so (I'm deploying it on a Linux machine):
source virtualEnvFolderName/bin/activate
Running the setup.py script with the install argument to "automatically" install all the project's dependencies with the following command:
python3 setup.py install
--> Up until couple of days ago, all the 3rd party packages (and their "dependencies sub-packages") listed in the setup.py file where downloaded (and then installed) using their whl file, i.e. - for example:The output for the paramiko package installation would have been:
Reading https://pypi.org/simple/paramiko/
Downloading https://files.pythonhosted.org/packages/4b/80/74dace9e48b0ef923633dfb5e48798f58a168e4734bca8ecfaf839ba051a/paramiko-2.6.0-py2.py3-none-any.whl#sha256=99f0179bdc176281d21961a003ffdb2ec369daac1a1007241f53374e376576cf
Best match: paramiko 2.6.0
Processing paramiko-2.6.0-py2.py3-none-any.whl
Installing paramiko-2.6.0-py2.py3-none-any.whl to
--> This way, the installation was very quick (~1-3 seconds per package).
Today, when I performed the same procedure, on the same machine (I'm quiet sure I did not change any settings on my Ubuntu 16.04 machine), for each package the setup.py tried to install, it installed "via" the tar.gz file (i.e. sources ?) and NOT using the whl file --> which takes MUCH longer since for some of the packages it actually builds (complies) all the "underlying C libraries". This "change" makes my "installation procedure" execution time increase from ~20 seconds to ~4 minutes.
My questions are:
a) How can I resolve this situation - preferably without changing the deployment procedure, i.e. - still perform the 3 steps mentioned above, taking into account that perhaps one or more of the commands will be slightly modified (the creation of the virtual environment and/or some additional argument in required to the setup.py ? ).
b) If I have no other option, then using a pip3 install -r requirement.txt ... "procedure" will also be good, if it also will use whl file(s) whenever applicable.
c) If I will need to "switch" my virtual environment "generator" to venv it is OK (and actually preferred, in case it will deploy the project in the "same" duration).
NOTES:
I tested it both on Ubuntu 16.04 and Ubuntu 18.04 machines with Python 3.5 and Python 3.6 respectively.
Thanks !!
It seems like there is no wheel compatible with your environment (operating system, Python interpreter version, CPU bitness) for the current version of the project PyNaCl. If you have a recent version of pip, the command path/to/venv/bin/python -m pip debug --verbose should list the tags that are compatible with your environment, so that you can compare with the list of wheels available on PyPI.
I'm using Windows 7 and am up to date on patches. I was using Python 3.5.2 and wanted to upgrade to 3.6, so I went to the Python site, downloaded 3.6.4.exe for Windows, and ran it. The Python seems to work fine and is 3.6.4, but trying to import any of the packages I was using (pandas, numpy, tensorflow, etc) now gives me ModuleNotFoundError: No module named <whichever module it was>. Also, pip list now shows only pip and setuptools.
It appears to be a known issue, for example this from nearly a year ago, which suggests that I should "uninstall the python bindings and install again", but I can't figure out what that means. Fortunately I can still access 3.5.2 by using py -3.5, and then my imports work. Can anyone tell me how to fix this for 3.6 without having to reinstall all my modules manually?
I was thinking possibly I should have upgraded through pip; it seems like that's possible but maybe a bad idea for some reason? On further investigation it looks like what I should have done was save my requirements with pip freeze > requirements.txt, and then after installing the new Python restore them with pip install -r requirements.txt. Is this right?
Hard to say if you have an install problem, but this is what I would try if I were in your place.
Create a virtual environment as per; docs
c:>c:\Python36\python -m venv c:\path\to\myenv
Activate your virtual environment
C:> \Scripts\activate.bat
Run your application from within your activated environment. Each time you get an import error, do a pip-install from within the active environment. (For your own modules, you may need to modify PYTHONPATH in 'activate.bat')
Once you have your application running again, do your pip freeze > requirements.txt, and keep that with your project.
Each time you run your application, do so from within the activated virtual environment.
This will give you a clean requirements.txt that doesn't include a bunch of junk from other projects. Then, when you go to 3.7, just create the virtualenv and run your requirements.txt and wala!
I suspect your issue is simply not running against the correct interpreter, running from within a virtual environment should at least rule it out.