From conda create requirements.txt for pip3 - python-3.x

I usually use conda to manage my environments, but now I am on a project that needs a little more horsepower than my laptop. So I am trying to use my university's workstations which have new Intel Xeons. But I don't have admin rights and the workstation does not have conda so I am forced to work with virtualenv and pip3.
How do I generate a requirements.txt from conda that will work with pip3 and venv?
conda list -e > requirements.txt
does not generate a compatible file:
= is not a valid operator. Did you mean == ?
The conda output is:
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-64
certifi=2016.2.28=py36_0
cycler=0.10.0=py36_0
freetype=2.5.5=2
icu=54.1=0
libpng=1.6.30=1
matplotlib=2.0.2=np113py36_0
mkl=2017.0.3=0
numpy=1.13.1=py36_0
openssl=1.0.2l=0
pip=9.0.1=py36_1
pyparsing=2.2.0=py36_0
pyqt=5.6.0=py36_2
python=3.6.2=0
python-dateutil=2.6.1=py36_0
pytz=2017.2=py36_0
qt=5.6.2=2
readline=6.2=2
scikit-learn=0.19.0=np113py36_0
scipy=0.19.1=np113py36_0
setuptools=36.4.0=py36_1
sip=4.18=py36_0
six=1.10.0=py36_0
sqlite=3.13.0=0
tk=8.5.18=0
wheel=0.29.0=py36_0
xz=5.2.3=0
zlib=1.2.11=0
I thought I would just manually change all = to == but the there are two = in the conda output. Which one to change? Surely there is an easier way?
EDIT: pip freeze > requirements.txt gives:
certifi==2016.2.28
cycler==0.10.0
matplotlib==2.0.2
matplotlib-venn==0.11.5
numpy==1.13.1
pyparsing==2.2.0
python-dateutil==2.6.1
pytz==2017.2
scikit-learn==0.19.0
scipy==0.19.1
six==1.10.0

As the comment at the top indicates, the output of
conda list -e > requirements.txt
can be used to create a conda virtual environment with
conda create --name <env> --file requirements.txt
but this output isn't in the right format for pip.
If you want a file which you can use to create a pip virtual environment (i.e. a requirements.txt in the right format)
you can install pip within the conda environment, then use pip to create requirements.txt.
conda activate <env>
conda install pip
pip freeze > requirements.txt
Then use the resulting requirements.txt to create a pip virtual environment:
python3 -m venv env
source env/bin/activate
pip install -r requirements.txt
When I tested this, the packages weren't identical across the outputs (pip included fewer packages) but it was sufficient to set up a functional environment.
For those getting odd path references in requirements.txt, use:
pip list --format=freeze > requirements.txt

Problem
In a conda environment with simply calling
pip freeze
Output:
ipykernel # file:///C:/ci/ipykernel_1607454116140/work/dist/ipykernel-5.3.4-py3-none-any.whl
ipython # file:///D:/bld/ipython_1612487184680/work
...
Wanted format:
ipykernel==5.3.4
ipython==7.20.0
...
Solution
In an activated conda environment I had to use
pip list --format=freeze
to get the correct format for generating a requirements file for people who prefer to use pip with virtual environments.
Save to file:
pip list --format=freeze > requirements.txt

Following the discussion, I'd like to mention that you can actually see some separation of pip and conda roles.
pip is a standard package manager, it does one thing and does it well. requirements.txt can be generated in one environment and installed by pip in a new environment.
Now there is conda output: you rightfully capture their comment which says 'we generated this list of libraries to work with conda'. Note that python itself is in the conda list and (properly) not in requirements.txt. conda replicates own installation, that is why its list of libraries is longer, and has python itself.
pip produces a list of packages that were installed on top of standard library to make the package you wrote work. Hope it helps to distinguish between the two.
Also pipenv is a newer tool, that can do both virtual environment and package management for you.

Just in case someone is looking to generate requirements.txt from an existing project in conda, use following
Go to your project environment conda activate <env_name>
conda list gives you list of packages used for the environment
conda list -e > requirements.txt save all the info about packages to your folder
conda env export > <env_name>.yml
pip freeze

As mentioned in the comments above, the correct full answer is:
pip list --format=freeze > requirements.txt

activate the conda env
conda activate flask-test
get the path of the conda env and copy it
conda list
append the copied path with lib\site-packages and use it in pip with --path option
pip freeze --path C:\Users\username\Miniconda3\envs\flask-test\lib\site-packages > requirements.txt
on Linux the path is like /home/username/miniconda3/envs/flask-app/lib/python3.8/site-packages/

I made some simple python script to convert conda requirements.txt for pip3.
Just copy below code.
f = open(r"requirements.txt", "r").read()
f1 = f.split("\n")
for line in f1:
if "=" in line:
a = line.split("=")[0:-1]
print(a[0]+"==" + a[1])
then just copy the printed output;
Sample output:
_libgcc_mutex==0.1
_openmp_mutex==4.5
blas==1.0
brotli==1.0.9
bzip2==1.0.8

Related

How to install packages on a conda environment with a specific python version

I need to use opencv and it needed python version older than 3.9. So I created a conda environment with python 3.7.9 and tried to use
pip install opencv-python
It installed but when I tried to import it gave an error:
ModuleNotFoundError: No module named 'cv2'
After I installed it on the anaconda environments tab I could import. I need to install pylibdmtx, too, but again I cannot import it after installing.
I see the version of python is 3.7.9 on the environment. But when I use pip it is using this:
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (0.1.9)
I do not know what I am missing here. What should I do?
Recommended Solution
Prefer specifying what you want in an environment up front, e.g.,
opencv.yaml
name: py37_opencv
channels:
- conda-forge
- defaults
dependencies:
- python=3.7
- opencv
Create Environment
conda env create -n py37_opencv -f opencv.yaml
Troubleshooting Issue
Otherwise, if you want get the version you have working, try ensuring that pip is actually installed in the environment of interest:
conda activate your_env
conda install pip
# then use pip …

My requirments.txt file doesn't have list of all library I have used in project

I have create a new conda env. Then after complete the project, I have generated pickle file. Now when I am trying to generated requirements.txt it is not giving me list of all library or packages
(carprediction) F:\Car Dekho Dataset>pip3 freeze > requirements.txt
(carprediction) F:\Car Dekho Dataset>pip freeze > requirements.txt
I have tried both the command still it's not working
Output in file is as follow
certifi==2020.12.5
wincertstore==0.2
Am I doing something wrong ?
Blockquote
wincertstore==0.2
Check your installed packages using
pip freeze
see if all the packages are showing properly
On the other hand you can try this:
conda list --export > requirements.txt
Works nearly similarly to pip freeze > requirements.txt
There's quite some difference between conda and pip. You can read about it here
You can read more about pip and conda in this article
Update 1:
the conda command to install numpy is
conda install -c anaconda numpy
activate your conda environment and install numpy and finally run pip freeze to check packages or conda list --export > requirements.txt to list all the packages in requirements.txt

Setting Up a Python Virtual Environment with Hydrogen in Atom

I'm in the middle of switching from VS Code to Atom and I'm trying to set up a virtual environment for my python project.
It was pretty easy to do in VS Code, I'd run the following script and it would automagically start using the new env (with all the required packages) when I'd run the script:
python3 -m venv my_env
source my_env/bin/activate
pip3 install -r requirements.txt
Now I'm trying to set up Hydrogen to work the same way. When I run lines of code inline with Hydrogen, I want them to be run in a virtual environment that has the imported modules I need from a requirements.txt file.
I was able to install the python3 kernel with the following commands:
python3 -m venv my_environment_name # create a virtual environment
source my_environment_name/bin/activate # activate the virtual environment
python -m pip install ipykernel # install the python kernel (ipykernel) into the virtual environment
python -m ipykernel install
And Atom is able to see it:
Screenshot
However, I'm still puzzled as how to install my dependencies into the kernel. And if I do install my dependencies there, I don't want my next python projects to have all those modules in there. I'd love to have the fresh-slate that virtual environments promise.
Any help here would be appreciated. Has anyone had experience setting up a virtual environment that can be used by the Hydrogen package?
Ok, after some more experimentation, I was able to connect to a kernel that I had installed my requirements.txt into.
Here are the steps I took:
python3 -m venv env
source env/bin/activate
# make sure requirements.txt has ipykernel in it
pip3 install -r requirements.txt
python3 -m ipykernel install --user --name=env
Then in Atom, press cmd-shift-p and find Hydrogen: Update Kernels.
Or manually Packages->Hydrogen->Update Kernels
After, I was able to use the kernel by doing cmd-shift-p again and selecting Hydrogen: Start Local Kernel and selecting env.
When I would run import statements via Hydrogen (selecting them and pressing cmd-enter), they would now know what to import! Horray!
Windows Version.
This is the windows version of the excellent answer by Tim Estes. Without the use of requirements.txt in case some folks such as myself would like to install ipykernel separately. (Thanks so much Tim, it saved me a ton of time)
python -m venv env
env/bin/activate
pip install ipykernel
python -m ipykernel install --user --name=env
Similar to Tim's answer, go ahead and update kernels at Atom.
Packages->Hydrogen->Update Kernels

Python 3 - How do you re-create your Pipfile?

I am a major Python noob, and I made the mistake of manually deleting my Pipfile and Pipfile.lock, thinking that they would eventually regenerate.
How can I re-create these files?
There is a simple fix to this:
First you need to install pipenv if you haven't already:
pip install pipenv
Then change directory to the folder containing your Python project and initiate Pipenv (replace my_project with the name of your project folder):
cd my_project
pipenv install
This will create two new files, Pipfile and Pipfile.lock, in your project directory, and a new virtual environment for your project if it doesn’t exist already.
For regular pip:
pip freeze > requirements.txt
For pipenv:
pipenv run pip freeze > requirements.txt
And to install:
pip install requirements.txt
Situation: you have deleted Pipfile and Pipfile.lock, but your pipenv environment is still there. By this I mean pipenv shell puts you in your environment with a python that has all your packages right there.
# Create a new Pipfile
pipenv install
# Get a list of your packages + version from the would-be "reqiurements.txt", then install them one by one.
pipenv run pip freeze|sed 's/^/pipenv install /'|source /dev/stdin
Note that this is not a perfect solution, as it will specify the versions in the production target environment as you would get from pip freeze i.e. requirements.txt. However this has saved me more than once, so I hope it helps you.

Can I create a virtualenv after making the mistake of installing all project packages globally?

So I am a python newb like first project ever newb. I jumped right in disregarding virtualenv and installed everything globally. Now I need to be able to share my project with other team members.
Can I create a virtualenv after making the mistake of installing all project packages globally?
I am using python 3. I've read these links:
pip installing in global site-packages instead of virtualenv
How to import a globally installed package to virtualenv folder
But I don't think thats what Im looking for. I want to go the requirements.txt route I think.
Any guidance or suggestions?
Yes, you can create a virtual env.
You can create a requirements.txt file for the packages you installed globally.
pip3 freeze > requirements.txt
and then you can use this requirements file for installing all the packages in the virtual env which will be isolated from your global environment.
First you need to install virtualenv:
pip3 install virtualenv
Create a new virtual env using the following command:
virtualenv -p python3 envname
You can activate the virtualenv by:
source /path/to/new/virtual/environment/bin/activate
To deactivate the environment and return to your local environment just run:
deactivate
Install the requirements from the file.
cat requirements.txt | xargs -n 1 pip3 install
This should install all your packages in the virtual environment.
To check which python you are using use which python command and to check installed packages use pip3 list
I hope this will clear your doubt.

Resources