I have tried using 0.4.2 s3fs pip module, moving greater than 5GB files are failing with write fail errors.
There is an issue with 0.4.2 version of s3fs pip module, which got fixed in master. When is next release planned so that i can use this feature.
Can i use master build directly ??
You can try installing it from source. Refer to the commands below:
git clone git#github.com:dask/s3fs
cd s3fs
git checkout master
python setup.py install
Related
It's extremely useful for the development workflow to be able to build and install the latest version of a package into a local environment. You can then interactively validate and debug by importing this latest version into a Python shell or a Jupyter notebook. The problem is I've recently adopted Poetry and cannot figure how to do this now. So...
How do I install the latest version of my package from the current working directory into my local environment using Poetry?
Moving on from setuptools
Back in the day, I used to always use setuptools and it worked great. I'd put a setup.py file in the root of my repository, create a virtual environment (let's say using conda) for the project, and do...
pip install -e .
From here, I could fire up a python shell, or even configure a jupyter kernel to use this virtual environment, and I'd always have the latest version of my package to interact with.
Now setuptools has its limitations, and we've since moved on to Poetry to more tightly control dependencies and handle more sophisticated build needs and such.
The problem with poetry
If you look up what's the pip install -e . equivalent in poetry you will find this issue. Looks like the creator of poetry thinks installing directly from source like this is a hack and has no interest in supporting it. (BTW: I've tried poetry build and then pulling out the setup.py file like he suggested and it does not work)
Linking directly to source is not necessary, I'm willing to run an install command to get the latest version of the package. And when I do this with poetry, it appears to work.
cd root/of/my/project
poetry install
Installing dependencies from lock file
No dependencies to install or update
Installing the current project: my-project (0.4.8) <-- this is the latest version according to the source code in the working directory
The problem is that if I open a Python shell and try and import my package for instance, it is linked to the last version of my package that installed from a remote artifact repository (via pip install my-package) – not what's in my working directory.
python
...
>>> import my_package
>>> my_package.__version__
'0.4.7`
Now, even though I'm using poetry I'm using a conda environment to specify the Python version for my project and then installing, and using, poetry from inside that.
source activate my-package
(my-package) ... $ poetry update
I also know that poetry (not very transparently) can create and manage its own virtual environment on your behalf. I thought maybe the reason this is not working is because I need to be inside this environment (whereas I was only inside my conda environment, while poetry is installing the 0.4.8 version of my package within the virtual environment it manages).
I tried both shell and run to test this out. I get the same result.
poetry shell
Virtual environment already activated: /Users/U6020643/.conda/envs/my-package
Python 3.8.5
...
>>> import my_package
>>> my_package.__version__
'0.4.7`
What gives?
The way I fixed this: I stopped using conda to manage the Python version for projects that involve poetry and instead use pyenv.
Below is how I made that work. This was very helpful!
1. Removing conda as default environment manager.
This involved removing conda base activate from ~/.bash_profile.
Now open a new shell and verify that there's no conda environment prefix, e.g. (base) ... $.
2. Installing pyenv using Homebrew
Had been awhile, and an OS upgrade or two, since I interacted with Homebrew. Needed to do some housekeeping.
brew cleanup # This made it so that brew update didn't take forever
brew update
brew upgrade
brew cleanup
Then...
brew install pyenv
3. Install Python version(s)
Let's say you need/want Python 3.
pyenv install 3.8.5
4. Set the local Python version for your project
cd your/project/root/
pyenv local 3.8.5
5. Install poetry
See here.
6. Use it
Now install and use shell. Check the version -> hey it works!
cd your/project/root
poetry install
poetry shell
my_package --version # Package has a CLI.
I'm developing an application via spyder that can read and convert dlt files into a txt extension file, therefore I need to add dltpy library to spyder
I found a site https://pypi.org/project/dltpy/ in which it provided commands( you'll find below) to install the "dltpy" library. however, I wasn't I able to install it via anaconda.
INSTALLATION:
git clone git+https://github.com/Equidamoid/dltpy
(cd dltpy; git checkout native-dltreader)
pip install --user ./dltpy
Is it possible to install a github library into spyder?
Thank you for your help.
I am trying to install deepmatcher package in python 3.6. To let this package to run in python you need have a torch==0.3.1 version. So I am trying to install torch==0.3.1 by running :
pip install torch==0.3.1
Error during installation:
Collecting torch==0.3.1
ERROR: Could not find a version that satisfies the requirement torch==0.3.1 (from versions: 0.1.2, 0.1.2.post1)
ERROR: No matching distribution found for torch==0.3.1
I even tried installing it using "peterjc123" package but still unable to uninstall it.
torch 0.3.1 is not listed official anymore. To install torch 0.3.1 one can either build it by source or using a whl file for the specific version.
Using a whl file
whl files can be found here provided by pytorch
The downloaded file can then be installed with pip install [path to downloaded file]
Build by source
To build torch by source one can checkout the desired version via git
git checkout v[version number]
For torch==0.3.1 this would be git checkout v0.3.1
After this follow the install instructions provided by README.md
More information can be found at: https://pytorch.org/get-started/previous-versions/
check if you have right python installed for you os architecture or
create a virtual env using conda and then try installing its better this way
I debated which Stackoverflow site this best fit but couldn't decide.
I'd like to contribute to an open-source project on Github, but I can't figure out how to prevent the stable version already installed on my machine and the development version I'd like to make a patch for from clashing on import.
The repository only suggests pip installing with the editable.
What I've done so far is: clone the repository locally, and then trying to import it in a Jupyter Notebook from the directory above. However, the Jupyter Notebook is referencing the stable version installed earlier with pip. I tried to append to sys.path the child directory holding the package but still the same issue. I can't seem to get relative imports working either. Do I need to uninstall the stable version?
Any tips are appreciated!
You'd use virtualenv for this. It will allow you to create an environment that is isolated from your system python and you can install the dev version of the library on it.
Basic usage (for Unix-like systems) is:
$ pip install virtualenv
$ virtualenv MY_ENV
$ cd MY_ENV
$ source bin/activate # activates the local python for this shell only
(MY_ENV)$ pip install <some-module> # installs to a local and isolated python
(MY_ENV)$ python ... # runs python in the local environment
(MY_ENV)$ deactivate # disable the isolated python
$
recently i found pip2pi, which is super useful to install pip package from local pypi repo. I just followed the post here to do that http://blog.nknj.me/python-guide-to-hacking-on-an-airplane
But when I use setup tools, requires, I do not know how I can make it to use my local repo. I do not want to download the packages each time when i run "python setup.py develop". I am getting started with pyramid, and it will be helpful if i can avoid downloading packages everytime i create a new virtualenv.
Thanks in advance
Here is workaround:
$ python setup.py egg_info
The command above creates a directory {PackageName}.egg-info and places requires.txt file into one, but downloads nothing. So you can use requires.txt file with pip command to install dependencies:
$ pip install -r {PackageName}.egg-info/requires.txt --index-url=file:///path/to/local/repo
After that, you can install your application without access to the internet:
$ python setup.py develop