I am trying to set up a mysql connection that will work with SqlAlchemy in Python 3.6.5 . I have the following in my Dockerfile:
RUN pip3 install -r /event_git/requirements.txt
I also have, in requirements.txt:
mysql-connector-python==8.0.15
However, I am not able to connect to the DB. Is there anything else that I need to do to set this up?
Update:
I got 8.0.5 working but not 8.0.15 . Apparently, a protobuff dependency was added; does anyone know how to handle that?
docker file is:
RUN apt-get -y update && apt-get install -y python3 python3-pip fontconfig wget nodejs nodejs-legacy npm
RUN pip3 install --upgrade pip
# Copy contents of this directory (i.e. full source) to image
COPY . /my_project
# Install Python dependencies
RUN pip3 install -r /event_git/requirements.txt
# Set event_git folder as working directory
WORKDIR /my_project
ENV LANG C.UTF-8
I am running it via
docker build -t event_git .;docker run -t -i event_git /bin/bash
and then executing a script; the db is on my local machine. This is working on mysql-connector-python==8.0.5 but not 8.0.15, so the setup is ok; I think I just need to fulfill the protobuff dependency that was added (see https://github.com/pypa/warehouse/issues/5537 for mention of protobuff dependency).
The mysql-connector-python has the Python Protobuf as an installation requirement, this means that protobuf will be installed along mysql-connector-python.
If this doesn't work, try to add protobuf==3.6.1 in your requirements.txt.
Figured out the issue. The key is that import mysql.connector needs to be at the top of the file where the create_engine is. Still not sure of the exact reason, but at the very least that seems to define _CONNECTION_POOLS = {}. If anyone knows why, please do give your thoughts.
Related
I'm trying to build a docker with python 3 and google-cloud-bigquery with the following docker file:
FROM python:3.10-alpine
RUN pip3 install google-cloud-bigquery
WORKDIR /home
COPY *.py /home/
ENTRYPOINT ["python3", "-u", "myscript.py"]
But getting errors on the pip3 install google-cloud-bigquery (too long for here)..
What's missing for installing this on python-alpine?
Looks like an incompatibility issue with the latest version of google-cloud-bigquery (>3) and numpy:
ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects
Try specifying a previous version, this works for me:
RUN pip3 install google-cloud-bigquery==2.34.4
Actually it seems like not a problem with numpy, which builds smoothly with all the dependency libs install, but rather with pyarrow, which does not support alpine+pip build. I've found a workaround by using alpine pre-built version of pyarrow. It is much easier than building pyarrow from source. This build works for me just fine:
FROM python:3.10.6-alpine3.16
RUN apk add --no-cache build-base linux-headers \
py3-apache-arrow=8.0.0-r0
# Copying pyarrow to site-package of actual python path. Alpine python path
# and python's docker hub path are different.
RUN mv /usr/lib/python3.10/site-packages/* \
/usr/local/lib/python3.10/site-packages/
RUN rm -rf /usr/lib/python3.10
RUN --mount=type=cache,target=/root/.cache/pip \
pip install google-cloud-bigquery==3.3.2
Update python version, alpine version and py3-apache-arrow version to install later versions. This is the latest one on the time of writing.
And make sure to remove build dependencies (build-base, linux-headers) for your release docker. I prefer multistage dockers for this.
Building a docker image, I've installed Negbio in my Dockerfile using:
RUN git clone https://github.com/ncbi-nlp/NegBio.git xyz && \
python xyz/setup.py install
When I try to run my Django application at localhost:1227 I get:
No module named 'negbio' ModuleNotFoundError exception
When I run pip list I can see negbio. What am I missing?
As per your comment, It wouldn't install with pip and hence not installing via pip.
Firstly, to make sure the https://github.com/ncbi-nlp/NegBio is properly installed via python setup.py install, you need to install it's dependencies via pip install -r requirements first. So either ways, you are doomed to have pip inside Docker.
For example, this is the sample Dockerfile that would install the negbio package properly:
FROM python:3.6-slim
RUN mkdir -p /apps
WORKDIR /apps
# Steps for installing the package via Docker:
RUN apt-get update && apt-get -y upgrade && apt-get install -y git gcc build-essential
RUN git clone https://github.com/ncbi-nlp/NegBio.git
WORKDIR /apps/NegBio
RUN pip install -r requirements.txt
RUN python setup.py install
ENV PATH=~/.local/bin:$PATH
EXPOSE 8000
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
So wouldn't harm if you actually install it via requirements.txt
I would do it like this:
requirements.txt --> have all your requirements added here
negbio==0.9.4
And make sure it's installed on the fly inside the docker using RUN pip install -r requirements.txt
I ultimately resolved my issue by going to an all Anaconda environment. Thank you for everyone's input.
I run through the following steps to attempt to start up an app for production:
-Setup a virtualenv for the python dependencies: virtualenv -p /usr/bin/python3.8 ~/app_env
-Install pip dependencies: . ~/app_env/bin/activate && pip install -r ~/app/requirements.txt
-Un-comment the lines for privilege dropping in uwsgi.ini and change the uid and gid to your account name
-Login to root with sudo -s and re-source the env with source /home/usr/app_env/bin/activate
-Set the courthouse to production mode by setting the environment variable with export PRODUCTION=1
-Start the app: cd /home/usr/app && ./start_script.sh
And I get the following error:
(app_env) root#usr-Spin-SP314-53N:/home/usr/Desktop/app# ./start.sh
uwsgi: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory
I tried a few things such as installing a newer libpcre version like mentioned here, tried also the steps mentioned here but that didn't work. Also the environment I'm setting up doesn't use anaconda but regular python. I even tried pip install uwsgi in my virtual env but it said the requirement was already satisfied. I'm not much of an expert when it comes to somewhat complex package management like this, help with how to solve this would be greatly appreciated. Thanks. I'm on Ubuntu 20.04, using python 3.8.
What solved it for me was apparently just reinstalling UWSGI, like in this thread, in my virtual env while forcing it to ignore the cache so it could know to use the pcre library I installed.
In order, doing this
uwsgi --version
Was giving me this
uwsgi: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory
So I made sure I had the latest libpcre installed
sudo apt-get install libpcre3-dev
And then what linked it all together was this
pip install uwsgi -I --no-cache-dir
I tried to solve this error but it did not work no matter what I did and then reinstalled uwsgi, but the following 2 lines solved my problem
sudo find / -name libpcre.so.*
#change the path of the /home/anaconda3/lib/libpcre.so.1 with the one appears after above one.
sudo ln -s /home/anaconda3/lib/libpcre.so.1 /lib
which python
I am using Apache Superset, specifically country_map to visualize data.
Is it possible to slice the country maps into more areas than just the regions of the country? How can I achieve that?
In order to make changes to country maps, you need to build Superset from sources.
First, fork the Apache Superset repo in your Github. Then clone the repo to your device and enter superset folder:
git clone https://github.com/username/incubator-superset.git
cd incubator-superset
Second:
sudo pip install -r docs/requirements.txt
python3 setup.py build_sphinx
Next, create virtual environment and install superset:
virtualenv -p python3 venv # if virtualenv not found use: `sudo -H pip3 install virtualenv`
source venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-dev.txt # Here I got error "python setup.py egg_info" failed with error code 1. You can skip it or try pip install --upgrade setuptools
pip install -e .
fabmanager create-admin --app superset
pip install python-dotenv # just in case you don't already have it
superset db upgrade # if error pip install pandas==0.23.4 plus pip install sqlalchemy==1.2.18
superset load_examples
superset init
Leave the venv environment to continue building the front-end:
deactivate # exit venv
cd superset/assets
npm ci
npm run dev
Next, go back to superset directory and start the flask local server:
cd superset
FLASK_ENV=development flask run -p 8088 --with-threads --reload --debugger
I got the instructions from Apache Superset GitHub Contributing page
Now, regarding the division of the country maps. What I did was download a new geojson format map and replace the superset map with the new map. Paste the new map in this directory.
cd incubator-superset/superset/assets/src/visualizations/CountryMap/countries
If this a new country and does not already exist in the directory, you also need to add the name at controls.jsx file. File is located here:
cd incubator-superset/superset/assets/src/explore
Open the file and add the new country at the select_country: {...} component. I got the instruction at Superset Visualization Tools Doc
In order for the new country map to show up in the web browser, you need to rerun the command npm run dev at assets directory, and restart the server.
That's what worked for me. Hope it'll be helpful for future users.
PS: Don't forget to upgrade npm in case you have an old version. You'll need it for npm ci command
How can I have python3.6 in tensorflow docker images.
All the images I tried (latest, nighty) are using python3.5 and I don't want to modify all my scripts.
The Tensorflow images are based on Ubuntu 16.04, as you can see from the Dockerfile. This release ships with Python 3.5 as standard.
So you'll have to re-build the image, and the Dockerfile will need editing, even though you need to do the actual build with the parameterized_docker_build.sh script.
This answer on ask Ubuntu covers how to get Python 3.6 on Ubuntu 16.04
The simplest way would probably be just to change the From line in the Dockerfile to FROM ubuntu:16.10, and python to python3.6 in the initial apt-get install line
Of course, this may break some other Ubuntu version-specific thing, so an alternative would be to keep Ubuntu 16.04 and install one of the alternative ppa's also listed in the linked answer:
RUN add-apt-repository ppa:deadsnakes/ppa &&
apt-get update &&
apt-get install -y python3.6
Note that you'll need this after the initial apt-get install, because that installs software-properties-common, which you need to add the ppa.
Note also, as in the comments to the linked answer, that you will need to symlink to Python 3.6.
Finally, note that I haven't tried any of this. The may be gotchas, and you may need to make another change to ensure that the correct version of Python is used by the running container.
You can use stable images which are supplied by third parties, like ufoym/deepo.
One that fits TensorFlow, python3.6 and cuda10 can be found here or you can pull it directly using the command docker pull ufoym/deepo:py36-cu100
I use their images all the time, never had problems
With this anwer, I just wanted to specify how I solved this problem (the previous answer of SiHa helped me a lot but I had to add a few steps so that it worked completly).
Context:
I'm using a package (segmentation model for unet++) that requires tensorflow==1.4.0 and keras==2.2.2.
I tried to use the docker image for tensorflow 1.4.0, however, the default version of python of this image is 3.5 which is not compatible with my package.
I managed to install python3.6 on the docker images thanks to the following files:
My Dockerfile contains the following lines:
Dockerfile:
FROM tensorflow/tensorflow:1.4.0-gpu-py3
RUN mkdir /AI_PLATFORM
WORKDIR /AI_PLATFORM
COPY ./install.sh ./install.sh
COPY ./requirements.txt ./requirements.txt
COPY ./computer_vision ./computer_vision
COPY ./config.ini ./config.ini
RUN bash install.sh
Install.sh:
#!/urs/bin/env bash
pip install --upgrade pip
apt-get update
apt-get install -y python3-pip
add-apt-repository ppa:deadsnakes/ppa &&
apt-get update &&
apt-get install python3.6 --assume-yes
apt-get install libpython3.6
python3.6 -m pip install --upgrade pip
python3.6 -m pip install -r requirements.txt
Three things are important:
use python3.6 -m pip instead of pip, else the packages are installed on python 3.5 default version of Ubuntu 16.04
use docker run python3.6 <command> to run your containers with python==3.6
in the requirements.txt file, I had to specify the following things:
h5py==2.10.0
tensorflow-gpu==1.4.1
keras==2.2.2
keras-applications==1.0.4
keras-preprocessing==1.0.2
I hope that this answer will be useful
Maybe the image I created will help you. It is based on the cuda-10.0-devel image and has tensorflow 2.0a-gpu installed.
You can use it as base image for your own implementation. The image itself doesn't do anything. I put the image on dockerhub https://cloud.docker.com/repository/docker/patientzero/tensorflow2.0a-gpu-py3.6
The github repo is located here: https://github.com/patientzero/tensorflow2.0-python3.6-Docker
Pulling it won't do much, but for completeness:
$ docker pull patientzero/tensorflow2.0-gpu-py3.6
edit: changed to general tensorflow 2.0x image.
Also as mentioned here, the official image for the beta 2.0 release now comes with python 3.6 support