I'm trying to create a setup script for a python project. Assume that python 3.8 and pip are already installed properly and that running them directly works, but when running them through an sh file breaks with a permission denied. Python is installed globally.
OS: Windwos10
Py: 3.8
Below are the sh scripts that break
ROOT_PATH='./'
if [ ! -f "$ROOT_PATH.env" ]; then
echo 'Copying .env file from template'
cp "$ROOT_PATH.env.template" "$ROOT_PATH.env"
fi
echo 'Installing app dependencies...'
pip3 install -r requirements.txt --user
echo 'Run migrations'
python -m flask db upgrade
Related
I am trying to set up a CI/CD pipeline for a python project using a Windows runner on GitLab.
However, when executing pytest, pytest collects 10 items, and opens the first test file. After that, the pipeline continues running, but nothing happens and the pipeline stops after time-out. All test work correctly locally and take around 30 seconds in total.
The root directoty for pytest is correct.
This is my Gitlab yml file:
image: python:3.11.0-buster
cache:
paths:
- .cache/pip
- venv/
tests:
tags:
- windows
before_script:
- mkdir -p ~/.ssh
- echo "$DEPLOY_KEY" > ~/.ssh/id_rsa
- echo "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- Import-Module "$env:ChocolateyInstall\helpers\chocolateyProfile.psm1"
- choco install python --version=3.11 -y -f
# - dir
- C:\\Python311\\python.exe -m pip install --upgrade pip
- C:\\Python311\\python.exe -m pip install --upgrade setuptools
- C:\\Python311\\python.exe -m pip install -r requirements.txt
# - where python
- C:\\Python311\\python.exe -m pip install virtualenv
script:
- C:\\Python311\\python.exe -m pytest -p no:faulthandler
I've also tried - C:\Python311\python.exe -m pytest which had the same result
I'm trying to do some image recognition inside the AWS nitro-enclave with python. But the code hangs when importing packages such as OpenCV, NumPy, and pandas. The dockerfile file used to build the enclave would function normally in my local machine or in EC2. The generated enclave console would output some openBLAS warning about L2 cache size and the process freezes. No error output of any sort.
Is there any additional dependencies I need to add when using packages in enclave or there are some conflicts with the kernel?
The docker, shell, and py test codes are shown below:
#amazonlinux still have the import issue
#python:3.7 libs importing crush
FROM amazonlinux
WORKDIR /app
#py 3.7
RUN yum install python3 zip -y
ENV VIRTIAL_ENV=/opt/venv
RUN python3 -m venv $VIRTIAL_ENV
ENV PATH="$VIRTIAL_ENV/bin:$PATH"
#3 libs needed for cv2 import
RUN yum install libSM-1.2.2-2.amzn2.x86_64 -y
RUN yum install libXrender-0.9.10-1.amzn2.x86_64 -y
RUN yum install libXext-1.3.3-3.amzn2.x86_64 -y
COPY requirements.txt ./
RUN pip3 install --no-cache-dir -r /app/requirements.txt
#shell script testing
COPY dockerfile_entrypoint.sh ./
COPY test_cv2.py ./
#ENV for shell testing printf loop
ENV HELLO="Hello from enclave side!"
RUN chmod +X dockerfile_entrypoint.sh
#shell script testing
CMD ["/app/dockerfile_entrypoint.sh"]
#!/bin/bash
#shell printf loop test in enclave
# go to work dir and check files
cd /app||return
ls
#cv2 imp issue
python3 test_cv2.py
#use shell loop to keep enclave live to see error message output
count=1
while true;do
printf "[%4d] $HELLO\n" $count
echo "$PWD"
ls
count=$((count+1))
sleep 5
done
import cv2
for i in range(10):
print('testing OpenCV')
These types of hangs can happen when applications or libraries attempt to read data from /dev/random but there is not sufficient entropy, which causes the process to block on the read. There are some possible solutions in this GitHub issue: https://github.com/aws/aws-nitro-enclaves-sdk-c/issues/41#issuecomment-792621500
I have a set of scripts in .travis.yml which runs perfectly fine at the moment
...
install:
- scripts/travis/install_deps.sh
- virtualenv -p /opt/pyenv/versions/3.6/bin/python3.6 venv
- source venv/bin/activate
- pip install -r requirements.txt
before_script:
- scripts/test.sh
script:
- scripts/travis/build.sh
after_success:
- deactivate
- virtualenv -p /opt/pyenv/versions/2.7/bin/python2.7 venv
- source venv/bin/activate
- pip install -r requirements.txt
...
However, I would like to clean it up a bit so there's less repetition such that .travis.yml looks like
...
install:
- scripts/travis/install_deps.sh
- export PYTHON_VERSION=3.6
- scripts/travis/install_python_deps.sh
before_script:
- scripts/test.sh
script:
- scripts/travis/build.sh
after_success:
- export PYTHON_VERSION=2.7
- scripts/travis/install_python_deps.sh
...
where install_python_deps.sh looks like
#!/usr/bin/env bash
set -e
if [ ! -z "$VIRTUAL_ENV" ]; then deactivate; fi
virtualenv -p "/opt/pyenv/versions/${PYTHON_VERSION}/bin/python${PYTHON_VERSION}" venv
source venv/bin/activate
pip install -r requirements.txt
The problem arises when this is run in travis. The build breaks when test.sh, which runs a python script that relies on a module declared in requirements.txt is not found. Any pointers as to why this is occurring would be greatly appreciated.
The source venv/bin/activate inside scripts/travis/install_python_deps.sh only has effect until the script install_python_deps.sh exits.
If you want to use the installed modules outside the install_python_deps.sh script,
you need to run source venv/bin/activate (again) outside the script too, for example:
...
install:
- scripts/travis/install_deps.sh
- scripts/travis/install_python_deps.sh 3.6
- source venv/bin/activate
before_script:
- scripts/test.sh
script:
- scripts/travis/build.sh
after_success:
- scripts/travis/install_python_deps.sh 2.7
- source venv/bin/activate
...
Note that to make it shorter, I replaced the PYTHON_VERSION environment variable with a command line parameter. You could adjust the scripts/travis/install_python_deps.sh script accordingly:
#!/usr/bin/env bash
set -euo pipefail
PYTHON_VERSION=$1
if [ "$VIRTUAL_ENV" ]; then deactivate; fi
virtualenv -p "/opt/pyenv/versions/${PYTHON_VERSION}/bin/python${PYTHON_VERSION}" venv
source venv/bin/activate
pip install -r requirements.txt
I have Ubuntu 16.04 (on Docker) and wanted to connect to remote Oracle DB using Python. For that - using cx_oracle module.
Tried:
pip install cx_oracle
--> Complained about libaio1 and libaio-dev missing..
apt-get install libaio1 libaio-dev
--> Complained again:
cx_Oracle.DatabaseError: DPI-1047: 64-bit Oracle Client library cannot be loaded: "libclntsh.so: cannot open shared object file: No such file or directory"
Is there a one command to install cx_Oracle properly on Ubuntu 16.04 (or need to do all from source manually -> trying to automate all steps...)?
Thanks.
Did not find (yet) easy way but this is what I did:
This just worked for me on Ubuntu 16:
Download ('instantclient-basic-linux.x64-12.2.0.1.0.zip' and 'instantclient-sdk-linux.x64-12.2.0.1.0.zip') from Oracle web site and then do following script (you can do piece by piece and I did as a ROOT):
apt-get install -y python-dev build-essential libaio1
mkdir -p /opt/ora/
cd /opt/ora/
## Now put 2 ZIP files:
# ('instantclient-basic-linux.x64-12.2.0.1.0.zip' and 'instantclient-sdk-linux.x64-12.2.0.1.0.zip')
# into /opt/ora/ and unzip them -> both will be unzipped into 1 directory: /opt/ora/instantclient_12_2
rm -rf /etc/profile.d/oracle.sh
echo "export ORACLE_HOME=/opt/ora/instantclient_12_2" >> /etc/profile.d/oracle.sh
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME" >> /etc/profile.d/oracle.sh
chmod 777 /etc/profile.d/oracle.sh
source /etc/profile.d/oracle.sh
env | grep -i ora # This will check current ENVIRONMENT settings for Oracle
rm -rf /etc/ld.so.conf.d/oracle.conf
echo "/opt/ora/instantclient_12_2" >> /etc/ld.so.conf.d/oracle.conf
ldconfig
cd $ORACLE_HOME
ls -lrth libclntsh* # This will show which version of 'libclntsh' you have... --> needed for following line:
ln -s libclntsh.so.12.1 libclntsh.so
pip install cx_Oracle # Maybe not needed but I did it anyway (only pip install cx_Oracle without above steps did not work for me...)
Now python scripts are ready to use 'cx_Oracle'.
Linux Mint 17 with Python 3.5.1
$ python3 -m venv ENV_DIR
returns:
Error: Command '['/home/path/to/ENV_DIR/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1
I have tried various ways to get the terminal to show I am in a venv with no success.
That's due to a bug in Ubuntu 14.04 (on which Linux Mint 17 is based)
You can either create the virtual environment without pip and then manually install it inside the environment:
$ python3 -m venv --without-pip my_venv
$ source my_venv/bin/activate
(my_venv)$ curl https://bootstrap.pypa.io/get-pip.py | python
Or you can manually install ensurepip in your system, by adapting these instructions to Python 3.5.1:
cd /usr/lib/python3.5
sudo mkdir -p ensurepip/_bundled
cd ensurepip
sudo wget https://github.com/akheron/cpython/raw/v3.5.1/Lib/ensurepip/__init__.py
sudo wget https://github.com/akheron/cpython/raw/v3.5.1/Lib/ensurepip/__main__.py
sudo wget https://github.com/akheron/cpython/raw/v3.5.1/Lib/ensurepip/_uninstall.py
cd _bundled
sudo wget https://github.com/akheron/cpython/raw/v3.5.1/Lib/ensurepip/_bundled/pip-1.5.4-py2.py3-none-any.whl
sudo wget https://github.com/akheron/cpython/raw/v3.5.1/Lib/ensurepip/_bundled/setuptools-