I'm trying to do some image recognition inside the AWS nitro-enclave with python. But the code hangs when importing packages such as OpenCV, NumPy, and pandas. The dockerfile file used to build the enclave would function normally in my local machine or in EC2. The generated enclave console would output some openBLAS warning about L2 cache size and the process freezes. No error output of any sort.
Is there any additional dependencies I need to add when using packages in enclave or there are some conflicts with the kernel?
The docker, shell, and py test codes are shown below:
#amazonlinux still have the import issue
#python:3.7 libs importing crush
FROM amazonlinux
WORKDIR /app
#py 3.7
RUN yum install python3 zip -y
ENV VIRTIAL_ENV=/opt/venv
RUN python3 -m venv $VIRTIAL_ENV
ENV PATH="$VIRTIAL_ENV/bin:$PATH"
#3 libs needed for cv2 import
RUN yum install libSM-1.2.2-2.amzn2.x86_64 -y
RUN yum install libXrender-0.9.10-1.amzn2.x86_64 -y
RUN yum install libXext-1.3.3-3.amzn2.x86_64 -y
COPY requirements.txt ./
RUN pip3 install --no-cache-dir -r /app/requirements.txt
#shell script testing
COPY dockerfile_entrypoint.sh ./
COPY test_cv2.py ./
#ENV for shell testing printf loop
ENV HELLO="Hello from enclave side!"
RUN chmod +X dockerfile_entrypoint.sh
#shell script testing
CMD ["/app/dockerfile_entrypoint.sh"]
#!/bin/bash
#shell printf loop test in enclave
# go to work dir and check files
cd /app||return
ls
#cv2 imp issue
python3 test_cv2.py
#use shell loop to keep enclave live to see error message output
count=1
while true;do
printf "[%4d] $HELLO\n" $count
echo "$PWD"
ls
count=$((count+1))
sleep 5
done
import cv2
for i in range(10):
print('testing OpenCV')
These types of hangs can happen when applications or libraries attempt to read data from /dev/random but there is not sufficient entropy, which causes the process to block on the read. There are some possible solutions in this GitHub issue: https://github.com/aws/aws-nitro-enclaves-sdk-c/issues/41#issuecomment-792621500
I created a Dockerfile and then built it for my team to use. Currently I am pulling from the CentOS:latest image, then building the latest version of Python and saving the image to a .tar file. The idea is for my colleagues to use this image to add their Python projects to the /pyscripts folder. Is this the recommended way of building a base image or is there a better way I can go about doing it?
# Filename: Dockerfile
From centos
RUN yum -y update && yum -y install gcc openssl-devel bzip2-devel libffi-devel wget make && yum clean all
RUN cd /opt && wget --no-check-certificate https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz && tar xzf Python-3.8.3.tgz && cd Python-3.8*/ && ./configure --enable-optimizations && make altinstall && rm -rf /opt/Python* && mkdir /pyscripts
Many thanks!
Yes this is the standard and recommended way of building a base image from a parent image (CentOS in this example) if that is what you need Python 3.8.3 (latest version) on CentOS system.
Alternatively you can pull a generic Python image with latest Python version (which is now 3.8.3) but based on other Linux distribution (Debian) from Docker HUB repository by running:
docker pull python:latest
And then build a base image from it where you will just need to create the directory /pyscripts
So the Dockerfile would look like that:
FROM python:latest
RUN mkdir /pyscripts
Or you can pull CentOS/Python already built image (with lower version 3.6) from Docker HUB repository by running:
docker pull centos/python-36-centos7
And then build a base image from it where you will just need to create the directory /pyscripts
So the Dockerfile would look like that:
FROM centos/python-36-centos7:latest
USER root
RUN mkdir /pyscripts
Remember to add this line just after the first line to run the commands as root:
USER root
Otherwise you would get a Permission Denied error message
I am having trouble with extracting a .tar.gz file and accessing its files on a docker image. I've tried looking around Stackoverflow, but the solutions didn't fix my problem... Below is my folder structure and my Dockerfile. I've made an image called modus.
Folder structure:
- modus
Dockerfile
ModusToolbox_2.1.0.1266-linux-install.tar.gz
Dockerfile:
FROM ubuntu:latest
USER root
RUN apt-get update -y && apt-get upgrade -y && apt-get install git -y
COPY ./ModusToolbox_2.1.0.1266-linux-install.tar.gz /root/
RUN cd /root/ && tar -C /root/ -zxzf ModusToolbox_2.1.0.1266-linux-install.tar.gz
I've been running the commands below, but when I try to check /root/ the extracted files aren't there...
docker build .
docker run -it modus
root#e19d081664e4:/# cd root
root#e19d081664e4:/# ls
<prints nothing>
There should be a folder called ModusToolBox, but I can't find it anywhere. Any help is appreciated.
P.S
I have tried changing ADD to COPY, but both don't work.
You didn't provide a tag option with -t but you're using a tag in
docker run -it modus. By doing that you run some other modus image, not the one you have just built. Docker should say something
like Successfully built <IMAGE_ID> at the end of the build, run
docker run -it <IMAGE_ID> to run a newly built image if you don't want to provide a tag.
I have Dockerfile that looks like this
FROM alpine:3.7
WORKDIR /home/tmp
RUN apk add autoconf && apk add py-pip && apk add python3 &&\
pip install --upgrade pip && pip install wheel
Originally I wanted to execute a .sh script on startup (via ENTRYPOINT) and immediately destroy the container. However as it failed to find the file I decided to do that manually.
I run container like this
docker run -it --rm -v c:/projects/mega-nz-sdk:/home/tmp mega_sdk_python
And it connects me to bash in the container.
In the list of files I can see the script I want to execute
/home/tmp # ls
Dockerfile compile.sh sdk-develop
/home/tmp #
However when I try to run it it cannot find the script
/home/tmp # ./compile.sh
/bin/sh: ./compile.sh: not found
/home/tmp #
What is the problem?
Script compile.sh looks like this
#!/bin/bash
cd sdk-develop
sh autogen.sh
./configure --disable-silent-rules --enable-python --disable-examples &&\
make
cd /bindings/python
python setup.py bdist_wheel
Ideally I would like to execute during instantiation of the container in order to have already configured container on startup (without need to run script each I run the container).
It seems in order to execute my .sh file I need to run it like this
sh compile.sh
So I added
CMD ["sh", "compile.sh"]
And I started to work (though failed with other errors like missing make etc. but that's due to missing packages in Alpine Linux itself so a separate matter).
Guess It is something to do with Alpine Linux itself. But I am not sure.
Alpine Linux is a very minimal distribution; it includes a minimal version of most Unix tools that conform to the POSIX specification, but no more. In particular it does not include GNU Bash.
Your script doesn't actually use any special Bash features, so it would be enough to change the first line of the script to run the default system Bourne shell
#!/bin/sh
Using the Alpine apk package manager to install bash would work too, but it's not necessary for what you're showing here.
Usually you'd run the sorts of "compile" commands you show during the course of building an image, not when the image starts up. I'd expect a much more typical Dockerfile to COPY the application source code and in then RUN the commands you show. That would happen just once, when you docker build the image, and not every time you want to run the packaged application.
you mounted the directory from your win host to the docker machine. I BELIEVE this is a permission problem - the file is not executable flagged.
Show detailed listing
# ls -lh
copy the folder to internal dir and add executable bit
cp /home/tmp /home/tmp2 -r
chmod +x /home/tmp2/*.sh
/home/tmp2/compile.sh
I am trying to run docker-compose using sudo.
I have both docker and docker-compose installed on Ubuntu 16.01.
Due to an error while trying to download compose using curl, I ended up installing it using pip.
Docker version 1.12.0, build 8eab29e
docker-compose version 1.8.0, build 94f7016
Yet, when I try to run docker-compose with sudo I get the following (using sudo with docker is fine)
sudo: docker-compose: command not found
I suppose there are differing definitions of what 'installed' means. I have been using docker-compose on the same computer that claims it is not installed.
$ dpkg -s docker-compose
dpkg-query: package 'docker-compose' is not installed and no information is available
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.
$ whereis docker-compose
docker-compose: /home/user/.local/bin/docker-compose
$ pip show --files docker-compose
---
Metadata-Version: 2.0
Name: docker-compose
Version: 1.8.0
Summary: Multi-container orchestration for Docker
Home-page: https://www.docker.com/
Author: Docker, Inc.
Author-email: UNKNOWN
Installer: pip
License: Apache License 2.0
Location: /home/anton/.local/lib/python2.7/site-packages
Requires: six, jsonschema, enum34, cached-property, websocket-client, docker-py, requests, docopt, dockerpty, PyYAML, texttable
Classifiers:
Development Status :: 5 - Production/Stable
Environment :: Console
Intended Audience :: Developers
License :: OSI Approved :: Apache Software License
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.4
Files:
../../../bin/docker-compose
compose/GITSHA
compose/__init__.py
compose/__init__.pyc
compose/__main__.py
compose/__main__.pyc
compose/bundle.py
compose/bundle.pyc
compose/cli/__init__.py
compose/cli/__init__.pyc
compose/cli/colors.py
compose/cli/colors.pyc
compose/cli/command.py
compose/cli/command.pyc
compose/cli/docker_client.py
compose/cli/docker_client.pyc
compose/cli/docopt_command.py
compose/cli/docopt_command.pyc
compose/cli/errors.py
compose/cli/errors.pyc
compose/cli/formatter.py
compose/cli/formatter.pyc
compose/cli/log_printer.py
compose/cli/log_printer.pyc
compose/cli/main.py
compose/cli/main.pyc
compose/cli/signals.py
compose/cli/signals.pyc
compose/cli/utils.py
compose/cli/utils.pyc
compose/cli/verbose_proxy.py
compose/cli/verbose_proxy.pyc
compose/config/__init__.py
compose/config/__init__.pyc
compose/config/config.py
compose/config/config.pyc
compose/config/config_schema_v1.json
compose/config/config_schema_v2.0.json
compose/config/environment.py
compose/config/environment.pyc
compose/config/errors.py
compose/config/errors.pyc
compose/config/interpolation.py
compose/config/interpolation.pyc
compose/config/serialize.py
compose/config/serialize.pyc
compose/config/sort_services.py
compose/config/sort_services.pyc
compose/config/types.py
compose/config/types.pyc
compose/config/validation.py
compose/config/validation.pyc
compose/const.py
compose/const.pyc
compose/container.py
compose/container.pyc
compose/errors.py
compose/errors.pyc
compose/network.py
compose/network.pyc
compose/parallel.py
compose/parallel.pyc
compose/progress_stream.py
compose/progress_stream.pyc
compose/project.py
compose/project.pyc
compose/service.py
compose/service.pyc
compose/state.py
compose/state.pyc
compose/utils.py
compose/utils.pyc
compose/volume.py
compose/volume.pyc
docker_compose-1.8.0.dist-info/DESCRIPTION.rst
docker_compose-1.8.0.dist-info/INSTALLER
docker_compose-1.8.0.dist-info/METADATA
docker_compose-1.8.0.dist-info/RECORD
docker_compose-1.8.0.dist-info/WHEEL
docker_compose-1.8.0.dist-info/entry_points.txt
docker_compose-1.8.0.dist-info/metadata.json
docker_compose-1.8.0.dist-info/pbr.json
docker_compose-1.8.0.dist-info/top_level.txt
Entry-points:
[console_scripts]
docker-compose=compose.cli.main:main
I have tried the following - but still get the same error:
$ chmod +x /home/username/.local/bin/docker-compose
$ chmod +x /home/username/.local/lib/python2.7/site-packages
On Ubuntu 16.04
Here's how I fixed this issue: Refer Docker Compose documentation
sudo curl -L https://github.com/docker/compose/releases/download/1.21.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
After you do the curl command , it'll put docker-compose into the
/usr/local/bin
which is not on the PATH.
To fix it, create a symbolic link:
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
And now if you do:
docker-compose --version
You'll see that docker-compose is now on the PATH
The output of dpkg -s ... demonstrates that docker-compose is not installed from a package. Without more information from you there are at least two possibilities:
docker-compose simply isn't installed at all, and you need to install it.
The solution here is simple: install docker-compose.
docker-compose is installed in your $HOME directory (or other location not on root's $PATH).
There are several solution in this case. The easiest is probably to replace:
sudo docker-compose ...
With:
sudo `which docker-compose` ...
This will call sudo with the full path to docker-compose.
You could alternatively install docker-compose into a system-wide directory, such as /usr/local/bin.
If you have tried installing via the official docker-compose v1 page, where you need to download the binary using curl:
sudo curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
Then do not forget to add executable flag to the binary:
sudo chmod +x /usr/local/bin/docker-compose
If docker-compose is installed using python-pip
sudo apt-get -y install python-pip
sudo pip install docker-compose
try using pip show --files docker-compose to see where it is installed.
If docker-compose is installed in user path, then try:
sudo "PATH=$PATH" docker-compose
As I see from your updated post, docker-compose is installed in user path /home/user/.local/bin and if this path is not in your local path $PATH, then try:
sudo "PATH=$PATH:/home/user/.local/bin" docker-compose
I have same issue , i solved issue :
Following installs Docker Compose v1:
step-1 : download docker-compose using following command.
1. sudo su
2. sudo curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
Step-2 : Run command
chmod +x /usr/local/bin/docker-compose
Step-3 : Check docker-compose version
docker-compose --version
I will leave this here as a possible fix, worked for me at least and might help others. Pretty sure this would be a linux only fix.
I decided to not go with the pip install and go with the github version (option one on the installation guide).
Instead of placing the copied docker-compose directory into /usr/local/bin/docker-compose from the curl/github command, I went with /usr/bin/docker-compose which is the location of Docker itself and will force the program to run in root. So it works in root and sudo but now won't work without sudo so the opposite effect which is what you want to run it as a user anyways.
If docker-compose is installed for your user but not installed for root user and if you need to run it only once and forget about it afterwords perform the next actions:
Find out path to docker-compose:
which docker-compose
Run the command specifying full path to docker-compose from the previous command, eg:
sudo /home/your-user/your-path-to-compose/docker-compose up
On Ubuntu just run
sudo apt install docker-compose
This worked on Ubuntu 22*
You might be using a deprecated code. Try writing docker compose instead of docker-compose.
Simple, just reinstall the docker-compose. It will fix the issue.
sudo apt install docker-compose
There are too many pending edits on Stack Overflow, so I'm submitting a new answer that supplements earlier answers.
I propose the answer is syntax: docker compose vs docker-compose depending on how you have it installed, which is the same proposed answer by ibrahim shazly on Sep 27, 2022 and Gaurav Kumar Verma on Sep 29, 2022.
Here is an explanation. I followed https://docs.docker.com/engine/install/ubuntu/#install-docker-engine to install Docker on Ubuntu 22 LTS. Then, I attempted to run docker compose like so:
$ docker-compose up -d myserver
Command 'docker-compose' not found, but can be installed with:
...
As Gaurav Kumar Verma points out, https://docs.docker.com/compose/install/other/ has a disclaimer:
Note that Compose standalone uses the -compose syntax instead of the current standard syntax compose.
For example type docker-compose up when using Compose standalone, instead of docker compose up.
Thus, I ran:
$ docker compose up -d myserver
And it ran successfully.
Or, just add your binary path into the PATH. At the end of the bashrc:
...
export PATH=$PATH:/home/user/.local/bin/
save the file and run:
source .bashrc
and the command will work.
docker-compose is not installed. Please try to install it first from their official documentation
Whenever this type of error occurs please try to verify the installation of the apps by typing which command_name it will search for the executables and show you the location.
Had to delete a large log file then restart docker-compose for the space to clear up.
I used a script shell that cron runs once a day:
rm /var/log/cron/log_prodTEST.log
cd /opt/test/metrics
/usr/local/bin/docker-compose up --build --force-recreate -d -T
Apperntly by using the full path to docker-compose, I could use it inside the corn shell
you might try run the code by using docker compose instead of docker-compose
if it work you might want to check this link her
https://docs.docker.com/compose/install/other/
sudo curl -L "https://github.com/docker/compose/releases/download/v2.12.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo mv /usr/local/bin/docker-compose /usr/bin/docker-compose
sudo chmod +x /usr/bin/docker-compose