docker-compose run: django and postgres "strange" behavior - python-3.x

When I run the following command:
docker-compose run django python manage.py startapp hello
OR
docker-compose run django python manage.py createsuperuser
Instead of starting a new container, it opens a Postgres shell.
However, when I'm using exec, like so:
docker-compose exec django python manage.py createsuperuser
Then it works as expected.
Here is my docker-compose.yml file:
version: '3.7'
services:
postgres:
image: 'postgres:9.6.9-alpine'
volumes:
- postgres_data:/var/lib/postgresql/data
env_file:
- ./.env.dev
django:
image: django_dev
build:
context: .
dockerfile: ./app/docker/django/Dockerfile
volumes:
- ./app/:/usr/src/app/
command: /usr/src/app/docker/django/start_dev
ports:
- 8000:8000
env_file:
- ./.env.dev
depends_on:
- postgres
node:
image: node_dev
build:
context: .
dockerfile: ./app/docker/node/Dockerfile
depends_on:
- django
volumes:
- ./app/:/usr/src/app/
- /usr/src/app/node_modules
command: npm run dev
ports:
- '3000:3000'
- "3001:3001"
volumes:
postgres_data:
Dockerfile:
FROM python:3.9.5-slim-buster
WORKDIR /usr/src/app
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
RUN apt update \
&& apt install -y curl \
&& curl -sL https://deb.nodesource.com/setup_lts.x | bash - \
&& apt-get -y install libpq-dev gcc \
&& pip install psycopg2 \
&& pip install psycopg2-binary
RUN pip install --upgrade pip \
&& pip install pipenv
COPY ./app/Pipfile /usr/src/app/Pipfile
RUN pipenv install --skip-lock --system --dev
RUN apt-get update && apt-get install -y libglu1
COPY ./app/docker/django/entrypoint /usr/src/app/docker/django/entrypoint
COPY ./app/docker/django/start_dev /usr/src/app/docker/django/start_dev
RUN sed -i 's/\r$//g' /usr/src/app/docker/django/start_dev
RUN chmod +x /usr/src/app/docker/django/start_dev
COPY . /usr/src/app/
ENTRYPOINT ["/usr/src/app/docker/django/entrypoint"]
entrypoint:
#!/bin/sh
bash | set -o errexit
bash | set +o pipefail
bash | set -o nounset
if [ -z "${POSTGRES_USER}" ]; then
base_postgres_image_default_user='postgres'
export POSTGRES_USER="${base_postgres_image_default_user}"
fi
export DATABASE_URL="postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}#${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}"
postgres_ready() {
python << END
import sys
import psycopg2
try:
psycopg2.connect(
dbname="${POSTGRES_DB}",
user="${POSTGRES_USER}",
password="${POSTGRES_PASSWORD}",
host="${POSTGRES_HOST}",
port="${POSTGRES_PORT}",
)
except psycopg2.OperationalError:
sys.exit(-1)
sys.exit(0)
END
}
until postgres_ready; do
>&2 echo 'Waiting for PostgreSQL to become available...'
sleep 1
done
>&2 echo 'PostgreSQL is available'
exec "$#"
start_dev:
#!/bin/sh
bash | set -o errexit
bash | set +o pipefail
bash | set -o nounset
python manage.py flush --no-input
python manage.py makemigrations
python manage.py migrate
python manage.py createsuperuser --noinput
python manage.py runserver 0.0.0.0:8000
I racked my brains for a few days, but couldn't find an explanation for this behavior.

Actually it is not opening postgres shell. It said that depended cotainer with postgres is running. A path that showed is /usr/src/app/ as in your django component. You can also checked hash of your django app and hash in bash line '7fbcde....'

Related

How to run custom django-admin commands as a cron job in docker?

I have a django-app sample. In this app I have created a file test.py and wrote my custom commands in it as mentioned in the django official documentation
sample/
__init__.py
models.py
management/
__init__.py
commands/
__init__.py
test.py
views.py
test.py:
from django.core.management.base import BaseCommand
class Command(BaseCommand):
def handle(self, *args, **options):
# Sample content
print("Hello World")
I had run the command python manage.py test and it is working. What I need is that I want to run this command as a cron job in docker. I had tried this, created a cron file hello-cron.
hello-cron:
* * * * * root python /home/manage.py test >> /home/cron.log 2>&1
# Empty line
Dockerfile:
FROM python:3.10.6
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
WORKDIR /home
COPY requirements.txt /home/
RUN pip install -r requirements.txt
COPY . /home/
RUN apt-get update && apt-get -y install cron
COPY hello-cron /home/hello-cron
RUN chmod 0644 /home/hello-cron
RUN chmod 0744 /home/sample/management/commands/test.py
RUN crontab /home/hello-cron
RUN touch /home/cron.log
CMD cron && tail -f /home/cron.log
docker-compose.yml:
version: "3.9"
services:
db:
image: postgres
volumes:
- ./data/db:/var/lib/postgresql/data
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
web:
restart: always
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/home
ports:
- "8000:8000"
environment:
- POSTGRES_NAME=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
depends_on:
- db
I had run the container by using the command docker-compose up --build, but cron job is not working. How can I run custom django-admin commands as a cron job in docker?

Dockerized Node JS application not accessible from host machine

I have Node JS application
docker-compose.yml
version: '3'
services:
app:
build:
context: .
dockerfile: Dockerfile
command: 'yarn nuxt'
ports:
- 3000:3000
volumes:
- '.:/app'
Dockerfile
FROM node:15
RUN apt-get update \
&& apt-get install -y curl
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
&& echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list \
&& apt-get update \
&& apt-get install -y yarn
WORKDIR /app
After running $ docker-compose up -d application starts and inside container it's accessible
$ docker-compose exec admin sh -c 'curl -i localhost:3000'
// 200 OK
But outside of container it's doesnt work. For example in chrome ERR_SOCKET_NOT_CONNECTED
Adding this to app service solves problem in docker-compose.yml
environment:
HOST: 0.0.0.0
Thanks to Marc Mintel article Development setup with Nuxt, Node and Docker
did you try to add
published: 3000
you can read more here - https://docs.docker.com/compose/compose-file/compose-file-v3/

Running chown in Dockerfile does nothing

I'm having some trouble setting up a Nuxt and Rails container using Docker. The two containers are separate, but interact with each other.
Currently, I'm having trouble running the dev servers for both the Nuxt and the Rails containers due to insufficient permissions. Looking at the logs for both of the containers, it seems that Docker can't do actions such as mkdir.
EACCESS: Permission Denied: 'mkdir: /usr/src/app/.nuxt' # nuxt
EACCESS: Permission Denied: 'mkdir: /usr/src/app/tmp' # rails
My docker-compose.dev.yml file
version: 3
services:
backend:
privileged: true
image: tablevibes-backend
build:
dockerfile: Dockerfile-dev
context: tablevibes-backend
args:
UID: ${UID:-1001}
BUNDLER_VERSION: 2.0.2
PG_MAJOR: 10
mode: development
tty: true
stdin_open: true
volumes:
- ./tablevibes-backend:/usr/src/app:Z
- gem_data_api:/usr/local/bundle:cached
ports:
- "3000:3000"
depends_on:
- db
user: rails
client-ui:
image: client-ui
command: yarn run dev
build:
context: client-ui
dockerfile: Dockerfile-dev
args:
UID: ${UID:-1001}
PORT: 5000
MODE: DEV
restart: always
volumes:
- ./client-ui:/usr/src/app
- client_ui_node_modules:/usr/src/app/node_modules:cached
ports:
- 5000:5000
user: client-ui
The 2 Dockerfiles
The Rails Dockerfile-dev
FROM ruby:2.6.3
ARG PG_MAJOR
ARG BUNDLER_VERSION
ARG UID
ARG MODE
RUN adduser rails --uid $UID --disabled-password --gecos ""
# Add POSTGRESQL to the source list using the right version
RUN curl -sSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
&& echo 'deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main' $PG_MAJOR > /etc/apt/sources.list.d/pgdg.list
ENV RAILS_ENV $MODE
RUN apt-get update -qq && apt-get install -y postgresql-client-$PG_MAJOR vim
RUN apt-get -y install sudo
WORKDIR /usr/src/app
CMD chown -R rails /usr/src/app
COPY Gemfile /usr/src/app/Gemfile
COPY Gemfile.lock /usr/src/app/Gemfile.lock
ENV BUNDLER_VERSION $BUNDLER_VERSION
RUN gem install bundler:$BUNDLER_VERSION
RUN bundle install
COPY . /usr/src/app
# Add a script to be executed every time the container starts.
COPY entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/entrypoint.sh
ENTRYPOINT ["entrypoint.sh"]
EXPOSE 3000
# Start the main process.
CMD ["rails", "server", "-b", "0.0.0.0"]
The Nuxt Dockerfile-dev
FROM node:10
ARG UID
ARG MODE=DEV
ARG PORT
RUN adduser client-ui --uid $UID --disabled-password --gecos ""
RUN apt-get update
RUN apt-get -y install sudo
RUN mkdir /usr/src/app
RUN chown -R client-ui /usr/src/app
COPY package.json yarn.lock /usr/src/app
RUN yarn install
COPY . /usr/src/app
ENV API_URL=http://localhost:3000/v1
ENV REVIEW_URL=http://localhost:8000
# expose 5000 on container
EXPOSE $PORT
# set app serving to permissive / assigned
ENV NUXT_HOST=0.0.0.0
# set app port
ENV NUXT_PORT=$PORT
My problem is that those lines where I do RUN chown ... never seem to take place. If I manually go into the containers with docker exec -u root -it backend bash and run chown -R rails . manually, everything works as expected. Likewise I tried running chmod 777 as a test, however that also had no effect on the permission denied error I keep getting.
What might be causing Docker to ignore my chown command?
This Stack Overflow question seems relevant, however it doesn't quite apply because I don't have any VOLUME mounts inside my Dockerfiles. A user in the comments of the accepted answer has my same issue, though unfortunately no solution.
Containers are like ogres, or onions, they have layers.
When using VOLUMEs or MOUNTs, the directory (or file) is not actually IN the container, but only appears to be in it.
Your Dockerfile uses a layer for /usr/src/app, which as you probably already know is your ./tablevibes-backend directory on your host computer.
services:
backend:
volumes:
- ./tablevibes-backend:/usr/src/app:Z
When you use a volume or mount in this way, the only thing Docker can do is simple CRUD (create, read, update, delete) options, it can not (and should not) modify the metadata as it is modifying your host drive, which could be a security issue.
try this:
sudo chown -R $USER:$USER .

How to fix PSQL connection error with Docker Compose

I'm trying to connect my Python-Flask app with a Postgres database in a docker environment. I am using a docker-compose file to build my web and db environment.
However, I am getting the following error:
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Here is my docker file:
FROM ubuntu:16.04 as base
RUN apt-get update -y && apt-get install -y python3-pip python3-dev postgresql libpq-dev libffi-dev jq
ENV LC_ALL=C.UTF-8 \
LANG=C.UTF-8
ENV FLASK_APP=manage.py \
FLASK_ENV=development \
APP_SETTINGS=config.DevelopmentConfig \
DATABASE_URL=postgresql://user:pw#postgres/database
COPY . /app
WORKDIR /app
RUN pip3 install -r requirements.txt
FROM base as development
EXPOSE 5000
CMD ["bash"]
Here is my Docker-compose file:
version: "3.6"
services:
development_default: &DEVELOPMENT_DEFAULT
build:
context: .
target: development
working_dir: /app
volumes:
- .:/app
environment:
- GOOGLE_CLIENT_ID=none
- GOOGLE_CLIENT_SECRET=none
web:
<<: *DEVELOPMENT_DEFAULT
ports:
- "5000:5000"
depends_on:
- db
command: flask run --host=0.0.0.0
db:
image: postgres:10.6
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=db

ServiceUnavailableException with Neo4j and Spark from Docker

I have created a Docker image for my application which runs with Spark Streaming, Kafka, ElasticSearch, and Kibana. I packaged it into an executable jar file. When I run the application with this command everything works fine as expected (the data starts to be produced):
java -cp "target/scala-2.11/test_producer.jar" producer.KafkaCheckinsProducer
However, when I run it from docker I get an error of connection to Neo4j, although database runs from docker-compose file:
INFO: Closing connection pool towards localhost:7687
Exception in thread "main" org.neo4j.driver.v1.exceptions.ServiceUnavailableException: Unable to connect to localhost:7687, ensure the database is running and that there is a working network connection to it.
I run my application this way:
docker run -v my-volume:/workdir -w /workdir container-name
What could cause this problem? And what should I change in my Dockerfile to execute this application?
Here is the Dockerfile:
FROM java:8
ARG ARG_CLASS
ENV MAIN_CLASS $ARG_CLASS
ENV SCALA_VERSION 2.11.8
ENV SBT_VERSION 1.1.1
ENV SPARK_VERSION 2.2.0
ENV SPARK_DIST spark-$SPARK_VERSION-bin-hadoop2.6
ENV SPARK_ARCH $SPARK_DIST.tgz
VOLUME /workdir
WORKDIR /opt
# Install Scala
RUN \
cd /root && \
curl -o scala-$SCALA_VERSION.tgz http://downloads.typesafe.com/scala/$SCALA_VERSION/scala-$SCALA_VERSION.tgz && \
tar -xf scala-$SCALA_VERSION.tgz && \
rm scala-$SCALA_VERSION.tgz && \
echo >> /root/.bashrc && \
echo 'export PATH=~/scala-$SCALA_VERSION/bin:$PATH' >> /root/.bashrc
# Install SBT
RUN \
curl -L -o sbt-$SBT_VERSION.deb https://dl.bintray.com/sbt/debian/sbt-$SBT_VERSION.deb && \
dpkg -i sbt-$SBT_VERSION.deb && \
rm sbt-$SBT_VERSION.deb
# Install Spark
RUN \
cd /opt && \
curl -o $SPARK_ARCH http://d3kbcqa49mib13.cloudfront.net/$SPARK_ARCH && \
tar xvfz $SPARK_ARCH && \
rm $SPARK_ARCH && \
echo 'export PATH=$SPARK_DIST/bin:$PATH' >> /root/.bashrc
EXPOSE 9851 9852 4040 9092 9200 9300 5601 7474 7687 7473
CMD /workdir/runDemo.sh "$MAIN_CLASS"
And here is a docker-compose file:
version: '3.3'
services:
kafka:
image: spotify/kafka
ports:
- "9092:9092"
environment:
- ADVERTISED_HOST=localhost
neo4j_db:
image: neo4j:latest
ports:
- "7474:7474"
- "7473:7473"
- "7687:7687"
volumes:
- /var/lib/neo4j/import:/var/lib/neo4j/import
- /var/lib/neo4j/data:/data
- /var/lib/neo4j/conf:/conf
environment:
- NEO4J_dbms_active__database=graphImport.db
elasticsearch:
image: elasticsearch:latest
ports:
- "9200:9200"
- "9300:9300"
networks:
- docker_elk
volumes:
- esdata1:/usr/share/elasticsearch/data
kibana:
image: kibana:latest
ports:
- "5601:5601"
networks:
- docker_elk
volumes:
esdata1:
driver: local
networks:
docker_elk:
driver: bridge
From error message - you're trying to connect to localhost that is local to your application, not to the host on which it's running. You need to connect to correct host name inside the Docker network - you don't need to map all ports into your host, you just need to check that all Docker images in the same network.

Resources