I have to build a simple app which reads a text file and process it's content (like remove multiple spaces, process words etc) but my I am confused about the first part of my homework.
"Initialize a git repository in a docker container then implement an app...."
I use Debian, I installed docker and git and I studied about it. From what I read I have to create a Dockerfile which will contain some instructions then I build the image and then run the container, run?
But I am still confused, what is the order of these thigs? Can I go firstly and write the app in Intelij and then to create that Dockerfiler? Or I have to create first the container then to code the app? But how I build the container? I read a lot about this, can you give me some advice? I mention that after every app "task" (read text file, process text etc) I have to execute git add, git commit and git push (if it helps for answer)
If the instruction says to "Initialize a Git repository in a docker container" then you are expected to:
run e.g. a Debian container
if Git is not present install it
initialize the repo
write your app
submit homework
You could:
docker run \
--interactive --tty --rm \
--name=homework \
--volume=${PWD}/homework:/homework \
--workdir=/homework \
debian:buster-slim
This will run a Debian "buster" image as a container and should (!) give you a shell prompt in the container.
A directory /homework in the container will be mapped to your host machine's ${PWD}/homework and you will be in the /homework directory when the container starts. This means that you won't lose your work if you exit the container.
From within the container's prompt:
# pwd
/homework
# git
bash: git: command not found
# apt update && apt install -y git
...
done.
# git
usage: git [--version] [--help] [-C <path>] [-c <name>=<value>]
[--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
[-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare]
[--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
<command> [<args>]
# git init
Initialized empty Git repository in /homework/.git/
Notes
If you exit the container, you can rerun the docker run ... command to return to it.
When you exist the container, you can ls -la ${PWD}/homework to see the .git directory (at least) persisted on your host.
Ensure you run it from the same directory where it created ${PWD}/homework. Or revise the --volume=...
I'd recommend an overall workflow of
Build the application, without Docker; then
Package it in a Docker image (if it makes sense to).
You should be able to build the application totally normally. Whatever language you're using to build the application, make sure to use its normal packaging tools. For example, your package.json/Gemfile/requirements.txt/go.mod should list out all of the library dependencies your application needs to run. Run it locally, write appropriate unit tests for it, and generally build something that works.
Once it works, then push it into Docker. You'll need to write a Dockerfile that builds the image. A generic recipe for this is
FROM language-base-image # python:3.9, node:14, ...
WORKDIR /app
COPY dependencies-file . # requirements.txt, package.json, ...
RUN install the dependencies # pip install, npm install, ...
COPY . .
RUN build the application # npm run build, ...
CMD ./the_application # npm run start, ...
You should then be able to docker build an image, and docker run a container from the resulting image. The Docker documentation includes a sample application that runs through this sequence.
Note in particular that the problem task of "read a text file" is substantially harder in Docker than without. You need to use a bind mount to give access to the host filesystem to the container, and then refer to the container-side path. For example,
docker run --rm -v $PWD/data:/data my-image \
./the_application --input /data/file.txt
I would not bother trying to use Docker as my primary development environment, especially for an introductory project. Docker is designed as an isolation system and it's intentionally tricky to work with host files from a container, and vice versa. Especially if you can use a fairly routine programming language that you can easily install with apt-get or brew, and you don't have tricky host-library dependencies, it's substantially easier to do most of your development in an ordinary host build environment use Docker only at a late stage.
Related
I have to deal with some very large vendor support packages for embedded development —- I’ve used docker successfully just as a means of keeping their installs segmented away from the rest of my system and for the sake of environment reproducibility. That works great, but often these installs are monoliths, including a ton of files and functionality I don’t need, especially in a CI environment. And moving giant, slow-to-recreate docker images around is a pain.
So, in the interest of teasing out just the features I need, and porting them to a much smaller image, I’m wondering:
Can I run a docker image, performing some CI-relevant task, and then find all the files that were accessed in the duration the docker image was running?
The plan after that would be to copy all those files into a tarfile or similar, then use that for specialized images in the future. So as an alternative question... is that plan worth pursuit?
Thanks :) -Chloë
Maybe it will not answer exactly to your question, however it may help.
You can check what is happening in the container by
checking its logs through the docker container logs command.
checking the modification performed in its filesystem through the docker diff command.
Here is an example
# run a ubuntu container
$ docker run -it --rm --name focal ubuntu:focal
# run a command in the container
$ echo "test" > test.txt
# messages in the logs
$ docker container logs --follow --details focal
# root#aa86b4988bfe:/# echo "test" > test.txt
# checking the differences
$ docker diff focal
# A /test.txt
I am still relatively new to docker. I have two git repos. One is a Next.js application and the other a nodejs app. I need to create a docker container but when building, I need to build the next.js code and move the build folder to the node app before creating an image. Not sure if this is possible.
I am not sure if this is the best route to take either. The end goal is to push the docker containers to AWS ECS.
Background, the next.js is a server rendered react framework. So in QA and PROD the node app serves the content.
By issuing RUN directives, executed commands are committed in new layers on top of the current image. The concept behind Docker is to keep your convergence steps under source control as so containers can be created from any point in time in an image's history. In this scenario, using a set of RUN instructions will commit each step as an individual layer.
WORKDIR /project
RUN git clone https://github.com/foo/next-js.git
RUN git clone https://github.com/baz/nodejs.git
WORKDIR /project/next-js
RUN npm run build
RUN cp ./build ../nodejs/
You could instead incorperate the above into a bash script and bypass Docker's layering mechanism.
COPY ./setup.sh /
RUN chmod u+x /setup.sh
RUN /setup.sh
However, doing so would defeat the purpose of using Docker to begin with, unless for some reason you need to ensure your container receives a set of instructions in the form of one layer.
So this is not about seeking workarounds to -v.
I have a Dockerfile whose intent is to install a cross-compiler in /usr/local/<cross-compiler-path>, inside the container. Later during a build process, a file would be mounted to this cross-compiler, like this:
root#5bee5daf8165:/# mount <blah.img.gz> /usr/local/<cross-compiler-path>
I get mount: /usr/local/<cross-compiler-path>: mount failed: Operation not permitted.
Although if I skip this step, finish build, run a --privileged container and mount, it works fine.
I understand the reason for not giving privileged mode in the build since it breaks the 'portability' of containers as they depend on host volumes. But in my case, I am attempting to mount it inside the Container's own file system. Why is that not allowed?
For the record, I tried installing the cross-compiler on a different path, like this:
root#5bee5daf8165:/# mount <blah.img.gz> /home/<cross-compiler-path>
But that doesn't work either. I want to attempt the build inside the Dockerfile and discard the build cache which bloat up my container once I no longer need them. What options do I have?
As mentioned in "Can You Mount a Volume While Building Your Docker Image to Cache Dependencies?" from Vladislav Supalov
Although there’s no functionality in Docker to have volumes at build-time, you can use multi-stage builds, benefit from Docker caching and save time by copying data from other images - be it multi-stage or tagged ones.
When building an image, you can’t mount a volume. However, you can copy (COPY) data from another image! By combining this, with a multi-stage build, you can pre-compute an expensive operation once, and re-use the resulting state as a starting point for future iterations.
Example:
FROM ubuntu as intermediate
RUN apt-get install -yqq python-dev python-virtualenv
RUN virtualenv /venv/
RUN mkdir -p /src
# those don't change often
ADD code/basic-requirements.txt /src/basic-requirements.txt
RUN /venv/bin/pip install -r /src/basic-requirements.txt
FROM ubuntu
RUN apt-get install -yqq python-dev python-virtualenv
# the data comes from the above container
COPY --from=intermediate /venv /venv
ADD code/requirements.txt /src/requirements.txt
# this command, starts from an almost-finished state every time
RUN /venv/bin/pip install -r /app/requirements.txt
The OP add in the comments:
I want to mount a volume internally to the container fs using the mount command while build, which currently doesn't work.
Just wanted to know if 'mount' operation, in general is tied to the kernel?
Kernel or not, using mount directly (outside of the sanctioned volumes) is not allowed for security reason, as described here by BMitch.
Docker removes the mount privilege from containers because using this you could mount the host filesystem and escape the container.
If you really need to mount something during the build process, you might consider buildah, which can build without running a container for each layer (like docker build does), and can do so without being root.
Use ONBUILD to read your existing Dockerfile.
Note that with "buildah mount, you can do the reverse: Mounts the specified container's root file system in a location which can be accessed from the host, and returns its location.
That is another alternative.
I'm pretty new to the bash/shell script world, I'm trying to do the below and it could be pretty simple but I wasn't able to figure out the command, would be great if someone could help me out here and also point me to some documentation wrt to shell script topics. Thank you in advance.
My build.sh and Dockerfile resides under a folder called .settings and this folder lives directly under the app root. Now inside my build.sh and Dockerfile when I refer something like $(pwd) or COPY . /apps/ it might not work since my build.sh and Dockerfile does not live directly under the app root.
What command I can use in this scenario inside the files that I referenced above. Hope I made it clear. Once again this could be very simple since I'm a newbie in this arena I find it a little difficult.
inside build.sh, reference to $(pwd)
docker run \
-u root \
--rm \
-v $(pwd):/app \ ----> this $(pwd) references the application root, but if I
move this build.sh inside a folder called .settings then the $(pwd) context
would change and I still want to refer it to the root.
<MYIMAGE NAME FROM LOCAL REPO>
The last arg to docker build, often something like docker build . is the build context in docker. This directory is sent to the server where the build runs and all COPY and ADD commands are performed using this context. These commands do not run on the client, and docker is a client/server application, so anything not in that context simply doesn't exist for the purpose of building an image.
So in the above example, docker build . the current directory is the build context and if that's run while you're inside of the .settings directory, only those files are part of the build context. Therefore your build.sh script needs to pass a different directory, and also reference where the Dockerfile is inside of that build context. That would look like:
docker build -f .settings/Dockerfile ..
When you do this, all of the COPY and ADD commands will now be relevant to parent directory, so you may need to adjust your Dockerfile to compensate.
For your $(pwd) reference, you can either cd .. before running your docker run command or update the command to look like:
docker run \
-u root \
--rm \
-v $(pwd)/..:/app \
<your image>
My project is setup like this:
./ -
Dockerfile
package.json
build
compiled files from frontend and backend directories get put here
backend
app.js
frontend
frontend files...
scripts
startServer.sh
build.sh
startServer.sh:
docker build ../ --tag myImage
# The build script compiles all my assets and places
# them in the top level 'build' directory which i am
# trying to link to my docker image so I can recompile
# on each file change and have the changes show in the docker image.
./build.sh
docker run --volume /path/to/build/dir:/src/app myImage
Dockerfile:
FROM node:4.4.7
RUN ls src/app
The RUN command in the Dockerfile gives me this error when the build command from the startServer script is called:
ls: cannot access src/app: No such file or directory
If I change RUN to CMD it gives no error. Also, even after the build gives that error, it finishes the build and the docker run command gives no error.
Is the 'docker build' command actually trying to add the 'build' folder to the image from which containers are launched? Or is it just compiling some commands for the images to use when they are made?
If it is the later, how do you make one Dockerfile that is used for both building and running that works in both cases?
I feel like I might be missing a crucial concept with Docker, but I've gone through the tutorials and docs and couldn't solve this.
There is no src/app folder in the node image, so this is an expected error. The node image expects you to add your own /usr/src/app, either with a COPY step in your build, or with a volume mapping after the build is finished.
The RUN gives a step to run to add a layer to the resulting built image, so an ls makes little sense there since you didn't modify the image with new content.
The CMD gives a default command to run if one is not passed at the end of the docker run, so if you do a docker run node /bin/bash, the ls src/app CMD will never be run. This also runs after other steps in your build, and after any volume mounts you may be running on your container, which would create this folder.
When you run the docker image, you mount data volume to /src/app
But in docker script, you tried to access src/app
Because the default working directory is not a root. you cannot access src directory.
so, edit your docker file to
FROM node:4.4.7
RUN ls /src/app