I'm trying to create a temporary folder and then set the path as an environment variable for use in later Dockerfile instructions:
FROM alpine
RUN export TEMPLATES_DIR=$(mktemp -d)
ENV TEMPLATES_DIR=$TEMPLATES_DIR
RUN echo $TEMPLATES_DIR
Above is what I've tried, any idea how I can achieve this?
Anything you run in a Dockerfile will be persisted forever in the resulting Docker image. As a general statement, you don't need to use environment variables to specify filesystem paths, and there's not much point in creating "temporary" paths. Just pick a path; it doesn't even need to be a "normal" Linux path since the container filesystem is isolated.
RUN mkdir /templates
It's common enough for programs to use environment variables for configuration (this is a key part of the "12-factor" design) and so you can set the environment variable to the fixed path too
ENV TEMPLATES_DIR=/templates
In the sequence you show, every RUN step creates a new container with a new shell, and so any environment variables you set in a RUN command get lost at the end of that step. You can't set a persistent environment variable in quite the way you're describing; Create dynamic environment variables at build time in Docker discusses this further.
If it's actually a temporary directory, and you're intending to clean it up, there are two more possibilities. One is to do all of the work you need inside a single RUN step that runs multiple commands. The environment variable won't outlive that RUN step, but it will be accessible within it.
RUN export TEMPLATES_DIR=$(mktemp -d) \
&& echo "$TEMPLATES_DIR" \
&& rm -rf "$TEMPLATES_DIR"
A second is to use a multi-stage build to do your "temporary" work in one image, but then copy the "permanent" parts of it out of that image into the final image you're actually shipping.
Related
I have an Alpine container that I copy a binary to (in this case it is spar). The entry point is dumb-init /usr/bin/spar but it results in a No such file or directory. When I run sh inside of the container, /usr/bin/spar exists. Trying to run it in
dumb-init ...
/usr/bin/spar / spar from /
./spar / spar from usr/bin/
All result in the same error. I tried changing the permissions with chmod 777 /usr/bin/spar giving everyone full access, still no luck.
Am I missing something that is specific to alpine? In another SO issue someone mentioned that switching from Alpine to Ubuntu solved their issue, but no further info was provided.
Here is the dockerfile that creates the image(s)
ARG intermediate=quay.io/wire/alpine-intermediate
ARG deps=quay.io/wire/alpine-deps
#--- Intermediate stage ---
FROM ${intermediate} as intermediate
#--- Minified stage ---
FROM ${deps}
ARG executable
ENV CACHEDEXECUTABLE ${executable}
COPY --from=intermediate /dist/${executable} /usr/bin/${executable}
# TODO: only if executable=brig, also copy templates. Docker image conditionals seem hacky:
# https://stackoverflow.com/questions/31528384/conditional-copy-add-in-dockerfile
# For now, adds ~2 MB of additional files into every container
COPY --from=intermediate /dist/templates/ /usr/share/wire/templates/
# ARGs are not available at runtime, create symlink at build time
# more info: https://stackoverflow.com/questions/40902445/using-variable-interpolation-in-string-in-docker
RUN ln -s /usr/bin/${executable} /usr/bin/service
ENTRYPOINT /usr/bin/dumb-init /usr/bin/${CACHEDEXECUTABLE}
If spar is a binary, that binary exists in the container, and you call it directly or it's in the containers path, then the two likely reasons for a file not found are:
Dynamically linked libraries that don't exist inside the container. E.g. of you run ldd spar it will show you those links, and there's a good chance you'll see libc despite trying to run on Alpine where it uses musl.
The binary is for another platform/architecture, and you have binfmt_misc setup, but without the --fix-binary option so it's looking for the interpreter path in the container filesystem rather than the host. This will be visible by the lack of an F flag in the /proc file for that platform.
I have a simple Program written in Rust.
When I type cargo run in terminal it always shows:
Updating crates.io index...
And this takes around 40 seconds.
But I just wan to execute my Program and I think cargo does not need to update the index every time I run the Program, since this makes testing very slow...
Is there an option to skip that?
I figured it out:
Since I am running cargo in a Docker container, I need to store the cargo cache persistently because it resets every time the container restarts.
There is The Cargo Book that contains all the information you'd ever want to know about cargo. See this for disabling index update.
I've tried to use this feature myself, and here's the command that worked:
cargo +nightly run -Z no-index-update
The +nightly thing is new to me as well, but I find it here.
This answer has been brought up by users thefeiter and Captain Fim but I think a more complete answer could be cool rust/linux newcomers
When we use docker run, the index is updated every time the container is run because the cache is not shared between runs. So to skip the index update, as Captain Fim mentioned, you need to set the CARGO_HOME environment variable on the container. This environment variable should contain the path to a persistent folder. One simple solution is using the docker volumes to share cache between host and container.
In my case, I created at cargo_home folder in my project (could be somewhere else) on my host. I have passed the whole project folder to the container and set the docker env variable of CARGO_HOME to the container path to the cargo_home folder.
The command to build my app looks like this
docker run --rm --user "$(id -u)":"$(id -g)" -e CARGO_HOME=/usr/src/myapp/cargo_home -v "$PWD":/usr/src/myapp -w /usr/src/myapp rust-compiler cargo build
The first time you will run this command, it will take some time, but you should see the cargo_home folder getting filled with files. The next time you run the command, it should use the cargo_home folder as cache. This should be almost instant if your app source code did not change.
I went through one of very few good dockerizing Vue.js tutorials and there is one thing I don't understand why is mandatory in Dockerfile:
# add `/app/node_modules/.bin` to $PATH
ENV PATH /app/node_modules/.bin:$PATH
COPY package.json /usr/src/app/package.json #not sure though how it relates to PATH...
I found only one explanation here which says:
We expose all Node.js binaries to our PATH environment variable and
copy our projects package.json to the app directory. Copying the JSON
file rather than the whole working directory allows us to take
advantage of Docker’s cache layers.
Still, it doesn't made me any smarter. Anyone able to explain it in plain english?
Error prevention
I think this is just a simple method of preventing an error where Docker wasn't able to find the correct executables (or any executables at all). Besides adding another layer to your image, there is in general as far as I know no downside in adding that line to your Dockerfile.
How does it work?
Adding node_modules/bin to the PATH environment variable ensures that the executables created during the npm build or the yarn build processes can be found. You could also COPY your locally builded node_modules folder to the image but it's advised to build it inside the Docker container to ensure all binaries are adapted to the underlying OS running in the container. The best practice would be to use multistage builds.
Furthermore, adding the node_modules/bin at the beginning of the PATH environment variable ensures that exactly these executables (from the node_modules folder) are used instead of any other executables which might also be installed on the system inside the Docker image.
Do I need it?
Short answer: Usually no. It should be optional.
Long answer: It should be enough to set the WORKDIR to the path where the node_modules is located for the issued RUN, CMD or ENTRYPOINT commands in your Dockerfile to find the correct binaries and therefore to successfully get executed. But I for example had a case where Docker wasn't able to find the files (I had a pretty complex setup with a so called devcontainer in VSCode). Adding the line ENV PATH /app/node_modules/.bin:$PATH solved my problem.
So, if you want to increase the stability of your Docker setup in order to make sure that everything works as expected, just add the line.
So I think the benefit of this line is to add the node_modules path from the Docker container to the list of PATHs on the relevant container. If you're on a Mac (or Linux I think) and run:
$ echo $PATH
You should see a list of paths which are used to run global commands from your terminal i.e. gulp, husky, yarn and so on.
The above command will add node_modules path to the list of PATHs in your docker container so that such commands if needed can be run globally inside the container they will work.
.bin (short for 'binaries') is a hidden directory, the period before the bin indicates that it is hidden. This directory contains executable files of your app's modules.
PATH is just a collection of directories/folders that contains executable files.
When you try to do something that requires a specific executable file, the shell looks for it in the collection of directories in PATH.
ENV PATH /app/node_modules/.bin:$PATH adds the .bin directory to this collection, so that when node tries to do something that requires a specific module's executable, it will look for it in the .bin folder.
For each command, like FROM, COPY, RUN, CMD, ..., Docker creates a image with the result of this command, and this images are called as layers. The final image is the result of merge of all layers.
If you use the COPY command to store all the code in one layer, it will be greater than store a environment variable with path of the code.
That's why the cache layers is a benefit.
For more info about layers, take a look at this very good article.
I have two .env files (.env.development and .env.production) and two different build scripts, one for dev and one for prod. Now I want exactly one build script and depending on a global environment variable I want to decide what of this two .env files should be used for the build.
Is there a way to write in a build script a checker on what the environment variable is set ?
So you can solve this problem by passing environmental variables from your unix server in production, but when in development pass it from .env file, this way you don't need to add twi build scripts because it will get variables from .env or from unix env.
To pass env variables to your Node.js app from Unix OS is like
Open Terminal and write the command as
> export MY_ENV=my environment value
After that you will see the env variable with
> echo "$MY_ENV"
But I suggest you to use Docker and set the env variables to your Docker env this way you will separate your env variables from OS env and prevent inconsistencies
I'm trying to build an image in Docker that requires a few secret files to do things like pulling from a private git repo. I've seen a lot of people with code like this:
ADD id_rsa /root/.ssh/id_rsa
RUN chmod 700 /root/.ssh/id_rsa
RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan github.com >> /root/.ssh/known_hosts
RUN git clone git#github.com:some/repo.git /usr/local/some_folder
Although that works, it means I have to store my private id_rsa with my image, which strikes me as a bad idea. What I'd much rather do is keep my secret files in some cloud storage like s3, and just pass in credentials as environment variables to be able to pull everything else down.
I know that I can pass environment variables in at docker run with the -e switch, but if I need some files at build time (like the id_rsa to perform a git clone), what can I do? Ideally I'd be able to pass environment variables to docker build, but that's not possible (I can't understand why).
So, ideas? What's the canonical/correct thing to do here? I can't be the first person with this issue.
I'll start with the easiest part, which I think is a common misconception:
Ideally I'd be able to pass environment variables to docker build, but that's not possible (I can't understand why).
A docker build is meant to be reproducible. Given the same context (the files under the same directory as the Dockerfile) the resulting image is the same. They are also meant to be simple. Both things together explain the absence of environment options or other conditionals.
Now, because the build needs to be reproducible, the execution of each command is cached. If you run the build twice, the git pull will only run the first time.
By your comment, this is not what you intend:
so on any new image build, we always want the newest version of the repo
To trigger a new build you need to either change the context or the Dockerfile.
The canonical way (I'm probably abusing the word, but this is how the automated builds work) is to include the Dockerfile in git.
This allows a simple workflow of git pull ; docker build ... and avoids the problem with storing your git credentials.