I'working on Asp.Bet Core 2 app with MSSQL on Linux. I would like to configure CI and CD to Azure.
I defined docker-compose task in Visual Studio Team Services, but I'm getting an error.
2017-11-09T20:24:09.4725450Z ##[section]Starting: Create images
2017-11-09T20:24:09.4801610Z ==============================================================================
2017-11-09T20:24:09.4817660Z Task : Docker Compose
2017-11-09T20:24:09.4835680Z Description : Build, push or run multi-container Docker applications. Task can be used with Docker or Azure Container registry.
2017-11-09T20:24:09.4852370Z Version : 0.4.7
2017-11-09T20:24:09.4867100Z Author : Microsoft Corporation
2017-11-09T20:24:09.4881820Z Help : [More Information](https://go.microsoft.com/fwlink/?linkid=848006)
2017-11-09T20:24:09.4897520Z ==============================================================================
2017-11-09T20:24:10.5040990Z [command]/usr/local/bin/docker-compose -f /opt/vsts/work/1/s/docker-compose.yml -f /opt/vsts/work/1/s/docker-compose.ci.build.yml -f /home/vsts/agent/.docker-compose.1510259050468.yml -p Travelingowe build
2017-11-09T20:24:10.8628630Z db uses an image, skipping
2017-11-09T20:24:10.8646610Z Building api
2017-11-09T20:24:10.9373040Z Step 1/7 : FROM microsoft/aspnetcore:2.0
2017-11-09T20:24:12.0041790Z 2.0: Pulling from microsoft/aspnetcore
2017-11-09T20:25:03.6328690Z Digest: sha256:e36cb8d1edcd1bfd7aea0412349482a9c1a601089d76d1a294067f5f7f1098a9
2017-11-09T20:25:03.6497680Z Status: Downloaded newer image for microsoft/aspnetcore:2.0
2017-11-09T20:25:03.6524910Z ---> e0e49def2506
2017-11-09T20:25:03.6551560Z Step 2/7 : MAINTAINER Maciej Skuratowski <maciejskuratowski#gmail.com>
2017-11-09T20:25:03.7496070Z ---> Running in e7667861d293
2017-11-09T20:25:03.9222120Z ---> 24ba03484562
2017-11-09T20:25:03.9436840Z Removing intermediate container e7667861d293
2017-11-09T20:25:03.9458600Z Step 3/7 : ARG source
2017-11-09T20:25:04.0562980Z ---> Running in 7b57f9fc515f
2017-11-09T20:25:04.2292420Z ---> 4d07188edf18
2017-11-09T20:25:04.2503550Z Removing intermediate container 7b57f9fc515f
2017-11-09T20:25:04.2581270Z Step 4/7 : WORKDIR /app
2017-11-09T20:25:04.5347440Z ---> 4546b8dc771f
2017-11-09T20:25:04.5527690Z Removing intermediate container 0a595d10e668
2017-11-09T20:25:04.5558170Z Step 5/7 : EXPOSE 80
2017-11-09T20:25:04.6765650Z ---> Running in aa273cc9ca2c
2017-11-09T20:25:04.8330020Z ---> 6dd874db617e
2017-11-09T20:25:04.8619290Z Removing intermediate container aa273cc9ca2c
2017-11-09T20:25:04.8650100Z Step 6/7 : COPY ${source:-obj/Docker/publish} .
2017-11-09T20:25:04.8686070Z Service 'api' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder211771018/obj/Docker/publish: no such file or directory
2017-11-09T20:25:04.9064960Z ##[error]db uses an image, skipping
2017-11-09T20:25:04.9127660Z ##[error]Building api
2017-11-09T20:25:04.9175860Z ##[error]Service 'api' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder211771018/obj/Docker/publish: no such file or directory
2017-11-09T20:25:04.9258880Z ##[error]/usr/local/bin/docker-compose failed with return code: 1
2017-11-09T20:25:04.9325830Z ##[section]Finishing: Create images
Here's my docker-compose:
version: '3'
services:
api:
image: api
container_name: api
build:
context: ./Api
dockerfile: Dockerfile
ports:
- "8000:80"
depends_on:
- db
db:
image: "microsoft/mssql-server-linux"
container_name: mssql
environment:
SA_PASSWORD: "testtest3030!"
ACCEPT_EULA: "Y"
MSSQL_PID: "Developer"
ports:
- "127.0.0.1:8001:1433"
and docker-compose.ci.build.yml file:
version: '3'
services:
ci-build:
image: microsoft/aspnetcore-build:1.0-1.1
volumes:
- .:/src
working_dir: /src
command: /bin/bash -c "dotnet restore ./Travelingowe.sln && dotnet publish ./Travelingowe.sln -c Release -o ./obj/Docker/publish"
Also I attached my VSTS docker-compose taks:
Do you have any idea for I'm doing wrong?
Aha, I faced the same issue a few days ago. Your issue is inside your DockerFile. For example, let's look at my file:
FROM microsoft/aspnetcore:2.0
ARG source
WORKDIR /app
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT [ "dotnet","MyWeb.Api.dll" ]
If docker cannot find the path that you specify in your COPY command, it will produce an error:
Service 'api' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder211771018/obj/Docker/publish: no such file or directory
So, basically, what you need to do is to fix your build release path (you need to be sure that files after the build in the right folder).
Also, there is a problem, if you execute:
docker-compose build
It doesn't build the solution. That is why DockerFile cannot find the needed folder. If you execute the following command before your docker-compose, it will work.
dotnet publish ./YourSolution.sln -c Release -o ./obj/Docker/publish
Regarding the error, the reason is that there isn’t the obj/Docker/publish folder or file. (check the answer of Leonid)
The docker-compose.ci.build.yml file is for building the project and the docker-compose.yml file is used for running the project, so you need to build the project through docker-compose.ci.build.yml file with Dokcer-compose task, then build/run image with docker-compose.yml file with Docker-compose task.
There is a thread about docker-compose.ci.build.yml, docker-compose.yml and Dockerfile: Asp.net core with linux docker container
Update:
Refer to these steps to deploy project to Azure:
Create a new build definition
Add Docker Compose task (Docker Compose File: **/docker-compose.ci.build.yml; Action: Run a Docker Compose command; Command:run ci-build)
Add Copy Files task (Source Folder: $(System.DefaultWorkingDirectory)/[your project path]/obj/Docker/publish; Contents: **; Target Folder: $(Build.ArtifactStagingDirectory))
Add Publish Build Artifacts task (Path to publish: $(Build.ArtifactStagingDirectory))
Create a new release definitioin
Add Azure App Service Deploy task (Package or folder: $(System.DefaultWorkingDirectory)/[artifact build definition name]/[artifact name]; Uncheck Publish using Web Deploy option)
Related
I am trying to build a docker container with private node packages in it. I have followed this guide to use secrets to reference npmrc file securely to install the dependencies. I can get this to work when building the image directly using a command like this: docker build --secret id=npm,src=$HOME/.npmrc . but I cannot get this working with docker compose. When running a docker compose build it acts like there is no npmrc file and gives me a 401 when trying to download dependencies.
I provided a stripped down version of Dockerfile and docker-compose.yml below.
Dockerfile
# syntax = docker/dockerfile:1.2
FROM node:14.17.1
COPY . .
RUN --mount=type=secret,id=npm,target=/root/.npmrc yarn --frozen-lockfile --production
EXPOSE 3000
CMD [ "npm", "start" ]
docker-compose.yml
version: '3.7'
services:
example:
build: packages/example
ports:
- "3000:3000"
secrets:
- npm
secrets:
npm:
file: ${HOME}/.npmrc
The problem appears to be that my docker-compose.yml is specifying secrets for runtime of a container vs build time. Support for build secrets from docker compose has not been implemented yet. Here is the outstanding PR: https://github.com/docker/compose/pull/7046.
For now, I have to build the image using docker build ... and reference the named image locally in docker-compose.yml instead of building through docker compose.
Since docker-compose v2.5.0 this is now possible.
Dockerfile:
# syntax=docker/dockerfile:1.2
RUN --mount=type=secret,id=mysecret,target=/root/mysecret cat /root/mysecret
docker-compose.yml
services:
my-app:
build:
context: .
secrets:
- mysecret
secrets:
mysecret:
file: ~/.npmrc
I'm trying to use docker-compose inside bitbucket pipeline in order to build several microservices and run tests against them. However I'm getting the following error:
Step 19/19 : COPY . .
Service 'app' failed to build: failed to copy files: failed to copy directory: Error processing tar file(exit status 1): Container ID 166535 cannot be mapped to a host ID
As of now, my docker-compose.yml looks like this:
version: '2.3'
services:
app:
build:
context: .
target: dev
ports:
- "3030:3030"
image: myapp:dev
entrypoint: "/docker-entrypoint-dev.sh"
command: [ "npm", "run", "watch" ]
volumes:
- .:/app/
- /app/node_modules
environment:
NODE_ENV: development
PORT: 3030
DATABASE_URL: postgres://postgres:#postgres/mydb
and my Dockerfile is as follow:
# ---- Base ----
#
FROM node:10-slim AS base
ENV PORT 80
ENV HOST 0.0.0.0
EXPOSE 80
WORKDIR /app
COPY ./scripts/docker-entrypoint-dev.sh /
RUN chmod +x /docker-entrypoint-dev.sh
COPY ./scripts/docker-entrypoint.sh /
RUN chmod +x /docker-entrypoint.sh
ENTRYPOINT ["/docker-entrypoint.sh"]
COPY package.json package-lock.json ./
# ---- Dependencies ----
#
FROM base as dependencies
RUN npm cache verify
RUN npm install --production=true
RUN cp -R node_modules node_modules_prod
RUN npm install --production=false
# ---- Development ----
#
FROM dependencies AS dev
ENV NODE_ENV development
COPY . .
# ---- Release ----
#
FROM dependencies AS release
ENV NODE_ENV production
COPY --from=dependencies /app/node_modules_prod ./node_modules
COPY . .
CMD ["npm", "start"]
And in my bitbucket-pipelines.yml I define my pipeline as:
image: node:10.15.3
pipelines:
default:
- step:
name: 'install docker-compose, and run tests'
script:
- curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- chmod +x /usr/local/bin/docker-compose
- docker-compose -v
- docker-compose run app npm run test
- echo 'tests done'
services:
- docker
However, this example works when I try to use docker without docker-compose, defining my pipeline as:
pipelines:
default:
- step:
name: 'install and run tests'
script:
- docker build -t myapp .
- docker run --entrypoint="" myapp npm run test
- echo 'done!'
services:
- postgres
- docker
I found this issue (https://jira.atlassian.com/browse/BCLOUD-17319) in atlassian community, however I could not find a solution to fix my broken usecase. Any suggestions?
I would try to use an image with installed docker-compose already instead of installing it during the pipeline.
image: node:10.15.3
pipelines:
default:
- step:
name: 'run tests'
script:
- docker-compose -v
- docker-compose run app npm run test
- echo 'tests done'
services:
- docker
definitions:
services:
docker:
image: docker/compose:1.25.4
try to add this to your bitbucket-pipelines.yml
if it doesn't work rename docker to customDocker in the definition and in the service sections.
if it doesn't work too, then because you don't need nodejs in the pipeline directly, try to use this approach:
image: docker/compose:1.25.4
pipelines:
default:
- step:
name: 'run tests'
script:
- docker-compose -v
- docker-compose run app npm run test
- echo 'tests done'
services:
- docker
TL;DR: Start from your baseimage and check for the ID that is creating the problem using commands in your dockerfile. Use "problem_id = error_message_id - 100000 - 65536" to find the uid or gid that is not supported. Chown copies the files that are modified inflating your docker image.
The details:
We were using base image tensorflow/tensorflow:2.2.0-gpu and though we tried to find the problem ourselves, we were looking too late in our Dockerfile and making assumptions that were wrong.With help from Atlassian support we found that /usr/local/lib/python3.6 contained many files belonging to group staff (gid = 50)
Assumption 1: Bitbucket pipelines have definitions for the standard "linux" user ids and group ids.
Reality: Bitbucket pipelines only define a subset of the standard users and groups. Specifically they do not define group "staff" with gid 50. Your Dockerfile base image may define group staff (in /etc/groups) but the Bitbucket pipeline is run in a docker container without that gid. DO NOT USE
RUN cat /etc/group && RUN /etc/passwd
to check for ids. Execute these commands as Bitbucket pipeline commands in your script.
Assumption 2: It was something we were installing that was breaking the build.
Reality: Although we could "move the build failure around" by adjusting which packages we installed. This was likely just a case of some packages overwriting the ownership of pre-existing
We were able to find the files by using the relationship between the id in the error message and the docker build id of
problem_id = error_message_id - 100000 - 65536
And used the computed id value (50) to fined the files early in our Dockerfile:
RUN find / -uid 50-ls
RUN find / -gid 50 -ls
For example:
Error processing tar file(exit status 1): Container ID 165586 cannot be mapped to a host ID
50 = 165586 - 100000 - 65536
Final solution (for us):
Adding this command early to our Dockerfile:
RUN chown -R root:root /usr/local/lib/python*
Fixed the Bitbucket pipeline build problem, but also increases the size of our Docker image because Docker makes a copy of all of the files that are modified (contents or filesystem flags). We will look again at multi-stage builds to reduce the size of our docker images.
I'm want to run aspnetcore 2.2 web app in azure, using windows docker containers,
but container fails starting with following message
ERROR - Site: test-container-bonee - Image pull reported error. Image:
containersbonee.azurecr.io/soarcdockertest:1.0. failed to register
layer: re-exec error: exit status 1: output: remove
\?\C:\DockerData\windowsfilter\1889bb89045f68ca9052065efa6ea07b24da7e725d2adb2aacc8294318fecb6c\UtilityVM\Files\Windows\servicing\Packages\Microsoft-UtilityVM-Core-Package~31bf3856ad364e35~amd64~~10.0.14393.0.cat:
The process cannot access the file because it is being used by another
process.
25/01/2019 10:31:06.022 INFO - Site: test-container-bonee -
Image: containersbonee.azurecr.io/soarcdockertest:1.0 Custom Registry:
https://containersbonee.azurecr.io
25/01/2019 10:31:06.025 INFO - Site: test-container-bonee - Purging
after container failed to start
25/01/2019 10:31:06.025 ERROR - Site:
test-container-bonee - Pull image completed but it was not found
locally. Image: containersbonee.azurecr.io/soarcdockertest:1.0.
Errors: failed to register layer: re-exec error: exit status 1:
output: remove
\?\C:\DockerData\windowsfilter\1889bb89045f68ca9052065efa6ea07b24da7e725d2adb2aacc8294318fecb6c\UtilityVM\Files\Windows\servicing\Packages\Microsoft-UtilityVM-Core-Package~31bf3856ad364e35~amd64~~10.0.14393.0.cat:
The process cannot access the file because it is being used by another
process.
Here is my dockerfiler
FROM microsoft/dotnet:2.2-aspnetcore-runtime-nanoserver-sac2016 AS base
WORKDIR /app
EXPOSE 80
FROM microsoft/dotnet:2.2-sdk-nanoserver-sac2016 AS build
WORKDIR /src
COPY ["Soarc.DockerTest/Soarc.DockerTest.csproj", "Soarc.DockerTest/"]
RUN dotnet restore "Soarc.DockerTest/Soarc.DockerTest.csproj"
COPY . .
WORKDIR "/src/Soarc.DockerTest"
RUN dotnet build "Soarc.DockerTest.csproj" -c Release -o /app
FROM build AS publish
RUN dotnet publish "Soarc.DockerTest.csproj" -c Release -o /app
FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "Soarc.DockerTest.dll"]
Any ideas how to fix this?
I've created a fairly standard asp.net core app which I'm hosting inside Docker. This runs locally and now I'm attempting to use VSTS to build and deploy to Azure.
The build step is failing with the message:
Service 'coredockerapi' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder145772260/obj/Docker/publish: no such file or directory
Here is the full log output from VSTS:
2017-11-18T11:52:07.4647720Z ##[section]Starting: Build Repo and create container
2017-11-18T11:52:07.5468610Z ==============================================================================
2017-11-18T11:52:07.5514890Z Task : Docker Compose
2017-11-18T11:52:07.5550720Z Description : Build, push or run multi-container Docker applications. Task can be used with Docker or Azure Container registry.
2017-11-18T11:52:07.5606920Z Version : 0.4.7
2017-11-18T11:52:07.5628440Z Author : Microsoft Corporation
2017-11-18T11:52:07.5655390Z Help : [More Information](https://go.microsoft.com/fwlink/?linkid=848006)
2017-11-18T11:52:07.5678480Z ==============================================================================
2017-11-18T11:52:09.5543380Z 50f571c1-89b9-4b2f-8ea8-097b0966e534 exists true
2017-11-18T11:52:10.4809920Z [command]/usr/local/bin/docker-compose -f /opt/vsts/work/1/s/DemoCoreDocker/docker-compose.yml -p DemoDocker run --rm -T coredockerapi
2017-11-18T11:52:10.8401740Z Creating network "demodocker_default" with the default driver
2017-11-18T11:52:10.9847570Z Building coredockerapi
2017-11-18T11:52:11.0663980Z Step 1/6 : FROM microsoft/aspnetcore:2.0
2017-11-18T11:52:12.3621450Z 2.0: Pulling from microsoft/aspnetcore
2017-11-18T11:52:27.7530280Z Digest: sha256:b5e62c20e77b287b014fc0182f0e21dc1bba7aa0a42ac40338e303fdb5b49638
2017-11-18T11:52:27.7774720Z Status: Downloaded newer image for microsoft/aspnetcore:2.0
2017-11-18T11:52:27.7814270Z ---> 757f574feed9
2017-11-18T11:52:27.7883450Z Step 2/6 : ARG source
2017-11-18T11:52:27.8732530Z ---> Running in bc9b4aa5e33b
2017-11-18T11:52:28.0333270Z ---> 612c57db7df1
2017-11-18T11:52:28.0598310Z Removing intermediate container bc9b4aa5e33b
2017-11-18T11:52:28.0636200Z Step 3/6 : WORKDIR /app
2017-11-18T11:52:28.3684690Z ---> fc4ab62cecd6
2017-11-18T11:52:28.3904360Z Removing intermediate container baedca1c9c5d
2017-11-18T11:52:28.3935980Z Step 4/6 : EXPOSE 80
2017-11-18T11:52:28.5164070Z ---> Running in c8bbbf929c3a
2017-11-18T11:52:28.6861730Z ---> 578c84507512
2017-11-18T11:52:28.7087730Z Removing intermediate container c8bbbf929c3a
2017-11-18T11:52:28.7172780Z Step 5/6 : COPY ${source:-obj/Docker/publish} .
2017-11-18T11:52:28.7256500Z Service 'coredockerapi' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder145772260/obj/Docker/publish: no such file or directory
2017-11-18T11:52:28.7798470Z ##[error]Creating network "demodocker_default" with the default driver
2017-11-18T11:52:28.7901050Z ##[error]Building coredockerapi
2017-11-18T11:52:28.8022370Z ##[error]Service 'coredockerapi' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder145772260/obj/Docker/publish: no such file or directory
2017-11-18T11:52:28.8334530Z [command]/usr/local/bin/docker-compose -f /opt/vsts/work/1/s/DemoCoreDocker/docker-compose.yml -p DemoDocker down
2017-11-18T11:52:29.2414990Z Removing network demodocker_default
2017-11-18T11:52:29.4374050Z ##[error]/usr/local/bin/docker-compose failed with return code: 1
2017-11-18T11:52:29.5268820Z ##[section]Finishing: Build Repo and create container
Here is my docker-compose.yml
version: '3'
services:
coredockerapi:
image: coredockerapi
build:
context: ./CoreDockerAPI
dockerfile: Dockerfile
Thank you.
if somebody still have issue when running compose build or up - with error message:
Service 'php' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder...
Issues in my side is due at .dockerignore where folder to copy was referenced. Just remove line of folder need to copy.
Refer to these steps to deploy project to Azure:
Create a new build definition
Add Docker Compose task (Docker Compose File: **/docker-compose.ci.build.yml; Action: Run a Docker Compose command; Command:run ci-build)
Add Copy Files task (Source Folder: $(System.DefaultWorkingDirectory)/[your project path]/obj/Docker/publish; Contents: **; Target Folder: $(Build.ArtifactStagingDirectory))
Add Publish Build Artifacts task (Path to publish: $(Build.ArtifactStagingDirectory))
Create a new release definitioin
Add Azure App Service Deploy task (Package or folder: $(System.DefaultWorkingDirectory)/[artifact build definition name]/[artifact name]; Uncheck Publish using Web Deploy option)
On the other hand, there is a related thread that may benefit you: Docker compose VSTS task failder
I'm using docker compose to create and run several services. One of the services is a nodeJS project in which I added a Makefile to run some custom build tasks.
That's how I define the service in docker-compose.yml file
web:
container_name: web
image: myApp/web
build:
context: .
dockerfile: operations/.docker/web/${APP_ENV}.dockerfile
ports:
- "8080"
volumes:
- ./src/js/web:/var/www/myApp/
working_dir: /var/www/myApp
env_file:
- operations/env/web/${APP_ENV}.env
networks:
- myApp_network
As you can see there is a volume with an alias to my local filesystem. The Makefile is under the ./src/js/web directory.
And this is the dockerfile
FROM node:latest
WORKDIR /var/www/myApp
RUN npm i
RUN make -f /var/www/myApp/Makefile build
RUN mkdir -p /var/log/pm2
EXPOSE 8080
ENTRYPOINT ["pm2", "start", "dist/server/bin/www.js","--name","web","--log","/var/log/pm2/pm2.log","--watch","--no-daemon"]
I had the same setup on my mac book and I manage to build and run the services.
However, I recently tried to run docker compose on linux mint and I get the following error.
make: /var/www/myApp/Makefile: No such file or directory
make: *** No rule to make target '/var/www/myApp/Makefile'. Stop.
ERROR: Service 'web' failed to build: The command '/bin/sh -c make -f /var/www/myApp/Makefile build' returned a non-zero code: 2
I also tried to tried to build the image by removing RUN make -f /var/www/myApp/Makefile build and it worked fine. When I entered the container I could see the Makefile sitting in /var/www/myApp.