In a bitbucket-pipelines.yml BitBucket Pipeline file I am trying to publish a DotNet Core solution, zip it into the correct form to be understood by AWS and then upload it S3.
My build is based on the image microsoft/dotnet:sdk.
image: microsoft/dotnet:sdk
pipelines:
default:
- step:
caches:
- dotnetcore
script:
- dotnet restore
- dotnet publish MyProj/MyProj.csproj -o ../output
- 7z a output.zip .\output\*
- 7z a MyPackage.zip service.zip aws-windows-deployment-manifest.json
This step fails on the first 7z command because 7Zip isn't installed. What is the best way from the Windows command line to zip these files? Alternatively, is there a different Docker image I should be using?
I'm using Amazon.Lambda.Tools to deploy and had a similar issue where I needed to install zip - you could use zip to do it, or install 7z and use that - just need a couple of extra commands to apt-get
If you use a deployment step you'll also get CI/CD metrics and visuals in BitBucket (this is my config)
image: microsoft/dotnet:sdk
pipelines:
default:
- step:
caches:
- dotnetcore
script:
- dotnet restore
- dotnet build
- dotnet test
- step:
deployment: test
script:
- dotnet tool install -g Amazon.Lambda.Tools
- export PATH="$PATH:/root/.dotnet/tools"
- apt-get update
- apt-get install zip -y # or install 7z instead
- dotnet lambda deploy-serverless --region $...... # or manually upload to S3
Related
I have a simple step with Gitlab CI to install Terraform on an Alpine based image:
step_1:
stage: stage_x
image: alpine/doctl:latest
script:
- wget https://releases.hashicorp.com/terraform/1.2.8/terraform_1.2.8_linux_amd64.zip
- unzip terraform_1.2.8_linux_amd64.zip
- mv terraform /usr/bin/terraform
- terraform version
When executed, I have this error from the unzip command:
unzip: 'terraform' exists but is not a regular file
I tried this command over the same image in my machine and it works fine.
Any ideas?
One way to overcome your issue is to use the -d option of unzip in order to unzip Terraform in the folder /usr/bin directly:
-d DIR Extract into DIR
Source: unzip --help
This way, you can also drop the mv line totally.
So, your step becomes:
step_1:
stage: stage_x
image: alpine/doctl:latest
script:
- wget https://releases.hashicorp.com/terraform/1.2.8/terraform_1.2.8_linux_amd64.zip
- unzip terraform_1.2.8_linux_amd64.zip -d /usr/bin
- terraform version
I have a project that requires npm and gradle for build, and docker for building and pushing the image.
At first I thought that I should create my own ubuntu image with gradle and npm setup, but I found out that is not what docker images are for.
So I hoped to run official Gradle and node images as a service so that my script can call those commands, but that is not happening for some reason.
My .gitlab-ci.yml:
variables:
IMAGE_NAME: my.registry.production/project
IMAGE_TAG: $CI_COMMIT_BRANCH
GIT_SUBMODULE_STRATEGY: recursive
stages:
- build
- deploy
build_project:
stage: build
image: ubuntu:jammy
services:
- name: node:12.20
alias: npm
- name: gradle:6.3.0-jre8
alias: gradle
before_script:
- git submodule init && git submodule update --remote --recursive
script:
- cd project-server && npm install && gradle clean build -Pprod -Pwar -x test -x integrationTest
deploy_image:
stage: deploy
image: docker:20.10.17
services:
- name: docker:20.10.17-dind
alias: docker
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
DOCKER_DRIVER: overlay2
script:
- docker login -u $REGISTRY_USER -p $REGISTRY_PASSWORD my.registry.production
- docker build -t $IMAGE_NAME:$IMAGE_TAG .
- docker push $IMAGE_NAME:$IMAGE_TAG
If anyone has any info on how to solve this I would greatly appreciate it, since I’m a novice DevOps.
Edit 1:
My Dockerfile for custom image with Gradle and Node installed.
FROM ubuntu:jammy
LABEL key=DevOps
SHELL ["/bin/bash", "--login", "-i", "-c"]
RUN apt update && apt upgrade -y && apt install curl -y
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
RUN source /root/.bashrc && nvm install 12.14.1
RUN nvm install 12.20.0
RUN apt install zip unzip
RUN curl -s "https://get.sdkman.io" | bash
RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
RUN sdk install java 8.0.302-open
RUN sdk install gradle 3.4.1
SHELL ["/bin/bash", "--login", "-c"]
CMD [ "bin/bash" ]
After I run it, it says that npm is not found in $PATH, I tried Java, Gradle as well but they weren't found in the path as well.
I don't know why since I installed them as you can tell from the Dockerfile.
As I know, a docker image is equal to one build. So if you have multiple services you need to build each one into docker image then you can encapsulate all images into docker-compose.yml file.
I think you can do the following:
Build the npm project into a docker image
Build the Gradle project into a docker image
Write the docker-compose.yml file and put both images.
Once you have done it, the pipeline calls the docker-compose.yml file.
I hope this will be helpful.
Consider a few suggestions based on the fundamental concepts about the deployment in your CI/CD pipeline:
Remove the services keyword. Reference GitLab's official documents on what the services keyword inside gitlab-ci.yaml file is not for. The feature is used
to provide network accessable services to your job runtime (like
a database): https://docs.gitlab.com/ee/ci/services/index.html
Your project uses npm as a dependency management system, Gradle is
a build tool. Both of these pieces of software are more than
appropriate to run on the host operating system of the container
runtime inside GitLab's Pipeline job. You need these tools to assemble some build artifact as a result of the job on the same host your code has been downloaded on in the Runner.
Think about the overall size of the base image in your build_project job and consider how time to download the image over the network on to the Runner will impact your job and overall pipeline duration. If performance can be improved by baking build dependencies into a custom Dockerfile do this. If your image is too large, instead use shell commands inside the script keyword block to download them at the runtime of the job. There can be pros and cons for both.
Break shell scripts to one command per line for easier troubleshooting of failures in your scripts. You will be able to see the line number of the command which returned a non-zero exit code in your job logs:
...
script:
- cd project-server
- npm install
- gradle clean build -Pprod -Pwar -x test -x integrationTest
...
It's recommended to use the Gradle wrapper (gradlew) most of the time instead of the gradle executable directly. Configure this within your project and check the configuration files for the wrapper into your version control system and this will simplify your build dependency: https://docs.gradle.org/current/userguide/gradle_wrapper.html
I'm new to Continous Intregration and recently I setup my first project in CircleCI.
Unfortunately I seems like it's not completely working as expected.
I want to deploy my application to Firebase (Hosting and Functions).
Of course I added Environment Variables to the project in CircleCI.
But Firebase Functions doesn't access my Environment Variables so it's running into errors.
In the Functions folder I created a new nodejs application incl. the dotenv package and I'm calling the variables with process.env.CIRCLECI_VARIABLE.
Would be great if someone could give me a hint what's missing.
config.yml
version: 2.1
jobs:
build:
docker:
- image: circleci/node:10
steps:
- checkout
- run:
name: Install packages
command: yarn install
- run:
name: Build project
command: yarn build
- run:
name: Install functions packages
command: cd ./functions && yarn install
deploy:
docker:
- image: circleci/node:10
steps:
- checkout
- run:
name: Install packages
command: yarn install
- run:
name: Build project
command: yarn build
- run:
name: Install functions packages
command: cd ./functions && yarn install
- run:
name: Installing Firebase-Tools
command: yarn add firebase-tools
- run:
name: Firebase Deploy
command: ./node_modules/.bin/firebase deploy --token "$FIREBASE_TOKEN"
workflows:
build_and_deploy:
jobs:
- build
- deploy:
requires:
- build
filters:
branches:
only: master
I've found the solution
I didn't know that I have to add the Environment Variables to the Google Cloud Function.
Now everything is working as expected
I tried to set up an Azure Build Pipeline that uses Bazel (0.26.0)
My pipeline YAML definition file looks like this:
trigger:
- master
pool:
vmImage: 'windows-2019'
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- script: |
bazel version
echo Add other tasks to build, test, and deploy your project.
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
Currently, I try only to find out which Bazel version is installed by calling bazel version - but Azure DevOps reports:
'bazel' is not recognized as an internal or external command,
operable program or batch file.
Cmd.exe exited with code '9009'.
I wonder how I can install and run Bazel in a Azure pipeline - any hints on this?
It seems that this project got it working. But I do not understand how.
Update in case anyone comes across this thread as of 2022:
Bazel is now installed on Microsoft-hosted agents for macOS-latest, ubuntu-latest, and windows-latest.
Try running this pipeline; you'll be able to see which version of Bazel is installed on each OS.
strategy:
matrix:
linux:
imageName: ubuntu-latest
mac:
imageName: macOS-latest
windows:
imageName: windows-latest
pool:
vmImage: $(imageName)
steps:
- script: bazel version
displayName: Show bazel version
You got this error because you use Microsoft-hosted agent, in those agents bazel is not installed. In the example you provided they use Self Hosted (Private) Agent and they install bazel in their agent machine.
1) Install Self Hosted Agent in your private machine and install bazel in the machine.
2) Install bazel during the build pipeline with choco (simple script task):
choco install bazel
After you install it you can use it.
P.S I tried to install via choco and I got an error but bazel indeed installed and in the next step bazel version gave results, so in the installation task put continueOnError = true. (the error is in the python step, if your project not with python it's ok).
This is how I install Bazel and Azure Pipeline on a windows-2019 image:
steps:
- script: |
echo 'Install Bazel via Choco'
choco install bazel
displayName: 'Install dependencies'
- script: |
bazel version
displayName: 'Show bazel version'
In the meantime, Bazel and Bazelisk come already preinstalled. Seems Bazel takes over the world.
Tested and works with: macOS-10.15, macOS-11, macOS-12, ubuntu-20.04, ubuntu-22.04, windows-2019, windows-2022
Just mount a Gitlab in digitalocean to keep track of versions of some projects, but now I've read a little about Gitlab I wonder if you can set Gitlab CI so that each time you do a commit automatically make a build of application and if the build is successful can do a deploy to OpenShift.
I think my .gitlab-ci.yml should look something like this:
stages:
- build
- deploy
before_script:
- npm install
job_build:
stage: build
script:
- grunt build
job_deploy:
stage: deploy
But I really do not know if this is as valid and neither tell Gitlab CI must only make a git push to OpenShift repository.
After much reading and searching finally found documentation about this [1], in the end I have resolved some file using the following .gitlab-ci.yml
stages:
- build
- deploy
job_build:
stage: build
script:
- npm install -g grunt-cli
- npm rebuild node-sass
- npm install
- grunt build
job_deploy:
stage: deploy
script:
- apt-get update -yq
- apt-get install -y ruby-dev rubygems
- gem install dpl
- dpl --provider=openshift --user=$OPENSHIFT_USER --password=$OPENSHIFT_PASS --domain=mydomain --app=example
only:
- master
The magic happens with a Travis library call dpl [2] that supports a lot of providers [3]
[1]http://doc.gitlab.com/ce/ci/deployment/README.html
[2]https://github.com/travis-ci/dpl
[3]https://github.com/travis-ci/dpl#supported-providers