Self-hosting gitlab autodeploy to aws ec2 server - gitlab

I'm asking this because because I cannot find a running example similar to my case. I have this self hosted Gitlab in a AWS EC2 machine (let's call this "machine 1" and I want to set autodeployment to my AWS EC2 remote server , called "machine 2".
My Gitlabs installation shows (machine 1):
gitlab-ce 10.4.4
gitlab-config-template 10.4.4
gitlab-cookbooks 10.4.4
gitlab-ctl 10.4.4
gitlab-healthcheck
gitlab-monitor
gitlab-pages
gitlab-psql
gitlab-rails
gitlab-scripts
gitlab-selinux
gitlab-shell
gitlab-workhorse
I have follow gitlab instructions to setting up CI & CD on gitlab documentation in my project I want to set autodeployment. The follow steps are follow:
1.I have create runner following gitlabs doc , not much to show here except (machine 2):
url: https://url.to.my.compute.amazonaws.com
Token : token given by gitlab
Executor: shell
Tags: build deploy qa stage
2.I have created my .gitlab-ci.yml (in root project) file with (even with two yml file version created i have tried):
yml 2:
stages:
- build
- deploy
build:
stage: build
script: echo "Building the app"
deploy_staging:
stage: deploy
script:
- echo "Deploy to staging server"
yml 1:
#develop stage
deploy:
stage: deploy
before_script:
#generate ssh key
- mkdir -p ~/.ssh
- echo -e "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
script:
- bash .gitlab-deploy.sh
environment:
name: develop
url: https://my.domain.com
when: manual
3.I have set two SECRET VARIABLES
SSH_PRIVATE_KEY and DEPLOY_SERVERS (with secret key and ips respectively)
4.I have add a deploy.sh file (in root of my project)
#!/bin/bash
#Get servers list
set -f
string=$DEPLOY_SERVERS
array=(${string//,/ })
#Iterate servers for deploy and pull last commit
for i in "${!array[#]}"do
echo "Deploy project on server ${array[i]}"
ssh ubuntu#${array[i]} "cd /var/www/html/app && git pull origin develop"
done
My gitlab-runner shows me at this momment:
gitlab-runner verify
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
And Running as sudo as says it show my runner :
Verifying runner... is alive runner=
Verifying runner... is alive runner=
Verifying runner... is alive runner=
but still in gitlabs ui is getting a "STUCK" tags and the job tell me to "job is stuck, check runners"
Questions:
Are this all the steps to follow?
Do you see anything (or process) I miss in all this configuration?
In my gitlab remote I have "master" permissions, is this what i need to run a runner ?
how can i debug at this point ( i'm using gitlab-runner --debug verify ) is that all I can?
Thanks in advance for you help.

When runner is "specific" , stages need a "tag" like:
stages:
- build
- deploy
build:
stage: build
script: echo "Building the app"
deploy_staging:
stage: deploy
script:
- echo "Deploy to staging server"
tags:
- deploy

Related

How can i create a file on gitlab-runner server by .gitlab-ci.yml file?

I am learning GitLab CI/CD. I use https://gitlab.com/ as repository and I installed an Ubuntu server on virtual machine as local and Gitlab-runner is running on it.
I want to create a file by .gitlab-ci.yml file by touch test.txt on Gitlab-runner server (deploy server).
For example:
build-job:
stage: build
script:
- pwd
- touch /srv/test.txt
- ls /srv/
This was run on gitlab.com, not deploy server. All config tested and worked properly.

Use pm2 with CircleCI

I am using pm2 on my remote ubuntu server and CircleCI for CI, I've got the following configuration files:
version: 2.1
orbs:
node: circleci/node#1.1.6
jobs:
deploy-prod:
docker:
# specify the version you desire here (you might not want node)
- image: circleci/node:7.10
steps:
- checkout
- run: ssh -oStrictHostKeyChecking=no -v $DROPLET_USER#$DROPLET_IP ./deploy_project.sh $MICROSERVICE_NAME
workflows:
build-and-test:
jobs:
- deploy-prod:
filters:
branches:
only:
- master
In my deploy script I do the following:
cd /var/www/nodejs/$1
git pull git#github.com:DevandScorp/hippocrates_authorizationmicroservice.git
cd ..
pm2 restart ecosystem.config.js --only $1
But I've got the following error:
./deploy_project.sh: line 4: pm2: command not found
Is it possible to run my server's pm2 in CircleCI config or can I reload my microservice automatically in another way?
So, if you want to make anything on your server using CircleCI, it's just a waste of time. CircleCI provides a virtual environment, where you can, for example, make some tests. Also you can push changes on your remote server, but CircleCI will not have any access to your server's system. So if we speak about pm2, you can enable watch mode and relaunch your microservice everytime CircleCI push changes to it
Add your deploy SSH keys to Github (or whatever your remote source control is), link it to CircleCI to allow Circle CI to SSH into your remote server and have it run pm2 there.
Adding your SSH Keys to Github
You'd follow similar steps for Bitbucket, etc
https://github.com/settings/ssh/new
Add your SSH fingerprint to your CI steps:
- add_ssh_keys:
fingerprints:
- "bb:bb:bb:bb:bb:bb:bb:bb:bb:bb:bb:bb:bb:bb:bb:bb"
CI SSHes into your remote server and runs pm2:
steps:
- checkout
- run:
name: Deploy over SSH
command: ssh -p your_port_number your_user#your_host "cd ../path/to/your/project; git pull; pm2 start hello_sts";
I followed this guide: https://scribe.rip/#blueish/how-to-setup-ci-cd-with-circleci-and-deploy-your-nodejs-project-to-a-remote-server-275a31d1f6c4
although it's for Bitbucket, not Github.

Gitlab runner job is stalling on cucumber test failure

I am new to gitlab runner and trying to setup acceptance_stage after unit test passes.
The tests run on a windows machine when the job is triggered.
The job exits when tests pass, but the job stalls when test fails. I have only 1 test right now and I am planning to run multiple tests in parallel when this issue is fixed.
The test is triggered using maven command. (I have tweaked the test in such a way so that it fails.)The test fails on this command :
mvn verify -P rest-test-run -Denvironment="qa" -Dwebdriver.chrome.driver="C:\tools\chromedriver.exe"
and the job stalls and is unable to execute the next command.
Here is the sample yaml file that i am using to trigger acceptance stage:
stages:
- test
acceptance_tests:
stage: test
tags:
- playground
only:
- ci_cd_test
before_script:
- export POM_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
- cd ..
- git clone "git#gitlab.<project>" "<automation project name>" 2> /dev/null || (cd "<automation project name>" ; git pull)
- cd <webapp project name>
- export DB_URL=jdbc url
- export DB_USER=username
- export DB_PASSWORD=password
variables:
GIT_STRATEGY: none
script:
# flyway migrates
- java -Dfile.encoding=UTF-8 -jar <webapp project name>/target/<webapp project name>-$POM_VERSION.jar flyway migrate
# Bring up the app
- ./start.sh
# Run cucumber test
- cd ../<automation project name>
- mvn verify -P rest-test-run -Denvironment="qa" -Dwebdriver.chrome.driver="C:\tools\chromedriver.exe"
- cd ../<webapp project name>
# Stop the app
- ./stop.sh
I have also attached a screen shot of the stalling job.
Some troubleshooting ideas that I tried implementing:
upgraded surefireplugin(the version that I use in 2.22.0)
added a cleanup_job in yaml file
ran tests not in parallel
the above ideas did not solve the problem.

gitlab-ci.yml fatal: unable to access http://gitlab-ci-token

I am new to gitlab CI/CD and I'm struggling to figure this out.
All I want to do is when I push to dev branch I want my react app to be built and the folder ./build to be pushed through SSH to my dev server.
Here is what I did so far, including a screenshot of the error message I get.
This is my gitlab-ci.yml
image: node:latest
cache:
paths:
- node_modules/
build_dev:
stage: build
environment: Development
only:
- dev
script:
- ls
- npm install
- npm run build
artifacts:
paths:
- build/
- ecosystem.config.js
deploy_dev:
stage: deploy
environment: Development
only:
- dev
script:
- rsync -r -a -v -e ssh --delete "./build" root#dev.teledirectasia.com:/var/www/gitlab/${CI_PROJECT_NAME}
- rsync -r -a -v -e ssh --delete "./ecosystem.config.js" root#dev.teledirectasia.com:/var/www/gitlab/${CI_PROJECT_NAME}/
- ssh root#dev.teledirectasia.com "cd /var/www/gitlab/${CI_PROJECT_NAME} && pm2 start ecosystem.config.js"
I don't know why I am getting this output with job failed
This is a DNS problem. Your runner cannot resolve the hostname of the GitLab server - gitlab.teledirectgroup.com. Did you set the GitLab hostname if your local workstation's host file manually, or did you set it up in a DNS server as a 'proper' hostname?
If you set up the hostname in a DNS server then the solution may be as simple as adding the DNS server to /etc/resolv.conf on the runner. However, if you just set the GitLab hostname in your workstation's hosts file then you'll need to set it in the runner's /etc/hosts file, too. It's hard to say what the exact solution is without knowing how you set up the GitLab hostname in the first place.
the solution that’s so applied to GitLab?
Use the git clone by ssh, I don’t have a good goal that’s so I can up to push that’s changes over a submodule from runner Shell by GitLab CI. The pipeline ever fails and prints this error.
ERROR PIPELINE JOB
In the local repo as a project the file config contains that line with the URL, more don’t have login with this about the pipeline.
.git/config
Some help or walkthrough of reference to culminate with that challenge in troubleshooting!
This is my code over the file ".gitlab-ci.yml"
variables:
TEST_VAR: "Update Git Submoudel in all Etecnic projects."
job1:
variables: {}
script:
- echo "$TEST_VAR"
job2:
variables: {}
script:
- echo "OK" >> exito.txt
- git add --all
- git commit -m "Update Submodule"
- git push origin HEAD:master
Versions:
GitLab:
gitlab-ce is already the newest version (15.6.0-ce.0).
Runner:
Version: 15.5.1
Git revision: 7178588d
Git branch: 15-5-stable
GO version: go1.18.7
Built: 2022-11-11T09:45:25+0000
OS/Arch: linux/amd64
Thanks so much for your attention.

Uploading artifacts is disabled

Does anyone know how to resolve "Missing gitlab-runner. Uploading artifacts is disabled."
I have two ssh runner on my server. One for local ssh runner and another one is remote server ssh runner. The local ssh runner working perfectly. But when I call the remote shh runner it's building successfully but it shows "Missing gitlab-runner. Uploading artifacts is disabled." so that I'm not able to call the artifact in the next stage.
The gitlab-ci.yml file looks like this:
stages:
- build
- deploy
cache:
paths:
- .gradle/wrapper
- .gradle/caches
build:
stage: build
script:
- build script
artifacts:
paths:
- build/
only:
- master
deploy:
stage: deploy
script:
- deploy script
dependencies:
- build
only:
- master
You need to install gitlab-runner also on the ssh target host. Gitlab somehow needs this to upload artifacts. This is also documented now: https://docs.gitlab.com/runner/executors/ssh.html:
If you want to upload job artifacts, install gitlab-runner on the host you are connecting to via SSH.
You can put the gitlab-runner in the path ~/bin and add ~/bin to your variable PATH.

Resources