[Disclamer] : this post focus exclusively on CI / build aspects and has nothing to do with CD / deployment
Say I have a project hosted on Gitlab involving some source code compilation. For the present purpose, let's say I wan't Maven to create two cars (STAGE PROD) of some java project every time a developer merges on master. Those cars:
will be stored on the project registry
need to contain the environment variables defined for this project.
Can I legitimately assume that those environment variables will remain safe (i.e. private) if the project is compiled with shared gitlab runners based on the assumption that Docker runners are ephemeral ? Is there a (better) way to enforce privacy despite using shared runners?
One possible alternative would be using Protected environment variables.
issue 1452 show them in "CI/CD" panel of Settings.
By prefixing them with STAGE_ or PROD_, your maven job could use the right variable depending on the current car.
By their protected nature, they would not be exposed.
Related
In Circle CI, context can allow me to set different values for same variables.
For example, I set two environments, such as dev and prod, in each of them, I set several variables
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
Since my environments are in different aws accounts, I can provide the different values to them.
Second, I can set permission that developers can only access dev context, support team can only access prod context.
But I don't get the same feature in Gitlab CI.
In one of their document, it mentions Group, but after I check, it doesn't work as expect at all.
https://docs.gitlab.com/ee/ci/migration/circleci.html#contexts-and-variables
Contexts and variables
CircleCI provides Contexts to securely pass environment variables across project pipelines. In GitLab, a Group can be created to assemble related projects together. At the group level, CI/CD variables can be stored outside the individual projects, and securely passed into pipelines across multiple projects.
Are there any way I can do that in Gitlab CI?
Sample usage of context in Circli CI for your reference
version: 2.1
workflows:
staging-workflow:
jobs:
- staging-build:
context:
- staging
prod-workflow:
jobs:
- prod-build:
context:
- prod
I think you could achieve something similar by utilising gitlabs schedules.
Simply create two schedules that contain the variables you want to pass.
Update your ci script to reference these variables.
Then each schedule can be owned by the respective parties.
You can also limit variables by group, based on which environment. So if you have a dev env - variables can be limited for that and the same for a prod env. Hope this helps.
Finally I got this problem fixed with Gitlab CI enviroments
set environments Staging and Produciton
when it asks for url, ignore the issue, put any url you like, I just put http://example.com .
still not sure the proper url I need put in.
update your variables in cicd, default is to all, change it to staging or Production.
Setting -> CICD -> Variables -> Expand
in the screenshot, I set variable AWS_ACCESS_KEY_ID two times, but assigned to different environments.
(Notes: AWS_ACCESS_KEY_ID doesn't need be masked, but AWS_SECRET_ACCESS_KEY should be)
update your pipeline to use the enviroment.
the gitlab ci environment document above confuse me again. you don't have to provide the url, just add the environment name is enough.
staging_deploy:
stage: deploy
environment:
name: staging
when: manual
script:
- echo "staging deployment"
The .gitlab-ci.yml configuration file should not be exposed to any user with the "developer" rule since it might grant unwanted access to variables and infrastructure or make different kinds of exploiting behaviour or simply unwanted changes possible.
therefore, according to https://gitlab.com/secure-ci-config-poc/ci-configs, i made projects and pipelines; but if the user who pushes don't have reporter or developer or other high permissions on ci-configs project which contains all configurations, pipeline fails!!
Found errors in your .gitlab-ci.yml:
Project `root/ci-configs` not found or access denied!
now, how can I fix this error?! so developers can run pipelines, but can not access the configuration files and .gitlab-ci.yml files?
Thanks All
You cannot stop users from reading the configuration. A user triggering a pipeline must have at least read access to the CI yaml file. However, secrets should never be stored in the YAML file, so read access should generally not be problematic.
You can prevent write access, but users triggering pipelines must be able to read all the configuration files. That is really the primary goal in securing your CI configurations -- preventing execution of malicious changes to the CI configuration.
if the user who pushes don't have reporter or developer or other high permissions on ci-configs project which contains all configurations, pipeline fails
The configuration project should have either public or internal visibility to avoid this problem, as described in the GitLab documentation:
If the configuration file is in a separate project, you can set more granular permissions. For example:
Create a public project to host the configuration file.
Give write permissions on the project only to users who are allowed to edit the file.
Then other users and projects can access the configuration file without being able to edit it.
(emphasis added)
If you absolutely needed the project to be set to private visibility, you might consider granting developer access, but creating protected branch rules that require maintainer access or higher to push changes.
Additional considerations
Even if you prevent access to writing changes to the CI configuration file, if the CI configuration executes any code written in the repository (say, for example running unit tests) then you really haven't solved any problems. Consider, for example, that malicious code can be embedded in test code!
It is possible to have a CI configuration that does not execute user code, but it's something you need to consider. If you need CI configurations to execute user-provided code (like running tests) then it's likely not very advantageous to protect your CI configuration in this way as a matter of securing your environment/variables.
We have hundreds of similar projects in GitLab which have the same structure inside.
To build these projects we use a one common TeamCity build. We trigger and pass project GitLab URL along with other parameters to the build via API, so TeamCity build knows which exact project needs to be fetched/cloned. TeamCity VCS root accepts target URL via parameter.
The question is how to replace existing TeamCity build with a GitLab pipeline.
I see the general approach is to have CI/CD configuration file(.gitlab-ci.yml) directly in project. Since the structure of the projects the same this is not the option to duplicate the same CI/CD config file across all projects.
I'm wondering is it possible to create a common pipeline for several projects which can accept the target project URL via parameter ?
You can store the full CICD config in a repository and put in all your projects a simple .gitlab-ci.yml which includes the shared file.
With thus approach there is no redundant definition of the jobs.
Still, you can add specific other jobs to specific projects (in the regarding .gitlab-ci.yml files or define variables in a problem and use some jobs conditionally) - you can also include multiple other definition files, e.g. if you have multiple similar projects.
cf. https://docs.gitlab.com/ee/ci/yaml/#include
With latest GitLab (13.9) there are even more referencing methods possible: https://docs.gitlab.com/ee/ci/yaml/README.html#reference-tags
As #MrTux already pointed out, you can use includes.
You can either use it to include a whole CI file, or to include just certain steps. in Having Gitlab Projects calling the same gitlab-ci.yml stored in a central location - you can find detailed explanation with examples of both usages
Only one gitlab runner is available in my project. However, in the runner page I have multiple other runners listed. How do I add these runners to the project such that a runner will be randomly selected to execute the build? The runners are running in Docker containers and I am using a self-managed version of GitLab.
These are the runners I have available
Only runner1 is currently shown in /settings/cicd under Runners:
runner1
Basically, if you go by runners there are actually three types of a runner in Gitlab
Specific: Only for Specific Project
Group: For all projects in the particular group (if need more info on groups https://docs.gitlab.com/ee/user/group)
Shared: These will be available for every project in the GitLab
It will be difficult to specify here that how we can configure the runners but I can point to documentation from where we actually configured our Runners
https://docs.gitlab.com/ee/ci/runners/
Also to make a note of tagging is a very important task that needs to be done so that you can use your runners with your requirement like you said you want to use runners randomly from the list then all the runners in the list should have a common tag that can be further used in your .gitlab-ci.yml file.
eg. We have tag runners as per the AWS ec2 instance size so (like macro, small, medium, large, etc) so that people using can use the runners as per the requirement of there job)
job1:
tags:
- shared-large-runner
job2:
tags:
- shared-micro-runner
Note: you can have multiple tags for a single runner for better control over its usage.
Hopefully, this helps!!
Looking at your screenshot, to be able to use group runners, your project just need to be inside that group
Because the group tag shows that the runner was configured just to accept jobs from one group
I think you have one runner for each group, this is the reason that your project only have the runner1
I'm a bit new to version control and deployment environments and I've come to a halt in my learning about the matter: how do deployment environments work if developers can't work on the same local machine and are forced to always work on a remote server?
How should the flow of the deployment environments be set up according to best practices?
For this example I considered three deployment environments: development, staging and production; and three storage environments: local, repository server and final server.
This is the flow chart I came up with but I have no idea if it's right or how to properly implement it:
PS. I was thinking the staging tests on the server could have restricted access through login or ip checks, in case you were wondering.
I can give you (according to my experience) a good and straightforwarfd practice, this is not the only approach as there is not a unique standard on how to work on all projects:
Use a distributed version control system (like git/github):
Make a private/public repository to handle your project
local Development:
Developers will clone the project from your repo and contribute to it, it is recommended that each one work on a branch, and create a new branch for each new feature
Within your team, there is one responsible for merging the branches that are ready with the master branch
I Strongly suggest working on a Virtual Machine during Development:
To isolate the dev environment from the host machine and deal with dependencies
To have a Virtual Machine identic to the remote production server
Easy to reset, remove, reproduce
...
I suggest using VirtualBox for VM provider and Vagrant for provisioning
I suggest that your project folder be a shared folder between your host machine and your VM, so, you will write your source codes on your host OS using the editor you love, and at the same time this code exists and runs inside your VM, is in't that amazingly awesome ?!
If you are working with python I also strongly recommend using virtual environments (like virtualenv or anaconda) to isolate and manage inner dependencies
Then each developer after writing some source code, he can commit and push his changes to the repository
I suggest using project automation setup tools like (fabric/fabtools for python):
Making a script or something that with one click or some commands, reproduces all the environment and all the dependencies and everything needed by the project to be up and running, so all developers backend, frontend, designers... no matter their knowlege nor their host machine types can get the project running very merely. I also suggest doing the same thing to the remote servers whether manually or with tools like (fabric/fabtools)
The script will mainly install os dependencies, then project dependencies, then cloning the project repo from your Version Control, and to do so, you need to Give the remote servers (testing, staging, and production) access to the Repository: add ssh public keys of each server to the keys in your Version Control system (or use agent forwarding with fabric)
Remote servers:
You will need at least a production server which makes your project accessible to the end-users
it is recommended that you also have a testing and staging servers (I suppose that you know the purpose of each one)
Deployment flow: Local-Repo-Remote server, how it works ?:
Give the remote servers (testing, staging, and production) access to the Repository: add ssh public keys of each server to the keys in your Version Control system (or user agent forwarding with fabric)
The developer writes the code on his machine
Eventually writes tests for his code and runs them locally (and on the testing server)
The developer commits and pushes his code to the branch he is using to the remote Repository
Deployment:
5.1 If you would like to deploy a feature branch to testing or staging:
ssh access to the server and then cd to the project folder (cloned from repo manually or by automation script)
git checkout <the branch used>
git pull origin <the branch used>
5.2 If you would like to deploy to production:
Make a pull request and after the pull request gets validated by the manager and merged with master branch
ssh access to the server and then cd to the project folder (cloned from repo manually or by automation script)
git checkout master # not needed coz it should always be on the master
git pull origin master
I suggest writing a script like with fabric/fabtools or use tools like Jenkins to automate the deployment task.
Voilà! Deployment is done!
This is a bit simplified approach, there are still a bunch of other recommended and best practice tools and tasks.