I'd like to use Gitlab CI in my workflow, but because my project relies on licensed software, so I need it to run on my machine, which does not have a public, routable IP. My thought is that I can create a simple server on heroku to accept the webhooks and put the requests into a message queue (a redis DB, for instance), which my local machine can poll and actually run the CI job. It appears, however, that the entire Gitlab CI system is written assuming that the gitlab.com server can directly speak to the runner. Does anybody know of a proof of concept for proxying the CI build trigger through a webhook or making the gitlab-runner pull build jobs rather than accepting push events? I could roll my own runner if necessary that polls for build events and runs the commands I need, but it'd be really nice to use the existing, documented infrastructure/file format rather than reinventing the wheel. Thanks for any suggestions.
It turns out that I had misinterpreted the documentation and this already works out of the box with the standard runner.
Related
I'm new to CI/CD.
I installed Gitlab Runner on my VPS for my project.
The first pipeline was successfully passed after the push to the master branch.
Questions:
Gitlab Runner does not listen to any port. Instead, it connects to the Gitlab server itself and checks if there are tasks for it. Is that true?
If so, how often does Gitlab Runner connect to gitlab.com?
Is it dangerous to keep Gitlab Runner and the project on the same VPS?
Gitlab Runner does not listen to any port. Instead, it connects to the Gitlab server itself and checks if there are tasks for it. Is that true?
Yes. It is a "pull" mechanism exclusively, as far as the jobs go. Though the runner may open ports for additional functionality, such as the session server.
If so, how often does Gitlab Runner connect to gitlab.com?
By default, every 3 seconds to check for new jobs (assuming the concurrency limit has not been reached). This can be configured through the check_interval configuration
Is it dangerous to keep Gitlab Runner and the project on the same VPS?
It can be, so this should be avoided where possible. The degree of danger depends somewhat on which executor you use and how you configure your server.
The file system of your GitLab server is extremely sensitive; it contains many critical secrets and operational components that should not be exposed to anyone (including CI jobs). Besides this, jobs would also have the potential to cause performance problems -- if a job uses too much memory/cpu/IO/etc., it could cause resource contention and harm/crash your other GitLab components on the server.
You might be able to get away with a single VPS if you deploy the server and runner as separate containers with a Docker or Helm deployment of GitLab and the runner on the VPS host; you can use separate logical Docker volumes and set resource constraints via Docker... but there is still some additional risk in such a scenario and configuration mistakes (which may be easier to make than one might think) can have serious consequences.
I'm hoping to leverage GitLab CI/CD / gitlab-runner to keep custom code up to date on a fleet of servers.
Desired effect is that when a commit is made against a certain project in GitLab, several servers then automatically pull those changes down.
Is it possible to leverage gitlab-runner's in this way, so that every runner registered with the project executes the contents of the .gitlab-ci.yml file? Or is there a better tool to accomplish this?
I could use Ansible to push updates files down to each server, but I was looking for something easier to solve for - something inherent in GitLab.
Edit: Alternative Solution
I decided to go the route of pre- and post-hook files in my repos as described here:
https://gist.github.com/noelboss/3fe13927025b89757f8fb12e9066f2fa
Basically I will be denoting a primary server as the main source for code pushes into the master repo, and have defined my entire fleet as remote repos in .git/config there. Using post-hooks inside of bare repo's on all of my servers, I can then copy my code into the proper execution path.
#ahxn81 Runners aren't really intended to be used in the pull way you describe. The Ansible push method you proposed is more in line with typical deploy flow. I can see why you might prefer the simplicity of the pull method over pushing via script. I guess a fleet of servers these days is via kubernetes or docker swarm which can simplify deployment after an initial setup headache.
I have multiple servers on a load balanced environment running the same Node application. I want the code on these servers be the same everywhere. I currently maintain a git repo for the code on these servers but have to manually SSH into each of these and pull the code from the git repo. Is there any simple way I can push the code onto all the servers?
Straightforward solutions that came in my mind:
1) use a cron job on the servers that makes the work you're doing manually, i.e., git pull (need linux), or
2) use git hooks to trigger the pull on the other servers. With this solution you need to have the list of servers to trigger the update. Hooks are basically scripts that are executed before/after events like commits, push, etc.
It looks your question is how to deploy Node.js app into multiple server. Here related question
Is there any existing tooling/platform I can use to do the following?
On any github PR or commit, have a custom "check", e.g the same as how travis-ci works.
Have this task talk to a remote machine on azure.
Execute a script on this machine and collect logs/exit code
Fail the check if the code is none zero or timeout is reached.
Handle queuing if two PR's come in, clean up on abort etc.
Have some sort of "status" badge like travis-ci to see the current test state/pass rate.
So far only travis-ci itself seems to work something like this, but whatever I execute will run in their cloud so I don't "own" the machine. Additionally my integration tests require copyrighted data which needs to be kept safe on my own cloud machine, and could take multiple hours to complete.
Yes you can. https://help.github.com/articles/about-webhooks/ describes how to do this. Your machine will need to be accessible to github to do this.
I'm relatively new to continuous integration servers. I've been using GitLab (v6.5) for a while to manage projects, but I'd like to begin using the GitLab CI to ensure tests pass and builds succeed.
My testing setup consists of two virtual machines: one machine for GitLab and another machine for the GitLab CI (and runners). However, in production I only have a single machine, which is running GitLab. The GitLab team posted an interesting blog post a while back that emphasized:
If you are running tests on the CI server you are doing it wrong!
It was a very informative post, but I didn't come away feeling like I understood this specific point. Does this mean one shouldn't run GitLab and GitLab CI on the same server? Does it mean one shouldn't run GitLab CI and GitLab CI runners on the same server? Or both-- Do I need three servers, one for each task?
From the same post:
Anybody who can push to a branch that is tested on a CI server can easily own that server.
This implies to me that the runners are the security risk since they can run stuff contained in a commit. If that's the case, what's the typical implementation? Put GitLab and GitLab CI on the same machine, but the runners on a separate machine? Wouldn't it still suck if the runner machine was compromised? So people are okay losing their runner machine as long as their code machine is safe?
I would really like to understand this a bit more-- definitely before I implement it in production. Is there any possible yet safe way to implement GitLab, GitLab CI, and GitLab CI runners all on the same machine?
Ideally you're fine running gitlab-ci and gitlab on the same host. Others may disagree with me but the orechestrator (the gitlab-ci node) doesn't do any of the heavy lifting. Its strictly job meta IO and warehousing the results.
With that being said, I would not put the runners on the same machine. Gitlab-CI Runners are resource intensive and will be executing at full tilt on whichever machine you place them on. Its a good idea if you're running in production to put these on spot instances to help curb some of the costs of running the often cpu/memory hungry builds - but can be impractical as your instances are not always on at that point.
I've had some success with putting my gitlab-ci runner's in digital ocean on small instances. I'm not doing HUGE builds, but the idea is to distribute the work load against several servers so your CI server:
Is responsive
Can build multiple project builds at once
Can exercise isolation (this is kind of arbitrary in this list)
and a few other things that don't come to mind right away.
Hope this helps!