I have a Jenkins installation running several dozen NodeJs applications.
Currently, each job just builds on the GitHub webhook for the master branch.
I would also like to build each project periodically, once per day. I don't want to have to copy/paste the same cron into 30 job configurations, though - mostly because updates would become tedious.
I looked around and did not find any obvious plugin for managing shared cron patterns, etc.
Any options?
A good idea here could have been to use a global variable defined at the level of Jenkins. Unfortunately the cron job field does not allow referencing a global variable.
However, there are multiple ways to achieve what you are aiming for.
The first solution is to create dummy job that has the schedule. This job will just trigger all the other jobs to run using the post build action Build other projects. That way you will only set the cron expression on the dummy job and all the other jobs will be triggered once the dummy job runs.
Another similar solution is to use Build after other projects are built option in Jenkins. You can set the cron schedule on one of the jobs that you have, and set the Build after other projects are built to build after the scheduled job has been built for the rest of the jobs. That way the "main" schedueled job will run and all the others will follow.
Related
Please note: I am not interested in any enterprise/for-pay (Tower?) solutions here, only solutions available via Ansible's OSS offering.
OK so I've got my Ansible project configured and working perfectly, woo hoo! Looks something like this:
myansible01.example.com:/opt/ansible/
site.yml
fizz.yml
buzz.yml
group_vars/
roles/
common/
tasks/
main.yml
handlers
main.yml
foos/
tasks/
main.yml
handlers/
main.yml
There's several things I need to accomplish to get this working in a production environment:
I need to be able to automate the deployment of changes to this project
I need to schedule playbooks to be ran, say, every 30 seconds (to ensure all managed nodes are always in compliance)
So my concerns:
How are changes usually deployed to live Ansible projects? Say the project is located at myansible01.example.com:/opt/ansible (my Ansible server). Is it sufficient to simply delete the Ansible project root (rm -rf /opt/ansible) and then copy the latest (containing changes) Ansible project back to the same location? What happens if Ansible is currently running any plays while I perform this "drop-n-swap"?
It looks like the commercial offering (Ansible Tower) has a scheduling feature built into it, but not the OSS offering. How can I schedule Ansible OSS to run plays at certain times? For instance, I might want certain plays to be ran every 30 seconds, so as to ensure nodes are always within compliance. Is cron sufficient to do this, or is there a more standard approach?
For this kind of task you typically want an orchestration engine such as Jenkins to do all your, well, orchestration.
You can set Jenkins to run playbooks on timers or other events such as a push to an SCM such as git.
Typically a job starts by checking out a tag/branch of our Ansible code base and then applying it to all of our specified servers so you always know what is being run. If you want, this can simply be the head on master (in git terms) so it's always applying the most recent changes. If you were also to have this to hook into your SCM repo then a simple push will force those changes to be applied to all of your servers.
Because of that immediacy you might want to consider only doing this on some test servers that then have some form of testing done against them (such as Serverspec) to verify that your changes are good before rolling them out to a production environment.
Jenkins, by default, will not run a job while the same job is running (or if you are maxed out on executor slots) so you can always be sure that it will only pull the repo (including any changes) after your Ansible run is complete. If you have multiple jobs running you can use blocking to prevent jobs running at the same time (both trying to apply potentially different configurations to the servers) but you don't have to worry about a new job starting and pulling the repo into the already running job as Jenkins separates these into separate work spaces.
We use Jenkins for manual runs of Ansible against our environment but we also have a "self healing" Jenkins job that simply runs a tagged commit of our Ansible code base against our environment, forcing it to an idempotent state to prevent natural drift of configurations. When we need to do something different to the environment or are running a slightly further ahead commit of our code base in to it we can easily disable the self healing job until we're happy with things and then either just re-enable the job to put things back or advance the tag that Jenkins is using to now use the more recent commit.
I'm looking for a tool that we allow me to script a Jenkins build pipeline made up of a series of individual build jobs.
Is the Jenkins Workflow plugin capable of doing this?
Ideally I would use the groovy script to define which jobs needs to run, in what order, which ones can run in parallel etc but leave the details of exactly what each job does to the job itself.
So I guess my question boils down to,
Can you trigger build jobs from a workflow script?
Can you define what should happen in parallel from the script?
Yes, you can use the parallel and build steps for this purpose.
There is no need for the Parameterized Trigger plugin unless you need to start a Workflow build with parameters from a traditional project, which does not sound like your use case.
Yes Jenkins workflow plugin is used for creating deployment pipelines. Individual workflow plugin can not do this job. You can also used parameterized trigger plugin which will trigger your jobs and by one. Suppose you have job A,B,C and D. You want to trigger Job A and it will trigger your other jobs sequentially.
A->B->C->D
With workflow you can see the complete workflow how the jobs are triggered and its status.
To trigger your first job which is job A, you can use Jenkins API's. We write git pre-commit hooks to trigger our CI cycle.
I need to spawn a variable number of jobs from one upstream job.
AFAIK, there is no plugin that can do this. The closest one is MultiJob plugin (https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin).
So I thought to create a build step that would use one of the Jenkins APIs (REST, groovy or jenkins-cli) to trigger those builds.
However, if I do that, those builds become "detached" (meaning they do not have an upstream job) and the main job has no linkage with those builds.
So it boils down to this: is it possible to start a job build and tell it who is its upstream?
There is Build Result Trigger plugin. It is literally the inverse of Parameterized Trigger Plugin. Instead of triggering downstream jobs, like the latter does, the Build Result Trigger lets your "downstream" jobs watch/monitor the progress of an upstream job, and trigger based on that result.
This way, your "upstream" job is actually not aware of downstream jobs that are watching it.
Check out the Groovy Plugin.
It'll let you fire as many jobs as you want, and set the upstream cause.
Code Example: http://wiki.jenkins-ci.org/display/JENKINS/Groovy+plugin#Groovyplugin-Retrievingparametersandtriggeringanotherbuild
job.scheduleBuild2(0, new Cause.UpstreamCause(build), new ParametersAction(params))
A related post is here:
How do I dynamically trigger downstream builds in jenkins?
However, from all the answers that I have read it's clear that using the Groovy/Java class hudson.model.Cause.UpstreamCause(currentBuild) does achieve the goal of programmatically triggering another job, but it does not fully establish an upstream/downstream relationship.
When you examine the builds you do not see any upstream/downstream information. The only way to see those is to open the console output of each.
Is it possible to use Jenkins server to run custom tasks one by one?
By task I mean to execute an external groovy program which designed as an independent performance and integration test for specific deployment.
If it is possible then how to:
To define tasks in Jenkins and group them so they can start by starting a group.
To see an output of each task (output log).
If there is a specific outcome like "-1" then stop execution of the whole group.
And all this should start automatically after software has been built and deployed.
I feel there has to be a way to do it with Jenkins utilising its out-of-the-box functionality, just not sure how. Or I am wrong and we are looking at custom plugin as a solution?
Thanks a lot!
P.S. I am not asking for detailed answer, just a general direction would be Ok. Also Jenkins is not a requirement, it can be another similar CI server.
It sounds like this could work by a simple Jenkins task with Execute shell commands.
The Console Output for the jobs will contain the output from the processes that you run externally, and the exit status of the script can cause the task to be in failure (any non-zero exit code will do this by default).
On unix systems, #! beginning the first line will denote the script environment to use.
To chain this together with the other Jenkins steps, you can use Build Triggers for Build after other projects are built and use your deployment step as the starting off point.
It is possible, but be careful. Normally Jenkins is used to run build jobs and to deploy software to a QA or staging server. It does not touch Production. But when you start doing this in Jenkins you increase the risk that someone will accidentally run a production job that should not have been run. So if you do decide to use Jenkins for this, set up an entirely separate instance of Jenkins that does nothing other than run these jobs. Then go to Manage Jenkins->Configure Global Security and set up login users. At the least, use "logged in users can do anything" but it would be better to set up "matrix-based security". Then run any jobs that you need by using an Execute Shell step. You can schedule jobs by using a Build Trigger, and you can connect jobs sequentially by setting up Build Other Projects in the post build section. If you want to do more complex job chaining, look into the Join Plugin.
Just keep this Jenkins entirely separate from the Jenkins which you use for CI.
We are using a dedicated Amazon Ubuntu ec2 instance as Cron server, which executed 16 cron jobs at different time intervals i.e, 10 cron jobs in morning 4:15 - 7:15 and the rest # 23:00 - 23:50. I get the results via email. I want to configure something, which shoots email message at the end of they day listing the cron jobs that are executed successfully and the one that failed.
I have a jenkins configured ubuntu instance for auto-building Dev, Beta, Staging & Live environments. Can i add these cron jobs(shell scripts) as external jobs in the jenkins and monitor them. Is it possible?
Definitely possible! You can monitor external cron jobs as described here:
https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+external+jobs
You can also add cron job (-like behavior) to Jenkins by creating a freestyle software project and add "Execute shell" as build process.
It's a bit more convenient since you can also trigger the execution via Jenkins ("Build now").
You might be able to combine the Jenkins monitor external job project type with a matrix project. At the very least the former will enable you to monitor the cron jobs individually.
Alternatively you could have the last monitored cron job of the day trigger building a project that checks the status of all the cron jobs (for example by retrieving and comparing the build numbers of the last and the last successful builds) and sends an email accordingly. The email plugin might be useful for the latter.
Check the CPAN or do some web digging for shell or perl script for managing cron jobs and extend its behaviour to do some reporting which you can render using HTML. Alternatively write a servlet and a some function calls to do just that.
This becomes your own standalone monitor application, which can sit in jenkins or deployed independently. If you choose to add it to jenkins, then add the reporting HTML file and its scripts to the container holding deployed web files for jenkins, word of advice place your files and script in a separate container.
Add a hyperlink to jenkins index html which will load your reporter. Now reboot tomcat and go from there.
Another option could be to take a look at Cronitor (https://cronitor.io). It basically boils down to being a tracking beacon that uses http requests to ping when a cron job/scheduled task starts and ends.
You'll be notified if your job doesn't run on schedule, or if it runs for too long/too short, etc. You can also configure it to send alerts to you via email, sms, but also Slack, Hipchat, Pagerduty and others.