I'm looking for a tool that we allow me to script a Jenkins build pipeline made up of a series of individual build jobs.
Is the Jenkins Workflow plugin capable of doing this?
Ideally I would use the groovy script to define which jobs needs to run, in what order, which ones can run in parallel etc but leave the details of exactly what each job does to the job itself.
So I guess my question boils down to,
Can you trigger build jobs from a workflow script?
Can you define what should happen in parallel from the script?
Yes, you can use the parallel and build steps for this purpose.
There is no need for the Parameterized Trigger plugin unless you need to start a Workflow build with parameters from a traditional project, which does not sound like your use case.
Yes Jenkins workflow plugin is used for creating deployment pipelines. Individual workflow plugin can not do this job. You can also used parameterized trigger plugin which will trigger your jobs and by one. Suppose you have job A,B,C and D. You want to trigger Job A and it will trigger your other jobs sequentially.
A->B->C->D
With workflow you can see the complete workflow how the jobs are triggered and its status.
To trigger your first job which is job A, you can use Jenkins API's. We write git pre-commit hooks to trigger our CI cycle.
Related
Is there a parameter or a setting for running pipelines in sequence in azure devops?
I currently have a single dev pipeline in my azure DevOps project. I use this for infrastructure because I build, test, and deploy using scripts in multiple stages in my pipeline.
My issue is that my stages are sequential, but my pipelines are not. If I run my pipeline multiple times back-to-back, agents will be assigned to every run and my deploy scripts will therefore run in parallel.
This is an issue if our developers commit close together because each commit kicks off a pipeline run.
You can reduce the number of parallel jobs to 1 in your project settings.
I swear there was a setting on the pipeline as well but I can't find it. You could do an API call as part or your build/release to pause and start the pipeline as well. Pause as the first step and start as the last step. This will ensure the active pipeline is the only one running.
There is a new update to Azure DevOps that will allow sequential pipeline runs. All you need to do is add a lockBehavior parameter to your YAML.
https://learn.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-190-update
Bevan's solution can achieve what you want, but there has an disadvantage that you need to change the parallel number manually back and forth if sometimes need parallel job and other times need running in sequence. This is little unconvenient.
Until now, there's no directly configuration to forbid the pipeline running. But there has a workaruond that use an parameter to limit the agent used. You can set the demand in pipeline.
After set it, you'll don't need to change the parallel number back and forth any more. Just define the demand to limit the agent used. When the pipeline running, it will pick up the relevant agent to execute the pipeline.
But, as well, this still has disadvantage. This will also limit the job parallel.
I think this feature should be expand into Azure Devops thus user can have better experience of Azure Devops. You can raise the suggestion in our official Suggestion forum. Then vote it. Our product group and PMs will review it and consider taking it into next quarter roadmap.
I have a Jenkins installation running several dozen NodeJs applications.
Currently, each job just builds on the GitHub webhook for the master branch.
I would also like to build each project periodically, once per day. I don't want to have to copy/paste the same cron into 30 job configurations, though - mostly because updates would become tedious.
I looked around and did not find any obvious plugin for managing shared cron patterns, etc.
Any options?
A good idea here could have been to use a global variable defined at the level of Jenkins. Unfortunately the cron job field does not allow referencing a global variable.
However, there are multiple ways to achieve what you are aiming for.
The first solution is to create dummy job that has the schedule. This job will just trigger all the other jobs to run using the post build action Build other projects. That way you will only set the cron expression on the dummy job and all the other jobs will be triggered once the dummy job runs.
Another similar solution is to use Build after other projects are built option in Jenkins. You can set the cron schedule on one of the jobs that you have, and set the Build after other projects are built to build after the scheduled job has been built for the rest of the jobs. That way the "main" schedueled job will run and all the others will follow.
I need to spawn a variable number of jobs from one upstream job.
AFAIK, there is no plugin that can do this. The closest one is MultiJob plugin (https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin).
So I thought to create a build step that would use one of the Jenkins APIs (REST, groovy or jenkins-cli) to trigger those builds.
However, if I do that, those builds become "detached" (meaning they do not have an upstream job) and the main job has no linkage with those builds.
So it boils down to this: is it possible to start a job build and tell it who is its upstream?
There is Build Result Trigger plugin. It is literally the inverse of Parameterized Trigger Plugin. Instead of triggering downstream jobs, like the latter does, the Build Result Trigger lets your "downstream" jobs watch/monitor the progress of an upstream job, and trigger based on that result.
This way, your "upstream" job is actually not aware of downstream jobs that are watching it.
Check out the Groovy Plugin.
It'll let you fire as many jobs as you want, and set the upstream cause.
Code Example: http://wiki.jenkins-ci.org/display/JENKINS/Groovy+plugin#Groovyplugin-Retrievingparametersandtriggeringanotherbuild
job.scheduleBuild2(0, new Cause.UpstreamCause(build), new ParametersAction(params))
A related post is here:
How do I dynamically trigger downstream builds in jenkins?
However, from all the answers that I have read it's clear that using the Groovy/Java class hudson.model.Cause.UpstreamCause(currentBuild) does achieve the goal of programmatically triggering another job, but it does not fully establish an upstream/downstream relationship.
When you examine the builds you do not see any upstream/downstream information. The only way to see those is to open the console output of each.
Is it possible to use Jenkins server to run custom tasks one by one?
By task I mean to execute an external groovy program which designed as an independent performance and integration test for specific deployment.
If it is possible then how to:
To define tasks in Jenkins and group them so they can start by starting a group.
To see an output of each task (output log).
If there is a specific outcome like "-1" then stop execution of the whole group.
And all this should start automatically after software has been built and deployed.
I feel there has to be a way to do it with Jenkins utilising its out-of-the-box functionality, just not sure how. Or I am wrong and we are looking at custom plugin as a solution?
Thanks a lot!
P.S. I am not asking for detailed answer, just a general direction would be Ok. Also Jenkins is not a requirement, it can be another similar CI server.
It sounds like this could work by a simple Jenkins task with Execute shell commands.
The Console Output for the jobs will contain the output from the processes that you run externally, and the exit status of the script can cause the task to be in failure (any non-zero exit code will do this by default).
On unix systems, #! beginning the first line will denote the script environment to use.
To chain this together with the other Jenkins steps, you can use Build Triggers for Build after other projects are built and use your deployment step as the starting off point.
It is possible, but be careful. Normally Jenkins is used to run build jobs and to deploy software to a QA or staging server. It does not touch Production. But when you start doing this in Jenkins you increase the risk that someone will accidentally run a production job that should not have been run. So if you do decide to use Jenkins for this, set up an entirely separate instance of Jenkins that does nothing other than run these jobs. Then go to Manage Jenkins->Configure Global Security and set up login users. At the least, use "logged in users can do anything" but it would be better to set up "matrix-based security". Then run any jobs that you need by using an Execute Shell step. You can schedule jobs by using a Build Trigger, and you can connect jobs sequentially by setting up Build Other Projects in the post build section. If you want to do more complex job chaining, look into the Join Plugin.
Just keep this Jenkins entirely separate from the Jenkins which you use for CI.
We are using a dedicated Amazon Ubuntu ec2 instance as Cron server, which executed 16 cron jobs at different time intervals i.e, 10 cron jobs in morning 4:15 - 7:15 and the rest # 23:00 - 23:50. I get the results via email. I want to configure something, which shoots email message at the end of they day listing the cron jobs that are executed successfully and the one that failed.
I have a jenkins configured ubuntu instance for auto-building Dev, Beta, Staging & Live environments. Can i add these cron jobs(shell scripts) as external jobs in the jenkins and monitor them. Is it possible?
Definitely possible! You can monitor external cron jobs as described here:
https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+external+jobs
You can also add cron job (-like behavior) to Jenkins by creating a freestyle software project and add "Execute shell" as build process.
It's a bit more convenient since you can also trigger the execution via Jenkins ("Build now").
You might be able to combine the Jenkins monitor external job project type with a matrix project. At the very least the former will enable you to monitor the cron jobs individually.
Alternatively you could have the last monitored cron job of the day trigger building a project that checks the status of all the cron jobs (for example by retrieving and comparing the build numbers of the last and the last successful builds) and sends an email accordingly. The email plugin might be useful for the latter.
Check the CPAN or do some web digging for shell or perl script for managing cron jobs and extend its behaviour to do some reporting which you can render using HTML. Alternatively write a servlet and a some function calls to do just that.
This becomes your own standalone monitor application, which can sit in jenkins or deployed independently. If you choose to add it to jenkins, then add the reporting HTML file and its scripts to the container holding deployed web files for jenkins, word of advice place your files and script in a separate container.
Add a hyperlink to jenkins index html which will load your reporter. Now reboot tomcat and go from there.
Another option could be to take a look at Cronitor (https://cronitor.io). It basically boils down to being a tracking beacon that uses http requests to ping when a cron job/scheduled task starts and ends.
You'll be notified if your job doesn't run on schedule, or if it runs for too long/too short, etc. You can also configure it to send alerts to you via email, sms, but also Slack, Hipchat, Pagerduty and others.