I have a dynamic list for which I'd like to run a given job for each item in the list.
Basically this is a list of tags (~20) for a docker container that would ideally all be processed concurrently. As the list of tags can change I'd like to define the list once and consume it in multiple places, including in the jobs definition.
On Travis-CI this is known as Matrix Expansion, but I've been unable to find a similar feature on Gitlab CI.
use the keyword parallel:matrix
Related
I have a practical use case. three notebooks (pyspark) all have one common parameter.
need to schedule all three notebooks in a sequence.
is there any way to run them by setting one parameter value, as they are same in all?
please suggest the best way to do it.
I'm trying to launch an heterogeneous job with SLURM using an arbitrary distribution (--distribution=arbitrary) defined in an host file. It seems that on SLURM there isn't a specific SLURM_HOSTFILE environment variable for each HET_GROUP so I tried to define it in one file including all the tasks in the groups.
The problem is that SLURM_JOB_NUM_NODES is equal to SLURM_JOB_NUM_NODES_HET_GROUP_0 so I get an error:
Requested node configuration is not available
Did someone manage to use an arbitrary distribution with heterogeneous jobs?
I'm using the counter function in Azure Pipelines to generate the version patch, allowing me to get an autoincremented version number as part of my CI/CD.
version.major: 4
version.minor: 2
version.patch: $[counter(format('{0}.{1}', variables['version.major'], variables['version.minor']), 0)]
version.full: $[format('{0}.{1}.{2}', variables['version.major'], variables['version.minor'], variables['version.patch'])]
However, I now need this version counter to be shared between two pipelines in the same repository. One of the pipelines publishes some packages, whilst the other deploys some resources. I'd like the pipelines to be separate since sometimes one would need to be run without the other – for example, because they're triggered for different path filters. However, I would like them both to contribute to the same versioning sequence – I don't want my packages to have version numbers disparate from the resources.
Is there a way of achieving this shared versioning sequence? I've already tried variable reuse, but the counter gets evaluated in the outer pipeline, not the inner template where it's defined.
What you need is passing variables from one pipeline to another. I'm afraid it can not be achieved by default.
As a workaround you could create a variable group. Then every time there is a new version number in first pipeline, update the value of the variables inside your variable group. Then in second pipeline use the variables in the variable group.
There are multiple ways to update variable group, Rest API, powershell, 3rd-party extension. Detail ways please refer answers in this question: How to Increase/Update Variable Group value using Azure Devops Build Definition?
I would be happy if there was the ability to loop tasks in pipeline especially "for each" loop.
sometimes I have a task that I need to run a few time with different parameters.
this different parameters i take it from json file each block in json file give me different parameter
how i can build the tasks in azure with the number of block in json file with different parameter
Currently there is no way of creating loop of tasks naively in Azure DevOps UI.
If you want to achieve such a results I suggest to change you pipeline from Visual/UI to Yaml Pipelines. Than you can just use each expression. Otherwise, you should be write script block that will run you task in a loop or parallel.
You can be also interested in that post: Looping Problem in Azure DevOps
I need to run identical jobs in schedule, and they differ only in few strings.
As you may know, there is no a convenient way to create identical jobs with different parameters. For now i prefer so "codeless" way to do so, or with "as less code as possilbe".
So lets imagine they are stored in a rows of JobsConfigurations table of the website-related database.
How I can get the Job name of job being running to pick the right configuration from the table?
Thanks for help!
See https://github.com/projectkudu/kudu/wiki/Web-Jobs#environment-settings
The WEBJOBS_NAME environment variable will give you the name of the current WebJob.