I have a data factory that I would like to publish, however I want to delay one of the pipelines from running as it uses a shared resource that isn't quite ready.
If possible I would like to allow the previous pipelines to run and then enable the downstream pipeline when the resource is ready for it.
How can I disable a pipeline so that I can re-enable it at a later time?
Edit your trigger and make sure Activated is checked NO. And of course don't forget to publish your changes!
Its not really possible in ADF directly. However, I think you have a couple of options to dealing with this.
Option 1.
Chain the datasets in the activities to enforce a fake dependency making the second activity wait. This is a bit clunky and requires the provisioning of fake datasets. But could work.
Option 2.
Manage it at a higher level with something like PowerShell.
For example:
Use the following cmdlet to check the status of the first activity and wait maybe in some sort of looping process.
Get-AzureRmDataFactoryActivityWindow
Next, use the following cmdlet to pause/unpause the downstream pipeline as required.
Suspend-AzureRmDataFactoryPipeline
Hope this helps.
You mentioned publishing, so if you are publishing trough Visual Studio, it is possible to disable a pipeline by setting its property "isPaused" to true in .json pipeline configuration file.
Property for making pipeline paused
You can disable pipeline by clicking Monitor & Manage in Data Factory you are using. Then click on the pipeline and in the upper left corner you have two options:
Pause: Will not terminate current running job, but will not start next
Terminate: Terminates all job instances (as well as not starting future ones)
GUI disabling pipeline
(TIP: Paused and Terminated pipeline have orange color, resumed have green color)
Use the powershell cmdlet to check the status of the activity
Get-AzureRmDataFactoryActivityWindow
Use the powershell cmdlet to pause/unpause a pipeline as required.
Suspend-AzureRmDataFactoryPipeline
Right click on the pipeline in the "Monitor and Manage" application and select "Pause Pipeline".
In case you're using ADF V2 and your pipeline is scheduled to run using a trigger, check which trigger your pipeline uses. Then go to the Manage tab and click on Author->Triggers. There you will get an option to stop the trigger. Publish the changes once you've stopped the trigger.
Related
I have an pipeline which will have few task mentioned in the image. I'm creating a bug work item when a particular task failed which is working fine using logic app.
Now my problem is I don't want to add every time new task for bug creation after each deployment task mentioned in the image.
Is there any way I can create only one bug work item based on failure in any of the task in the pipeline. may be in the last or somewhere..?
Not sure why you had to go the Logic app route as there is an option to do this with Azure Pipelines itself out of the box.
Navigate to {your pipeline} > Options as shown below:
If the build pipeline fails, you can automatically create a work item to track getting the problem fixed. You can specify the work item type. You can also select if you want to assign the work item to the requestor. For example, if this is a CI build, and a team member checks in some code that breaks the build, then the work item is assigned to that person.
Additional Fields: You can also set the value of other work item fields. For example:
Field Value
------- -------
System.Title Build $(Build.BuildNumber) failed
System.Reason Build failure
Check Build Options for more details.
UPDATE:
Doing this for Release Pipelines is not supported as an out of the box feature as of today. However, there are extensions available in the Visual Studio marketplace that can be used as alternatives until it is supported.
Here are two such extensions:
Create Bug on Release failure
Create Work Item
Another idea with PowerShell tasks is discussed here.
So to give you a bit of context we have a service which has been split into two different services ie one for the read and one for the write side operations. The read side is called ProductStore and the write side is called ProductCatalog. The issue were facing was down the write side as the load tests create 100 products in the write side resource web app and then they are transferred to the read side for the load test to then read x number of times. If a build is launched in the product catalog because something new was merged to master then this will cause issues in the product store pipeline if it gets run concurrently.
The question I want to ask is there a way in the ProductStore yaml file to directly query via a specified azure task or via an AzurePowershell script to check if a build is currently running in the ProductCatalog pipeline.
The second part of this would be to loop/wait until that pipeline has successfully finished before resuming the product store pipeline.
Hope this is clear as I'm not sure how to best ask this question as I'm very new to the DevOps pipelines flow but this would massively help if there was a good way of checking this sort of thing.
As a workaround , you can set Pipeline completion trigger in ProductStore pipeline.
To trigger a pipeline upon the completion of another, specify the triggering pipeline as a pipeline resource.
Or configure build completion triggers in the UI, choose Triggers from the settings menu, and navigate to the YAML pane.
I would like to move the existing Azure DevOps pipelines to YAML based for obvious advantages. The problem is there are many of these and each one has many jobs.
When I click around in Azure DevOps, the "View YAML" link only appears for one job at a time. So that's gonna be a lot of manual work to view YAMLs for each pipeline x jobs and move that to code.
But for each pipeline there seems to be a way to "export" the entire pipeline in json. I was wondering if there is a similar way to at least dump the entire pipeline as YAML if not an entire folder.
If there is an API which exports the same then even better.
Until now, what we supported is what you see, use View YAML to copy/paste the definition of agent job. There has another workaround to get the entire definition of one pipeline is use the API to get the JSON from a build definition, convert it to YAML, tweak the syntax, then if needed, update the tasks which are referenced.
First, use Get Build Definition api to get the entire definition
of one pipeline.
Invoke JSON to YAML converter. Copy/paste the JSON of definition
into this converter.
Copy the YAML to a YAML editor of Azure Devops. Then the most important step is tweak the syntax.
Replace the refName key values with task names and version. For this, you can go our tasks source code which opened in github, built in tasks can be found there(note: please see the task.json file of corresponding task)
Noted: Use this method has another disadvantage that you need very familiar with YAML syntax so that you can tweak the content which convert from JSON successfully.
This is done and there is blog post about exporting pipelineas YAML on devblogs
It's it worth to mention that the new system knows how to handle every feature listed here:
Single and multiple jobs
Checkout options
Execution plan parallelism
Timeout and name metadata
Demands
Schedules and other triggers
Pool selection, including jobs which differ from the default
All tasks and all inputs, including optimizing for default inputs
Job and step conditions
Task group unrolling
In fact, there are only two areas which we know aren’t covered.
Variables
Variables defined in YAML “shadow” (hide) variables set in the UI. Therefore, we didn’t want to export them into the YAML file in case you need an ability to alter them at runtime. If you have UI variables in your Classic pipeline, we mention them by name in the comments to remind you that you need to configure them in your new YAML pipeline definition.
Timezone translation
cron schedules in YAML are in UTC, while Classic schedules are in the organization’s timezone. Timezone handling has a lot of sharp edges, so we opted not to try to be clever. We export the schedule without doing any translation, so your scheduled builds might be off by a certain number of hours unless you manually modify them. Here again, we make a note in the comments to remind you.
But there won't be support for release pipelines:
No plans to do so. Classic RM pipelines are different enough in their execution that I can’t make the same strong guarantees about correctness as I can with classic Build. Also, a number of concepts were re-thought between RM and unified YAML pipelines. In some cases, there isn’t a direct translation for an RM feature. A human is required to think about what the pipeline is designed to accomplish and re-implement it using new constructs.
I tried yamlizr https://github.com/f2calv/yamlizr
It works pretty well for exporting Release Pipelines, except it doesn't export out Pre/Post deployment conditions. We use these for Approval gates. So hopefully in a future release it will be supported.
But per Microsoft they won't support the export to YAML for Release Pipelines it sounds like.
https://devblogs.microsoft.com/devops/replacing-view-yaml/#comment-2043
I've created a new onetime pipeline in Azure Data Factory using Copy Data wizard.
The pipeline has 4 activities and it was run just fine - all 4 activities succeeded.
Now I'd like to rerun the pipeline. So I do:
Clone the pipeline.
Change name to [name]_rev2.
Remove start and end properties.
Deploy the cloned pipeline.
Now the status of the new cloned pipeline is Running.
But no activities are executed at all.
What's wrong?
Mmmmm. Where to start!
In short. You can't just clone the pipeline. If its a one time data factory that you've created it won't have a schedule attached and therefore won't have any time slices provisioned that now require execution.
If your unsure how time slices in ADF work a recommend some reading around this concept.
Next, I recommend opening up a Visual Studio 2015 solution and downloading the data factory that did run as a project. Check out the various JSON blocks for scheduling and interval availability in the datasets, activities and the time frame in the pipeline.
Here's a handy guide for scheduling and execution to understand how to control your pipelines.
https://learn.microsoft.com/en-us/azure/data-factory/data-factory-scheduling-and-execution
Once you've made all the required changes (not just the name) publish the project to a new ADF service.
Hope this helps
We have an azure webjob that was deployed as on demand. Is there a way to change this to run on a schedule without redeploying?
Not a lot of info out there on this.
I tried creating a new schedule collection like this and adding a job to run the existing webjob, but that didn't seem to work either.
I prefer to do this in the GUI portal, but if its not possible, I'll do it in powershell (if it is possible like that).
(Also, if it can only be changed by redeploying, I need to know that and it effectively answers the question)
To easily add a schedule to your triggered (on demand) webjob add a file called settings.job at the root of your webjob with this content:
{"schedule": "the schedule as a cron expression"}
Find out more about this here
Note: it'll only work properly for Standard or Premium sites and requires you to set the site as always on.
In the end the link in the original post I referenced worked. The thing that was missing for me was the understanding that creating a trigger job in scheduler will not affect the run status (on-demand or scheduled) of the web job itself. In my case it stayed "on-demand", but the schedule was in fact running it.
This should point you in the direction on how to do this via PowerShell. It looks possible to add already existing WebJobs to a scheduler.
Create a Scheduled Azure WebJob with PowerShell