I come across the terms "Compile time" and "Run time" pretty much everywhere when i learn about
azure pipelines.
However, i still didn't find a clear explanation about them.
I have found this page in Microsoft's documentation, but it doesn't explain these terms very clearly.
I would be happy if someone could explain these terms in the context of the whole
run sequence of Azure Pipelines.
Thanks!
When using YAML Azure Devops pipelines you have your pipelines as code definition. Compile time happens before runtime and you can pass parameters to your YAML before it is compiled (parsed in reality). It will evaluate expressions and replace them in your YAML before even starting any tasks. On runtime, the "compiled" yaml will try for example try to read variables from your Azure Devops pipeline.
Here is an example from Microsoft DOSC:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
Expressions are probably the most affected thing when it comes to differences between compile time and run time.
Also a pretty nice article about this:
https://adamtheautomator.com/azure-devops-variables-complete-guide/
Related
I want to change the process in my project and I cannot find anything online on the impact this change has on the pipelines, test suites, etc. Would really appreciate some input from someone more experienced than I am.
The migration to CMMI is not supported now. You will get the following error:
Basically, you should not have issues with test cases, test plans, because all process templates contain them. Pipelines and source code are not a part of process templates.
Many of my Azure ML Studio Designer pipelines began failing today. I was able to make a minimum repro:
Simply excluding columns with the Select Columns In Dataset node will fail with a JobConfigurationMaxSizeExceeded error.
This appears to be a bug introduced by Microsoft's rollout of their new Compute Common Runtime.
If I go into any nodes failing with the JobConfigurationMaxSizeExceeded exception and manually set AZUREML_COMPUTE_USE_COMMON_RUNTIME:false in their Environment JSON field, then they will subsequently work correctly. This is not documented anywhere that I could find, I stumbled over this fix through trial-and-error, and I wasted many hours trying to fix our failing pipelines today.
Does anyone know where I can find a list of possible effects of the Compute Common Runtime migration in Azure ML? I could not find any documentation on this and/or how it might affect existing Azure ML pipelines.
runtime environment variable should be set on the run configuration. property on the Environment object is deprecated.
https://learn.microsoft.com/en-us/python/api/azureml-core/azureml.core.runconfig.runconfiguration?view=azure-ml-py#variables
I would like to move the existing Azure DevOps pipelines to YAML based for obvious advantages. The problem is there are many of these and each one has many jobs.
When I click around in Azure DevOps, the "View YAML" link only appears for one job at a time. So that's gonna be a lot of manual work to view YAMLs for each pipeline x jobs and move that to code.
But for each pipeline there seems to be a way to "export" the entire pipeline in json. I was wondering if there is a similar way to at least dump the entire pipeline as YAML if not an entire folder.
If there is an API which exports the same then even better.
Until now, what we supported is what you see, use View YAML to copy/paste the definition of agent job. There has another workaround to get the entire definition of one pipeline is use the API to get the JSON from a build definition, convert it to YAML, tweak the syntax, then if needed, update the tasks which are referenced.
First, use Get Build Definition api to get the entire definition
of one pipeline.
Invoke JSON to YAML converter. Copy/paste the JSON of definition
into this converter.
Copy the YAML to a YAML editor of Azure Devops. Then the most important step is tweak the syntax.
Replace the refName key values with task names and version. For this, you can go our tasks source code which opened in github, built in tasks can be found there(note: please see the task.json file of corresponding task)
Noted: Use this method has another disadvantage that you need very familiar with YAML syntax so that you can tweak the content which convert from JSON successfully.
This is done and there is blog post about exporting pipelineas YAML on devblogs
It's it worth to mention that the new system knows how to handle every feature listed here:
Single and multiple jobs
Checkout options
Execution plan parallelism
Timeout and name metadata
Demands
Schedules and other triggers
Pool selection, including jobs which differ from the default
All tasks and all inputs, including optimizing for default inputs
Job and step conditions
Task group unrolling
In fact, there are only two areas which we know aren’t covered.
Variables
Variables defined in YAML “shadow” (hide) variables set in the UI. Therefore, we didn’t want to export them into the YAML file in case you need an ability to alter them at runtime. If you have UI variables in your Classic pipeline, we mention them by name in the comments to remind you that you need to configure them in your new YAML pipeline definition.
Timezone translation
cron schedules in YAML are in UTC, while Classic schedules are in the organization’s timezone. Timezone handling has a lot of sharp edges, so we opted not to try to be clever. We export the schedule without doing any translation, so your scheduled builds might be off by a certain number of hours unless you manually modify them. Here again, we make a note in the comments to remind you.
But there won't be support for release pipelines:
No plans to do so. Classic RM pipelines are different enough in their execution that I can’t make the same strong guarantees about correctness as I can with classic Build. Also, a number of concepts were re-thought between RM and unified YAML pipelines. In some cases, there isn’t a direct translation for an RM feature. A human is required to think about what the pipeline is designed to accomplish and re-implement it using new constructs.
I tried yamlizr https://github.com/f2calv/yamlizr
It works pretty well for exporting Release Pipelines, except it doesn't export out Pre/Post deployment conditions. We use these for Approval gates. So hopefully in a future release it will be supported.
But per Microsoft they won't support the export to YAML for Release Pipelines it sounds like.
https://devblogs.microsoft.com/devops/replacing-view-yaml/#comment-2043
So I am getting confused with what is the right approach to implementing code that fires on a scheduled bases within azure.
Originally we were using a standard console app that would be put in the webjob folder on deployment. I found this a bit noddy as we had logic looping and waiting for the right time to fire.
I then tried the azure webjob package https://github.com/Azure/azure-webjobs-sdk-extensions, but see this has gone quiet and the master branch is currently broken! I like because it has a CRON type approach with a function.cs, but now not sure if this is being maintained.
So do people have a preference on how a background process would run, e.g. a scheduled task that would run at 2am every day against a database?
Too much choice and not enough consensus on what the right way is?
Much appreciated in advance
I can think of three options, all of which are valid and can suit your needs. Which one to choose in the end comes down to your requirement specifics and your technical expertise.
WebJobs. These are the most powerful and most difficult to build and maintain. You typically use a dedicated project template in Visual Studio to author these. You can ignore that GitHub link - that's not what you need. Make sure you have the Azure workload enabled in Visual Studio and create a WebJob project.
Azure Functions. These are a more lightweight alternative to WebJobs. There is Visual Studio tooling available for this as well but you also have the option of writing your code directly in the portal. Azure Functions will time out after some period of time, so if your job runs more more than a minute or two this might not be the best option.
Logic Apps. This is more of a power user tool with an easy to use (debatable) designer interface. But it's also incredibly powerful and you can call WebJobs or Functions if you need to from a Logic App.
I could add links but I'm sure you could find them easily enough.
There are tons of resources online on how to replace JSON configuration files in a release pipeline like this one. I configured this. It works. However, we have multiple integration tests which reach the database too. These tests are run during build time. I haven't seen any option yet to replace config values in the build pipeline. Does it exist? Or do I really have to use this custom task (see screenshot below)?
There is an out-of-the-box task since recently by Microsoft. It's called File Transform. It's currently in preview but it works really well! Haven't had any issues whatsoever with it and it works the same as you would configure it in the release pipeline. Would recommend this any day!
Below you can see my configuration.
There is no out-of-the-box task only to replace tokens/values in files (also in the release pipline the task is Azure App Service Deploy and not only for replace json configuration).
You need to use an external extension from here or write a PowerShell script for that.