For what I can understand one can build Logic Apps with Terraform. However, the docs are still not very good, and it looks like this feature is pretty new.
What are the limitations when it comes to TF and Azure Logic Apps? Are there any?
I want to build a two apps, one that is triggered every month and another that is triggered by a https request. I want these then to run two python scripts, and I want the later one to return the result from this script to the client that called the https.
Is this possible to automate in Terraform? At this moment, there are very little examples and documentation on this. Any comment or tip is helpful and greeted with open arms!
You can create a blank Logic App instance through Terrform (TF). But, if you want to add triggers and actions, I wouldn't recommend using TF at all, as of the provider version of 1.20.0.
TF lacks document around parameters. As you know there are two parameters properties – right under the properties property and right under the definitions property. This document states parameters but it doesn't clearly say which one. I'm guessing this refers to the one under the definitions property, but it actually doesn't work – throws Invalid Template error without enough explanation.
UPDATE: I just reverse engineered by importing a Logic App instance using terraform import. The parameters is actually pointing to the one under the properties property. However, it still doesn't work as the Logic App's parameter value can be anything – object, string, integer, etc, while TF's parameter expects string only. Also, there is no way to create parameters under the definitions property.
TF only supports two triggers – HTTP trigger and Timer trigger. All other triggers should use the azurerm_logic_app_trigger_custom resource, but it requires the body part to manually write a JSON object or import from a file, which can't be parameterised through variables or locals.
TF only supports one action – HTTP action. All other actions should use the azurerm_logic_app_action_custom resource, but, like the same issue above, it's not that useful.
In conclusion, TF lacks supports parameters, triggers and actions. So, unless you just create a blank Logic App instance, TF wouldn't be an option for Logic Apps. If you still want to create a blank Logic App instance with TF, then I would recommend this approach using Azure PowerShell or Azure CLI.
For clarity, you don't use Terraform to create LogicApps. LogicApps are designed in either the Portal or Visual Studio.
Terraform is a deployment/management tool. You can almost surely deploy your LogicApps, and other resources, with Terraform, but they're are already created.
isn't the point of terraform to stand up resources across various environments just by passing in -var environment=qa to create a qa instance of the logic app? prod? uat? marcplaypen? I was hoping to use terraform import to create the terraform file, then create multiple versions of it. I can do it, but not with any parameters, which breaks one of my 'actions'.
I was using a combo of terraform import and logic app code view. most of my actions are pretty much a combo of copying the json block for each action, and modifying based on the first entry of the 'body' of the action generated from terraform import.
Then setting up the dependencies manually based off of runAfter, which tells me what an action is dependent on.
but, it fails on parameters, complaining there's only these declared parameters for my definition are ''.'
Related
While using runtime parameters in Azure DevOps build pipeline, is there a way to mask the values of the parameters in pipeline logs ?
Based on my research and going through the documentation, there does not seem to be a way to achieve this (at the time of writing this). There are alternatives like using variable groups/secret variables but since the parameter values are user provided and would change for each pipeline trigger, such options are not ideal.
If the parameters values cannot be masked, can we turn off pipeline logs altogether?
During our tests, we tried to define a variable as secret with the same value to the parameter, and in the pipeline logs, the value of the parameters could not be echoed anymore.
Even though the value is still observed in the ui triggering. So for further secure setting, we suppose that you could set more specific permission for the pipeline or the yaml repo, to limit the accessibility. And you could also create a Feature Request to raise your concern with more developers.
==========================================================
11/7 4:20PM UTC+8 Update.
Hi Vineet,
If you are going to hide the logs, you could try to limit the accessibility for the pipeline builds. Even though it could not be achieved directly via UI setting.
I'm coming from a long SSIS background, we're looking to use Azure data factory v2 but I'm struggling to find any (clear) way of working with multiple environments. In SSIS we would have project parameters tied to the Visual Studio project configuration (e.g. development/test/production etc...) and say there were 2 parameters for SourceServerName and DestinationServerName, these would point to different servers if we were in development or test.
From my initial playing around I can't see any way to do this in data factory. I've searched google of course, but any information I've found seems to be around CI/CD then talks about Git 'branches' and is difficult to follow.
I'm basically looking for a very simple explanation and example of how this would be achieved in Azure data factory v2 (if it is even possible).
It works differently. You create an instance of data factory per environment and your environments are effectively embedded in each instance.
So here's one simple approach:
Create three data factories: dev, test, prod
Create your linked services in the dev environment pointing at dev sources and targets
Create the same named linked services in test, but of course these point at your tst systems
Now when you "migrate" your pipelines from dev to test, they use the same logical name (just like a connection manager)
So you don't designate an environment at execution time or map variables or anything... everything in test just runs against test because that's the way the linked servers have been defined.
That's the first step.
The next step is to connect only the dev ADF instance to Git. If you're a newcomer to Git it can be daunting but it's just a version control system. You save your code to it and it remembers every change you made.
Once your pipeline code is in git, the theory is that you migrate code out of git into higher environments in an automated fashion.
If you go through the links provided in the other answer, you'll see how you set it up.
I do have an issue with this approach though - you have to look up all of your environment values in keystore, which to me is silly because why do we need to designate the test servers hostname everytime we deploy to test?
One last thing is that if you a pipeline that doesn't use a linked service (say a REST pipeline), I haven't found a way to make that environment aware. I ended up building logic around the current data factories name to dynamically change endpoints.
This is a bit of a bran dump but feel free to ask questions.
Although it's not recommended - yes, you can do it.
Take a look at Linked Service - in this case, I have a connection to Azure SQL Database:
You have possibilities to use dynamic content for either the server name and database name.
Just add a parameter to your pipeline, pass it to the Linked Service and use in the required field.
Let me know whether I explained it clearly enough?
Yes, it's possible although not so simple as it was in VS for SSIS.
1) First of all: there is no desktop application for developing ADF, only the browser.
Therefore developers should make the changes in their DEV environment and from many reasons, the best way to do it is a way of working with GIT repository connected.
2) Then, you need "only":
a) publish the changes (it creates/updates adf_publish branch in git)
b) With Azure DevOps deploy the code from adf_publish replacing required parameters for target environment.
I know that at the beginning it sounds horrible, but the sooner you set up an environment like this the more time you save while developing pipelines.
How to do these things step by step?
I describe all the steps in the following posts:
- Setting up Code Repository for Azure Data Factory v2
- Deployment of Azure Data Factory with Azure DevOps
I hope this helps.
I would like to move the existing Azure DevOps pipelines to YAML based for obvious advantages. The problem is there are many of these and each one has many jobs.
When I click around in Azure DevOps, the "View YAML" link only appears for one job at a time. So that's gonna be a lot of manual work to view YAMLs for each pipeline x jobs and move that to code.
But for each pipeline there seems to be a way to "export" the entire pipeline in json. I was wondering if there is a similar way to at least dump the entire pipeline as YAML if not an entire folder.
If there is an API which exports the same then even better.
Until now, what we supported is what you see, use View YAML to copy/paste the definition of agent job. There has another workaround to get the entire definition of one pipeline is use the API to get the JSON from a build definition, convert it to YAML, tweak the syntax, then if needed, update the tasks which are referenced.
First, use Get Build Definition api to get the entire definition
of one pipeline.
Invoke JSON to YAML converter. Copy/paste the JSON of definition
into this converter.
Copy the YAML to a YAML editor of Azure Devops. Then the most important step is tweak the syntax.
Replace the refName key values with task names and version. For this, you can go our tasks source code which opened in github, built in tasks can be found there(note: please see the task.json file of corresponding task)
Noted: Use this method has another disadvantage that you need very familiar with YAML syntax so that you can tweak the content which convert from JSON successfully.
This is done and there is blog post about exporting pipelineas YAML on devblogs
It's it worth to mention that the new system knows how to handle every feature listed here:
Single and multiple jobs
Checkout options
Execution plan parallelism
Timeout and name metadata
Demands
Schedules and other triggers
Pool selection, including jobs which differ from the default
All tasks and all inputs, including optimizing for default inputs
Job and step conditions
Task group unrolling
In fact, there are only two areas which we know aren’t covered.
Variables
Variables defined in YAML “shadow” (hide) variables set in the UI. Therefore, we didn’t want to export them into the YAML file in case you need an ability to alter them at runtime. If you have UI variables in your Classic pipeline, we mention them by name in the comments to remind you that you need to configure them in your new YAML pipeline definition.
Timezone translation
cron schedules in YAML are in UTC, while Classic schedules are in the organization’s timezone. Timezone handling has a lot of sharp edges, so we opted not to try to be clever. We export the schedule without doing any translation, so your scheduled builds might be off by a certain number of hours unless you manually modify them. Here again, we make a note in the comments to remind you.
But there won't be support for release pipelines:
No plans to do so. Classic RM pipelines are different enough in their execution that I can’t make the same strong guarantees about correctness as I can with classic Build. Also, a number of concepts were re-thought between RM and unified YAML pipelines. In some cases, there isn’t a direct translation for an RM feature. A human is required to think about what the pipeline is designed to accomplish and re-implement it using new constructs.
I tried yamlizr https://github.com/f2calv/yamlizr
It works pretty well for exporting Release Pipelines, except it doesn't export out Pre/Post deployment conditions. We use these for Approval gates. So hopefully in a future release it will be supported.
But per Microsoft they won't support the export to YAML for Release Pipelines it sounds like.
https://devblogs.microsoft.com/devops/replacing-view-yaml/#comment-2043
Being novice to ADF CICD i am currently exploring how we can update the pipeline scoped parameters when we deploy the pipeline from one enviornment to another.
Here is the detailed scenario -
I have a simple ADF pipeline with a copy activity moving files from one blob container to another
Example - Below there is copy activity and pipeline has two parameters named :
1- SourceBlobContainer
2- SinkBlobContainer
with their default values.
Here is how the dataset is configured to consume these Pipeline scoped parameters.
Since this is development environment its OK with the default values. But the Test environment will have the containers present with altogether different name (like "TestSourceBlob" & "TestSinkBlob").
Having said that, when CICD will happen it should handle this via CICD process by updating the default values of these parameters.
When read the documents, no where i found to handle such use-case.
Here are some links which i referred -
http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html
https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment
Thoughts on how to handle this will be much appreciated. :-)
There is another approach in opposite to ARM templates located in 'ADF_Publish' branch.
Many companies leverage that workaround and it works great.
I have spent several days and built a brand new PowerShell module to publish the whole Azure Data Factory code from your master branch or directly from your local machine. The module resolves all pains existed so far in any other solution, including:
replacing any property in JSON file (ADF object),
deploying objects in an appropriate order,
deployment part of objects,
deleting objects not existing in the source any longer,
stop/start triggers, etc.
The module is publicly available in PS Gallery: azure.datafactory.tools
Source code and full documentation are in GitHub here.
Let me know if you have any question or concerns.
There is a "new" way to do ci/cd for ADF that should handle this exact use case. What I typically do is add global parameters and then reference those everywhere (in your case from the pipeline parameters). Then in your build you can override the global parameters with the values that you want. Here are some links to references that I used to get this working.
The "new" ci/cd method following something like what is outlined here Azure Data Factory CI-CD made simple: Building and deploying ARM templates with Azure DevOps YAML Pipelines. If you have followed this, something like this should work in your yaml:
overrideParameters: '-dataFactory_properties_globalParameters_environment_value "new value here"'
Here is an article that goes into more detail on the overrideParameters: ADF Release - Set global params during deployment
Here is a reference on global parameters and how to get them exposed to your ci/cd pipeline: Global parameters in Azure Data Factory
I am confused about where to use a variable and where to use a parameter in ARM templates. How do we make this call ?
The referenced script uses both. I am more curious of the justification of using variables.
Reference
Sample Service Fabric Azure Deploy Script
https://raw.githubusercontent.com/azure/azure-quickstart-templates/master/service-fabric-oms/azuredeploy.json
In the Azure template json file:
parameters:Values that are provided when deployment is executed to customize resource deployment.
variables:Values that are used as JSON fragments in the template to simplify template language expressions.
More information please refer to this official document:Understand the structure and syntax of Azure Resource Manager templates.
I am more curious of the justification of using variables.
Based on my experience, if you only use the variable once, you don't need use variables. But if you want to use the variable multiple times, you had better use variables. Using variable can simplify your template to avoid duplication of content.
For example, if you don't use supportLogStorageAccountName more than once, you can just do:
"name": "[toLower(concat('sf', uniqueString(resourceGroup().id),'2'))]"
However if you use provide variable supportLogStorageAccountName several\many times, you can use variable to avoid duplication.
ARM templates are usually used to create a set of close to identical environments. The parameters are what differs them. This is commonly used for environment type (prod, dev, test) and performance/cost related parameters.
Variables are used to create unique names for services based on or calculated from the parameters.
An example of this would the name of a storage account. This is usually done by concatenating a common name like _storage and an environment name parameter like “test” and store it in a variable. When you create another environment all you must do is change the environment type parameter.