We are in the process of setting up new project. Our requirements is to invoke multiple rest API's and aggregate the response and send it back to mobile client.
We are exploring these 2 options for our experience layer(Integration )
1. Logic Apps
2. Azure Function
We have observed one major difference with respect to performance between these two.
We run through simple use case to compare the performance.
we are just invoking a rest API to get some metrics with different options available
Just integrate with APIM as back-end service
Using Azure Function
Using Logic Apps
Below are the metrics
Logic app is taking longer time for execution compare to other options. Below is the simple logic app to invoke rest api
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"GetReferenceData": {
"inputs": {
"headers": {
"Authorization": "#variables('AuthToken')"
},
"method": "GET",
"uri": "url"
},
"runAfter": {
"Initialize_AuthToken": [
"Succeeded"
]
},
"type": "Http"
},
"Initialize_AuthToken": {
"inputs": {
"variables": [
{
"name": "AuthToken",
"type": "string",
"value": "#{triggerOutputs()['headers']?['Access-Token']}"
}
]
},
"runAfter": {},
"type": "InitializeVariable"
},
"Response": {
"inputs": {
"body": "#body('GetReferenceData')",
"statusCode": "#outputs('GetReferenceData')['statusCode']"
},
"kind": "Http",
"runAfter": {
"GetReferenceData": [
"Succeeded"
]
},
"type": "Response"
},
"Response_2": {
"inputs": {
"body": "#body('GetReferenceData')",
"statusCode": "#outputs('GetReferenceData')['statusCode']"
},
"kind": "Http",
"runAfter": {
"GetReferenceData": [
"Failed",
"Skipped",
"TimedOut"
]
},
"type": "Response"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {
"storageLocation": {
"defaultValue": [],
"type": "Array"
}
},
"triggers": {
"manual": {
"inputs": {
"method": "GET",
"relativePath": "/referenceData",
"schema": {}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {}
}
We have so many use cases where we need to invoke the multiple rest API's and aggregate the result. With the above numbers it seems Function App is doing a way better job than Function App.For parallel operations i may rely upon durable functions over Logic apps.
So i just want to understand why logic app is taking longer time almost double time compared to function for the similar operation?
Is logic app is not meant for these operations?
Refer another thread where this question is already answered. Is Logic Apps performance slower compared to a direct .NET REST Call?
Following addition may provide more insight
“Azure Functions is code being triggered by an event, whereas Logic Apps is a separate framework of workflow triggered by an event.”
The Logic App is a logical container for one workflow you can define using triggers and actions.
A Logic App runs on a set of infrastructure of an Azure region (VM’s in a data centre), and it consists of several components not visible to you as it is abstracted away. By provisioning a Logic App, you leverage a bit of that infrastructure (indirectly via the Logic App Service) once you define a workflow and the flow gets triggered.
Azure Functions are part of the Azure Web + Mobile suite of App Services and are designed to enable the creation of small pieces of meaningful, reusable methods, easily shared across services.
Related
I created a Service Health alert to fire whenever Azure Data Factory was experiencing any issue from their side.
I have been creating alerts for other services of Azure but somehow only the one for Service Health is not working, even though the history log said that the alert was "fired" the last few days. I ran the test with "Service Health" sample type and it was also not working, even though for other sample types it did send me the test alert email.
My setup for the action group and the alert rule is really simple: send email notification whenever Service Health was showing issue with Data Factory. Here is the json code for the action group setup
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"actionGroups_Adf_alert_actiongroup_name": {
"defaultValue": "Adf-alert-actiongroup",
"type": "String"
}
},
"variables": {},
"resources": [
{
"type": "microsoft.insights/actionGroups",
"apiVersion": "2022-06-01",
"name": "[parameters('actionGroups_Adf_alert_actiongroup_name')]",
"location": "GermanyWestCentral",
"properties": {
"groupShortName": "Adf-alert-ac",
"enabled": true,
"emailReceivers": [
{
"name": "TeamName_-EmailAction-",
"emailAddress": "TeamName#mycompany.com",
"useCommonAlertSchema": false
},
{
"name": "MyName_-EmailAction-",
"emailAddress": "MyName#mycompany.com",
"useCommonAlertSchema": false
}
],
"smsReceivers": [],
"webhookReceivers": [],
"eventHubReceivers": [],
"itsmReceivers": [],
"azureAppPushReceivers": [],
"automationRunbookReceivers": [],
"voiceReceivers": [],
"logicAppReceivers": [],
"azureFunctionReceivers": [],
"armRoleReceivers": []
}
}
]
Any suggestion for what might have gone wrong in my setup? Many thanks!
I found out that "Enable the common alert schema" has to be turned on for the alert to work. Very simple but quite easy to overlook.
I have an Azure Stream Analytics job which adds JSON data to an Azure storage container as an output. The data is organized in a folder structure based on the date, as specified in the stream analytics output here:
ASA Output Details
I also have a Logic App which I want to add data to the same place. I am looking at the logic app Blob Storage actions and cant figure out how to do this. The Update Blob action seems to want to point to a single blob file, rather than having it integrated into the data based on date.
Is there a way to do this with the Logic Apps actions? Or maybe there is a better way to structure my data so that I can add events both from stream analytics as well as from logic apps?
Thanks!
To add blobs to the same folder structure you need to mention the path. To achieve your requirement I have used Create blob action and added blobs to the required folder.
In case if you are trying to automate this process to run everyday and add files to everyday to the date path, then try following the below flow.
You can reproduce the same in your environment using the below code view
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Compose": {
"inputs": {
"Sample": "Sample Json from Logic Apps"
},
"runAfter": {},
"type": "Compose"
},
"Create_blob_(V2)": {
"inputs": {
"body": "#outputs('Compose')",
"headers": {
"ReadFileMetadataFromServer": true
},
"host": {
"connection": {
"name": "#parameters('$connections')['azureblob']['connectionId']"
}
},
"method": "post",
"path": "/v2/datasets/#{encodeURIComponent(encodeURIComponent('AccountNameFromSettings'))}/files",
"queries": {
"folderPath": "/container1/#{formatDateTime(utcNow(),'yyyy')}/#{formatDateTime(utcNow(),'MM')}/#{formatDateTime(utcNow(),'dd')}",
"name": "LAEvent.json",
"queryParametersSingleEncoded": true
}
},
"runAfter": {
"Compose": [
"Succeeded"
]
},
"runtimeConfiguration": {
"contentTransfer": {
"transferMode": "Chunked"
}
},
"type": "ApiConnection"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {
"$connections": {
"defaultValue": {},
"type": "Object"
}
},
"triggers": {
"manual": {
"inputs": {
"schema": {}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {
"$connections": {
"value": {
"azureblob": {
"connectionId": "/subscriptions/<SUB_ID>/resourceGroups/<RG>/providers/Microsoft.Web/connections/azureblob",
"connectionName": "azureblob",
"id": "/subscriptions/<SUB_ID>/providers/Microsoft.Web/locations/centralus/managedApis/azureblob"
}
}
}
}
}
In Storage account:
Updated Answer
In the case where you want to add data to an existing blob, you can concat both the data which is already present in the blob and the new logic app's data using concat function.
concat(body('Get_blob_content_(V2)'),outputs('Compose'))
RESULTS:
Using logic apps to extract some logs from Okta and come across an interesting issue.
The okta management api paginates the responses and provide the link to the next list of paginated responses in a 'Link' header; however they also provide another 'Link' header which gives you the link to the current page.
Ie:
link : <https://dev1-web.okta.com/api/v1/logs?since=2022-07-07T20%3A19%3A.0837307Z&sortOrder=ASCENDING>; rel="self"
link : <https://dev1-web.okta.com/api/v1/logs?since=2022-07-07T20%3A19%3A.0837307Z&sortOrder=ASCENDING&after=1657289356194_1>; rel="next"
When retrieving the headers from the http call it merges them as:
<https://dev-web.okta.com/api/v1/logs?since=2022-07-07T20%3A19%3A.0837307Z&sortOrder=ASCENDING>; rel="self",<https://dev-web.okta.com/api/v1/logs?since=2022-07-07T20%3A19%3A.0837307Z&sortOrder=ASCENDING&after=1657289356194_1>; rel="next"
I am trying to find a way to pull the link header out which has the rel="next" wihtout having to add a load of condition logic to check which side of the split has the rel="next". Any ideas?
Since I don't find the response to be in any format, One of the workarounds that can meet the requirement is to extract the link using string functions such as lastIndexOf and slice functions in Compose connector. Here is my logic app flow.
Below is the expression I'm using in my compose connector.
slice(outputs('Compose'),add(lastIndexOf(outputs('Compose'),'<'),1),lastIndexOf(outputs('Compose'),'>'))
RESULTS:
Below is the code-view of my logic app
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Compose": {
"inputs": "link : <https://dev1-web.okta.com/api/v1/logs?since=2022-07-07T20%3A19%3A.0837307Z&sortOrder=ASCENDING>; rel=\"self\"\nlink : <https://dev1-web.okta.com/api/v1/logs?since=2022-07-07T20%3A19%3A.0837307Z&sortOrder=ASCENDING&after=1657289356194_1>; rel=\"next\"",
"runAfter": {},
"type": "Compose"
},
"Compose_2": {
"inputs": "#slice(outputs('Compose'),add(lastIndexOf(outputs('Compose'),'<'),1),lastIndexOf(outputs('Compose'),'>'))",
"runAfter": {
"Compose": [
"Succeeded"
]
},
"type": "Compose"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {},
"triggers": {
"manual": {
"inputs": {
"schema": {}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {}
}
Consider if you are receiving a JSON response, then you can parse the response and can extract it directly.
I have multiple IR for each client. I want to use one pipeline to copy data from client SAP system to Blob.
I have created all the configuration variables like in below code but i am not able to create parameter for Integration Runtime variable.
Is there any JSON syntax where we can make it dynamic.
{
"name": "LS_SAP_TBL",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"type": "SapTable",
"annotations": [],
"typeProperties": {
"clientId": "#{linkedService().ClientId}",
"language": "",
"sncMode": false,
"userName": "#{linkedService().userName}",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "KAJDSKJDHLSJDFHALKFD",
"type": "LinkedServiceReference"
},
"secretName": "#{linkedService().SecretName}"
},
"server": "#{linkedService().server}",
"systemNumber": "#{linkedService().systemNumber}"
},
"connectVia": {
"referenceName": "#IntegrationRuntime_Param - Need to pass this dynamically",
"type": "IntegrationRuntimeReference"
},
"parameters": {
"SecretName": {
"type": "String"
},
"ClientId": {
"type": "String"
},
"userName": {
"type": "String"
},
"server": {
"type": "String"
},
"systemNumber": {
"type": "String"
},
"IntegrationRuntime_Param": {
"type": "String"
}
}
}
}```
Unfortunately, as of now there in no option to parameterize the Integration Runtime.
You can parameterize the different parameters in a single IR to use it for multiple resources. But different IRs can't be parameterize.
Please refer this similar request on Microsoft Q&A for better understanding.
Need help on parameterization of Integration Runtime and Azure Key vault Secrets on link service in Azure Data Factory pipeline
Yes, I second UtkarshPal's response. I hope we get this option ASAP.
In our case, we do have multiple ADFs (one per project), and each ADF has it's own SHIR.
The only method to reuse linked services with self hosted IR is to give all SHIRs the same name..
P.S. - If we are having multiple SHIRs within a single ADF, then it's not possible to have same name.
I'm working on deploying an application to Azure Gov Cloud Stage & Prod. On Dev & QA we use APIM connected to AppInsights to gather statistics and generate alerts, however, AppInsights is not available in Stage Gov Cloud.
Is there any official timeline for availability of Application Insights in Gov Cloud on Stage?
As a workaround we planned to create an Application Insights resource on Dev (say AI-dev) which would be associated with APIM in Stage (say APIM-stage).
However, when we want to do the association, we go to APIM-stage in Azure Portal and try to select an Application Insights resource - there is none available, the existing AppInsights resources in Dev and QA are not visible from the the APIM in Stage.
Is it possible to configure Stage in such a way they will be visible? If yes, then how? We are looking for any means to make this association - either manually or automatically with an API.
Is there any other workaround available to gather application/APIM request statistics for Stage/Prod deployments? The ultimate goal is to have request alerts (e.g. for bad requests) working for Stage/Prod.
Here is what we've found after investigation:
Microsoft says that the target date for having Application Insights in Gov Cloud on Stage is end of Q4 2018 - as of 10/11/2018.
2./3.
It is possible to associate Application Insights located in Dev (Commercial Cloud) with APIM located in Gov Cloud Stage in two ways - either using VSTS pipeline tasks or REST API. It turns out that Azure Portal GUI will still not show any association or show an invalid association after doing that but the end result is it's working.
Method 1 (tested and working)
VSTS task:
task: AzureResourceGroupDeployment#2
VSTS task template:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"appinsights-name": {
"type": "string"
},
"instrumentation": {
"type": "string"
},
"apim-name": {
"type": "string"
},
"api-name": {
"type": "string"
}
},
"resources": [
{
"type": "Microsoft.ApiManagement/service/loggers",
"name": "[concat(parameters('apim-name'), '/', parameters('appinsights-name'))]",
"apiVersion": "2018-01-01",
"scale": null,
"properties": {
"loggerType": "applicationInsights",
"description": null,
"credentials": {
"instrumentationKey": "[parameters('instrumentation')]"
},
"isBuffered": true
}
},
{
"type": "Microsoft.ApiManagement/service/apis/diagnostics",
"name": "[concat(parameters('apim-name'), '/', parameters('api-name'), '/', 'applicationinsights')]",
"apiVersion": "2018-01-01",
"scale": null,
"properties": {
"enabled": true
}
},
{
"type": "Microsoft.ApiManagement/service/apis/diagnostics/loggers",
"name": "[concat(parameters('apim-name'), '/', parameters('api-name'), '/', 'applicationinsights', '/',parameters('appinsights-name'))]",
"apiVersion": "2018-01-01",
"scale": null,
"properties": {
"loggerType": "applicationInsights",
"description": null,
"credentials": {
"instrumentationKey": "[parameters('instrumentation')]"
},
"isBuffered": true,
"resourceId": "[parameters('appinsights-name')]"
},
"dependsOn": [
"[resourceId('Microsoft.ApiManagement/service/apis/diagnostics', parameters('apim-name'), parameters('api-name'), 'applicationinsights')]"
]
}
]
}
Method 2 (not tested)
PUT https://management.usgovcloudapi.net/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ApiManagement/service/{apimServiceName}/loggers/applicationinsights?api-version=2018-01-01 HTTP/1.1
Authorization: Bearer
Content-Type: application/json
{
"properties": {
"loggerType": "applicationinsights",
"description": null,
"isBuffered": true,
"resourceId": null,
"credentials":{
"instrumentationKey":"<ApplicationInsights-InstrumentationKey>"
}
}
}
PUT https://management.usgovcloudapi.net/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ApiManagement/service/{apimServiceName}/diagnostics/applicationinsights?api-version=2018-01-01 HTTP/1.1
Authorization: Bearer
Content-Type: application/json
{
"properties": {
"enabled": true
}
}