I have a stream anaytics job which gets data from IoT hub and sends it to PowerBi and azure data lake. Writing data azure data lake works alone perfectly but when I try to stream data to powerbi my steam analytics job fails after writing first out put and I am getting following error log.
{
"channels": "Operation",
"correlationId": "ff2593c9-cda2-498c-8e1f-6f13260ae32f",
"description": "Stream Analytics job status changed for last user action [Start job 'AVLStreamAnalytics'].",
"eventDataId": "953665c1-aec0-482d-852c-fdf3160cd6fb",
"eventName": {
"value": "Start job 'AVLStreamAnalytics'",
"localizedValue": "Start job 'AVLStreamAnalytics'"
},
"category": {
"value": "Administrative",
"localizedValue": "Administrative"
},
"eventTimestamp": "2017-07-13T16:35:46.2226177Z",
"id": "/subscriptions/161fc731-59a3-4173-9cf5-866b3188abf3/resourceGroups/AVLPOC/providers/Microsoft.StreamAnalytics/streamingjobs/AVLStreamAnalytics/events/953665c1-aec0-482d-852c-fdf3160cd6fb/ticks/636355605462226177",
"level": "Error",
"operationId": "2e5ed092-bb88-4dde-8e9f-2a9c0adc1f78",
"operationName": {
"value": "Start job 'AVLStreamAnalytics'",
"localizedValue": "Start job 'AVLStreamAnalytics'"
},
"resourceGroupName": "AVLPOC",
"resourceProviderName": {
"value": "Microsoft.StreamAnalytics",
"localizedValue": "Microsoft.StreamAnalytics"
},
"resourceType": {
"value": "Microsoft.StreamAnalytics/streamingjobs",
"localizedValue": "Microsoft.StreamAnalytics/streamingjobs"
},
"resourceId": "/subscriptions/161fc731-59a3-4173-9cf5-866b3188abf3/resourceGroups/AVLPOC/providers/Microsoft.StreamAnalytics/streamingjobs/AVLStreamAnalytics",
"status": {
"value": "Failed",
"localizedValue": "Failed"
},
"subStatus": {
"value": "",
"localizedValue": ""
},
"submissionTimestamp": "2017-07-13T16:35:59.7904001Z",
"subscriptionId": "161fc731-59a3-4173-9cf5-866b3188abf3",
"properties": {
"JobFailedMessage": "The streaming job failed.",
"JobId": "ff2593c9-cda2-498c-8e1f-6f13260ae32f",
"JobRunId": "d43b6623-3f9a-4e88-8c7d-21f8788e5cdd",
"JobRunCreatedDateTime": "2017-07-13 16:31:17Z",
"JobRunLastUpdateDateTime": "2017-07-13 16:35:46Z",
"JobRunStatus": "Failed",
"JobContentVersion": "1.10",
"JobFrameworkVersion": "1.2.61907.22"
},
"relatedEvents": []
}
Any ideas why job fails would be perfect. I am quite sure that I ma within limits of power bi app.
You need to authorize the Power BI subscription from within Stream Analytics.
Related
I was looking for an API which gives me the details of a Azure Datafactory Pipeline
including start time,end time, duration etc. This is required to generate a report without manually looking into the ADF monitoring window
You can get a list of pipeline runs using Pipeline Runs - Query By Factory and then for each result run [Pipeline Runs - Query By Factory]
POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/queryPipelineRuns?api-version=2018-06-01
Checkout the sample request here
You can pick the necessary fields from the response body.
{
"value": [
{
"runId": "2f7fdb90-5df1-4b8e-ac2f-064cfa58202b",
"pipelineName": "examplePipeline",
"parameters": {
"OutputBlobNameList": "[\"exampleoutput.csv\"]"
},
"invokedBy": {
"id": "80a01654a9d34ad18b3fcac5d5d76b67",
"name": "Manual"
},
"runStart": "2018-06-16T00:37:44.6257014Z",
"runEnd": "2018-06-16T00:38:12.7314495Z",
"durationInMs": 28105,
"status": "Succeeded",
"message": "",
"lastUpdated": "2018-06-16T00:38:12.7314495Z",
"annotations": [],
"runDimension": {
"JobId": "79c1cc52-265f-41a5-9553-be65e736fbd3"
}
},
{
"runId": "16ac5348-ff82-4f95-a80d-638c1d47b721",
"pipelineName": "examplePipeline",
"parameters": {
"OutputBlobNameList": "[\"exampleoutput.csv\"]"
},
"invokedBy": {
"id": "7c5fd7ef7e8a464b98b931cf15fcac66",
"name": "Manual"
},
"runStart": "2018-06-16T00:39:49.2745128Z",
"runEnd": null,
"durationInMs": null,
"status": "Cancelled",
"message": "",
"lastUpdated": "2018-06-16T00:39:51.216097Z",
"annotations": [],
"runDimension": {
"JobId": "84a3c493-0628-4b44-852f-ef5b3a11bdab"
}
}
]
}
And here is a simple WebActivity setup:
Further, you can find similar queries answered by me here and here
I have an Azure resource manager template that quite often, but now always, give me a strange error when I try to deploy it. My gut feeling is that this started a few days ago, but I’m not sure what the reason is. The deployments most of the times end with the error message:
"error": {
"code": "properties.search.query is not a valid log or metric query.An error occurred while sending the request.",
"message": {
"lang": "en-US",
"value": "properties.search.query is not a valid log or metric query.An error occurred while sending the request."
},
"innererror": {
"stacktrace": "System.ArgumentException: properties.search.query is not a valid log or metric query.An error occurred while sending the request.\r\n at Microsoft.AppInsights.Nexus.ResourceProvider.Validation.Ensure.IsTrue(Boolean value, String message)…
This doesn’t say me much :-( I have noticed that they deployment works fine if I remove this section from my template:
{
"name": "[concat('ExceptionOccured ', variables('insightsNames')[copyIndex()])]",
"apiVersion": "2018-04-16",
"type": "Microsoft.Insights/scheduledQueryRules",
"location": "East US",
"tags": {
"[concat('hidden-link:', resourceId('microsoft.insights/components', variables('insightsNames')[copyIndex()]))]": "Resource"
},
"properties": {
"description": "Some exception has been logged.",
"enabled": "true",
"source": {
"query": "exceptions",
"authorizedResources": [],
"dataSourceId": "[resourceId('microsoft.insights/components', variables('insightsNames')[copyIndex()])]",
"queryType": "ResultCount"
},
"schedule": {
"frequencyInMinutes": 60,
"timeWindowInMinutes": 60
},
"action": {
"severity": "1",
"status": "Active",
"aznsAction": {
"actionGroup": [
"[resourceId('microsoft.insights/actionGroups', variables('alertGroupName'))]"
]
},
"trigger": {
"thresholdOperator": "GreaterThan",
"threshold": 0,
"consecutiveBreach": 1
},
"odata.type": "Microsoft.WindowsAzure.Management.Monitoring.Alerts.Models.Microsoft.AppInsights.Nexus.DataContracts.Resources.ScheduledQueryRules.AlertingAction"
}
},
"dependsOn": [
"[resourceId('microsoft.insights/components', variables('insightsNames')[copyIndex()])]",
"[resourceId('microsoft.insights/actionGroups', variables('alertGroupName'))]"
],
"copy": {
"name": "insightscopy",
"count": "[length(variables('insightsNames'))]"
},
"condition": "[greater(length(parameters('alertEmail')), 0)]"
},
This is a loop to setup alerts for some application insights projects. Is there some error in this section?
I started getting this yesterday too. Same alerting structure, same error. I tried ~30mins later and it worked fine. Not what you wanted to hear, but I couldnt't find the reason why either.
I am deploying an new metric alert to Azure with an ARM template.
I am following the exact same way of Microsoft doc.
With the only change that I deploy just 1 metric to an Automation account and not to an storage account
Template file
"variables": {
"criterion1": "[array(parameters('criterion1'))]",
"criteria": "[concat(variables('criterion1'))]"
},
"resources": [
{
"name": "[parameters('alertName')]",
"type": "Microsoft.Insights/metricAlerts",
"location": "global",
"apiVersion": "2018-03-01",
"tags": {},
"properties": {
"description": "[parameters('alertDescription')]",
"severity": "[parameters('alertSeverity')]",
"enabled": "[parameters('isEnabled')]",
"scopes": [
"[parameters('resourceId')]"
],
"evaluationFrequency": "[parameters('evaluationFrequency')]",
"windowSize": "[parameters('windowSize')]",
"criteria": {
"odata.type": "Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria",
"allOf": "[variables('criteria')]"
},
"actions": [
{}
]
}
}
]
parameter file
"criterion1": {
"value": {
"name": "1st criterion",
"metricName": "TotalJob",
"dimensions": [
{
"name": "Status",
"operator": "Include",
"values": [
"Failed"
]
},
{
"name": "Status",
"operator": "Include",
"values": [
"Completed"
]
}
],
"operator": "GreaterThan",
"threshold": "5",
"timeAggregation": "Total"
}
}
But when i deploy this to Azure my Powershell command get stuck without giving any errors even with -DeploymentDebugLogLevel All parameter on it. In Azure portal I got the error "Internal server error" without any context. The json log gives me following logs:
{
"authorization": {
"action": "Microsoft.Insights/metricAlerts/write",
"scope": "/subscriptions/xxxxxx/resourcegroups/bilalachahbar/providers/Microsoft.Insights/metricAlerts/New Metric Alert"
},
"caller": "xxxx",
"channels": "Operation",
"claims": {
"aud": "https://management.azure.com/",
"iss": "https://sts.windows.net/17b5a1d-057c-4ac-a15a-08758f7a7064/",
"iat": "15596014",
"nbf": "15596014",
"exp": "15599914",
"aio": "42RgYDgypS7rfe/Of0l1R+q3TbCgA=",
"appid": "0e4a093a-c6fd-4fba-b4e5-f07ba479f203",
"appidacr": "1",
"http://schemas.microsoft.com/identity/claims/identityprovider": "https://sts.windows.net/17xxxxxc5-a15a-08758f7a7064/",
"http://schemas.microsoft.com/identity/claims/objectidentifier": "a3db39bf-8c65-4b84-b049-d7af99bfb3e",
"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier": "a3db39bf-8c65-4b84-b049-d7af99bfb3e",
"http://schemas.microsoft.com/identity/claims/tenantid": "1xxxxxx057c-4ac5-a15a-087f7a7064",
"uti": "SCkIk235EScz0Hst20AA",
"ver": "1.0"
},
"correlationId": "8013b5-9788-41ed-afcf-0dbd8276349c",
"description": "",
"eventDataId": "e39509-0837-4435-af7a-02ba1462055f",
"eventName": {
"value": "EndRequest",
"localizedValue": "End request"
},
"category": {
"value": "Administrative",
"localizedValue": "Administrative"
},
"eventTimestamp": "2018-12-27T14:11:48.1462445Z",
"id": "/subscriptions/xxxxx/resourcegroups/xxxxxx/providers/Microsoft.Insights/metricAlerts/New+Metric+Alert/events/e39509-0837-4435-af7a-02ba1462055f/ticks/815167081462445",
"level": "Error",
"operationId": "e390389-ecc1-4a2-8c2-d94ea635cb",
"operationName": {
"value": "Microsoft.Insights/metricAlerts/write",
"localizedValue": "Create or update metric alert"
},
"resourceGroupName": "xxxxx",
"resourceProviderName": {
"value": "Microsoft.Insights",
"localizedValue": "Microsoft Insights"
},
"resourceType": {
"value": "Microsoft.Insights/metricAlerts",
"localizedValue": "Microsoft.Insights/metricAlerts"
},
"resourceId": "/subscriptions/xxxxxx/resourcegroups/bilalachahbar/providers/Microsoft.Insights/metricAlerts/New Metric Alert",
"status": {
"value": "Failed",
"localizedValue": "Failed"
},
"subStatus": {
"value": "InternalServerError",
"localizedValue": "Internal Server Error (HTTP Status Code: 500)"
},
"submissionTimestamp": "2018-12-27T14:12:05.0719055Z",
"subscriptionId": "xxxxxx",
"properties": {
"statusCode": "InternalServerError",
"serviceRequestId": "8613b5-9788-41d-afcf-0dbd27639c",
"statusMessage": "{\"error\":{\"code\":\"InternalServerError\",\"message\":\"The server encountered an internal error, please retry. If the problem persists, contact support.\"}}"
},
"relatedEvents": []
}
An other stack overflow question got an sort of same question.
He got the problem when using an resource that is not supported anymore but I guess that is not the case with me because the official MS documentation is from september this year. I got the same issues when I use the exact same arm template that is provided in the documentation
I found my own error
Action groups are required when you want to deploy metric alerts.
As you can see in the documentation they provide an action ID, and I didn't. As I thought that it wasn't necessary it actually is.
I know this is obvious but unfortunately I did not saw this in the documentation or in the error. After some debugging and looking in the Resource Explorer I've noticed this.
SO future reader I hope this will solve your issue
One little feedback is that there is no depends on value ATM so I can not create an action group resource first in the same arm template
i'm having continual trouble keeping a triggered webjob up and running on Azure, after digging around in some logs i located one for the 'start' webjob it looks to me there is an authorisation on the service which is specifically scoped for the triggered webjob.
what i'm thinking is the authentication is expiring and not auto-renewing so the triggered webjob fails and moves to an aborted state.
we have already tried setting the configuration setting for the webjob to 12 months (in seconds) and we have applied a heartbeat, service still shuts down though.
log details below;
also does anybody know what e_exp is?
that roughly equates to 3 days in seconds
{
"authorization": {
"action": "Microsoft.Web/sites/TriggeredWebJobs/run/action",
"scope": "/subscriptions/some GUID/resourceGroups/.../providers/Microsoft.Web/sites/.../TriggeredWebJobs/Processing"
},
"caller": "some GUID",
"channels": "Operation",
"claims": {
"aud": "https://management.core.windows.net/",
"iss": "https://sts.windows.net/some GUID/",
"iat": "1537204596",
"nbf": "1537204596",
"exp": "1537208496",
"aio": "numbers and letters",
"appid": "some GUID",
"appidacr": "1",
"e_exp": "262800",
"http://schemas.microsoft.com/identity/claims/identityprovider": "https://sts.windows.net/some GUID/",
"http://schemas.microsoft.com/identity/claims/objectidentifier": "some GUID",
"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier": "some GUID",
"http://schemas.microsoft.com/identity/claims/tenantid": "some GUID",
"uti": "numbers and letters",
"ver": "1.0"
},
"correlationId": "some GUID",
"description": "",
"eventDataId": "some GUID",
"eventName": {
"value": "EndRequest",
"localizedValue": "End request"
},
"category": {
"value": "Administrative",
"localizedValue": "Administrative"
},
"eventTimestamp": "2018-09-17T17:21:50.2300614Z",
"id": "/subscriptions/some GUID/resourceGroups/qcp-ghs-prd/providers/Microsoft.Web/sites/.../TriggeredWebJobs/Processing/events/some GUID/ticks/...",
"level": "Informational",
"operationId": "some GUID",
"operationName": {
"value": "Microsoft.Web/sites/TriggeredWebJobs/run/action",
"localizedValue": "Run Web Apps Triggered WebJobs"
},
"resourceGroupName": "...",
"resourceProviderName": {
"value": "Microsoft.Web",
"localizedValue": "Azure Web Sites"
},
"resourceType": {
"value": "Microsoft.Web/sites/TriggeredWebJobs",
"localizedValue": "Microsoft.Web/sites/TriggeredWebJobs"
},
"resourceId": "/subscriptions/some GUID/resourceGroups/.../providers/Microsoft.Web/sites/qcp-ghs-prd-ase/TriggeredWebJobs/Processing",
"status": {
"value": "Succeeded",
"localizedValue": "Succeeded"
},
"subStatus": {
"value": "OK",
"localizedValue": "OK (HTTP Status Code: 200)"
},
"submissionTimestamp": "2018-09-17T17:22:18.6120141Z",
"subscriptionId": "some GUID",
"properties": {
"statusCode": "OK",
"serviceRequestId": "some GUID"
},
"relatedEvents": []
}
I am trying to integrate azure metrics with my dashboard application(web app) for live monitoring using api request.I tried with alert rules but no response coming.
this sample request i made:
url:https://management.azure.com/subscriptions/XXXXXX-XXXX-XXXX-XXXX-XXXXX/resourceGroups/pwcdfp/providers/microsoft.insights/alertrules/test?api-version=2016-03-01
payload:
{
"id": "/subscriptions/b3728f4e-cba5-44e2-bf02-b03e7fd1d94e/resourceGroups/pwcdfp/providers/microsoft.insights/alertrules/test",
"name": "test",
"type": "Microsoft.Insights/alertRules",
"location": "southeastasia",
"tags": {
"$type": "Microsoft.WindowsAzure.Management.Common.Storage.CasePreservedDictionary, Microsoft.WindowsAzure.Management.Common.Storage",
"hidden-link:/subscriptions/b3728f4e-cba5-44e2-bf02-b03e7fd1d94e/resourceGroups/pwcdfp/providers/Microsoft.ClassicCompute/virtualMachines/pwcdfp": "Resource"
},
"properties": {
"name": "test",
"description": "test",
"isEnabled": true,
"condition": {
"$type": "Microsoft.WindowsAzure.Management.Monitoring.Alerts.Models.ThresholdRuleCondition, Microsoft.WindowsAzure.Management.Mon.Client",
"odata.type": "Microsoft.Azure.Management.Insights.Models.ThresholdRuleCondition",
"dataSource": {
"$type": "Microsoft.WindowsAzure.Management.Monitoring.Alerts.Models.RuleMetricDataSource, Microsoft.WindowsAzure.Management.Mon.Client",
"odata.type": "Microsoft.Azure.Management.Insights.Models.RuleMetricDataSource",
"resourceUri": "/subscriptions/b3728f4e-cba5-44e2-bf02-b03e7fd1d94e/resourceGroups/pwcdfp/providers/Microsoft.ClassicCompute/virtualMachines/pwcdfp",
"metricName": "Disk Read Bytes/sec"
},
"threshold": 1,
"windowSize": "PT5M",
"timeAggregation": "Average"
},
"lastUpdatedTime": "2016-09-19T05:34:38.6351894Z",
"provisioningState": "Succeeded",
"actions": []
}
}
content type: application/json
Authorization token:
SharedAccessSignature uid=57d95d630ac0d2007b030003&ex=2016-09-17T09:10:00.0000000Z&sn=F3s+t2ufw76Bmh9SuoYYnF4zek0v/2jz7NXbLYssk1vwnMhcAeNxLwiA1Sg2Q5aEiQw/yPPYrmVNJbaiJOC61g==
The question is a little unclear, since you first ask how to get Azure monitoring data, but then describe an alert, which is an action taken based off of monitoring data. If you're attempting to regularly get metric data from a Classic Compute VM, an alert is not the right way to do that. An alert only fires when a specific threshold is met. Instead, you can use the .NET SDK to get that data.
Not all resources provide metrics using this SDK today. For a Classic Compute VM in particular, you will not receive any metrics unless you have turned on Azure Diagnostics. You can do this by navigating to the blade in the Azure portal for the VM, going to "Diagnostics" and checking "Basic metrics."