I want to delete the traces data from my application insights table. Can one tell me how do i do that.
I'm following the below article from microsoft,
https://learn.microsoft.com/en-us/rest/api/application-insights/components/purge
I'm trying to pass the below Request body,
{
"table": "traces",
"filters": [
{
"column": "timestamp",
"operator": ">",
"value": "2017-09-01T00:00:00"
}
]
}
I'm getting Response Code: 202, but i'm still seeing the data in the analytics, where i query traces.
Please let me know, how i can delete the data with traces on my application insights resource.
As per this SO issue:
Yes, it is possible to purge Application Insights data, but it may take a while (e.g. 2-3 days) for the operation to complete.
So please wait:)
For traces delete operation you would need to use the separate table name as "Traces" in the body of purge operation. then you can check the status using get purge status REST API operation. Refer to know more - https://sanganakauthority.blogspot.com/2019/06/how-to-delete-or-purge-azure.html
Related
I need to retrieve azure data factory pipelines execution logs. I've tried with Web Acticity by using the following request:
GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}?api-version=2018-06-01
Unfortunatelly I have the following error:
Invoking Web Activity failed with HttpStatusCode - 'NotFound', message - 'The requested resource does not exist on the server. Please verify the request server and retry'
Should I set some additionall configuration to be able to retrieve these logs?
You can get a list of pipeline runs using Pipeline Runs - Query By Factory
Next, you can Get a pipeline run by its run ID.
GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}?api-version=2018-06-01
Here is a sample URL:
https://management.azure.com/subscriptions/b83c1ed3-XXXX-XXX-XXXXX-2n83a074t23f/resourceGroups/resource-grp/providers/Microsoft.DataFactory/factories/ktestadf/pipelineruns/0bdaba11-47b7-4885-9796-5801b4bb856a?api-version=2018-06-01
If you are constructing URL dynamically, using Pipeline RunID system variable, you can using string interpolation method. Notice #{pipeline().RunId} in place of {runId}.
https://management.azure.com/subscriptions/b83c1ed3-XXXX-XXXX-XXXXX-2n83a074t23f/resourceGroups/resource-grp/providers/Microsoft.DataFactory/factories/ktestadf/pipelineruns/#{pipeline().RunId}?api-version=2018-06-01
Note: You would have to Trigger run and not debug, since this will create pipeline. Make sure you have published all before trigering
run, debug can take the changes but pipeline run needs the changes be
published.
And here is a simple WebActivity setup:
Input
{
"url": " https://management.azure.com/subscriptions/b83c1td3-XXXX-XXXX-XXXXX-2b83a074c13f/resourceGroups/myrg/providers/Microsoft.DataFactory/factories/ktestadf/pipelineruns/0bdaba11-47b7-4885-9796-5801b4bb856a?api-version=2018-06-01 ",
"method": "GET",
"headers": {
"Content-Type": "application/json"
},
"authentication": {
"type": "MSI",
"resource": " https://management.azure.com/ "
}
}
You can get a pipeline run ID from here manually to test.
I did repro before posting this and the only reason this error pops is cause any value you provided is wrong or does not already exist (in case of pipeline run id)
We have an Azure function v3 running Node, consumption plan, with an input trigger connected to a cosmos database. The function.json looks like this:
{
"disabled": false,
"bindings": [
{
"type": "cosmosDBTrigger",
"name": "productDocuments",
"collectionName": "products",
"direction": "in",
"connectionStringSetting": "DB_CONNECTION_STRING",
"databaseName": "product-management",
"createLeaseCollectionIfNotExists": true,
"maxItemsPerInvocation": 1
},
{
"name": "productDocument",
"type": "cosmosDB",
"databaseName": "product-management",
"collectionName": "products",
"createIfNotExists": true,
"connectionStringSetting": "_DB_CONNECTION_STRING",
"direction": "out"
}
],
"scriptFile": "dist/nameOfFunction.js"
}
But this trigger is working really, really slow and unreliable. If we add an item to the DB it sometimes triggers straight away, sometimes it seems to take hours and sometimes not at all. I am manually monitoring the cosmos db so I can see that items are added.
I am looking at this page, and most of the time nothing happens. I don't know how else to debug this
Should it really take hours for an invocation to show up here? Or is it the trigger that's unreliable?
General guidance is in this doc: https://learn.microsoft.com/azure/cosmos-db/troubleshoot-changefeed-functions#my-changes-take-too-long-to-be-received
What happens on Consumption Plan is that, after a period of inactivity, instances are deprovisioned. When a new instance is provisioned, it hits a cold start.
The key part here is that, when your instances are deprovisioned, they are not checking the Change Feed for events, so how does Functions know when to "wake them up"?
There is a periodic check done by an external component that checks to see if there are new changes, if there are new, then it would provision new instances of your Function to start consuming them.
This external component in your case, could be having an issue or delays in this checks.
If you have no Function logs for an hour even though you are making changes to the monitored collection, I would try to contact Azure Support to understand why is your Function not "waking up".
One of the known issues I've heard about was related to where the Cosmos DB Connection Strings were stored. Apparently this component at some point (maybe it's already fixed) had a problem where it could not access the Connection String if it was saved in "Connection Strings" section of the Function configuration, but was looking for it only on the "App Settings". In this cases, it could not wake up the Function and the Function only woke up if someone opened it on the Azure Portal. My recommendation would be to check where are you storing your connection string and see if you can move it to "App Settings" and see how it behaves.
Our problem with this was that we had two separate functions that both had a CosmosDBTrigger on the same collectionm but used the same lease, and apparently you can't do that. So it was solved by setting two separate leases (we used the leaseCollectionPrefix in the function.json.)
I have written a blob-triggered function that uploads data on a CosmosDB database using the Gremlin API, using Azure Functions version 2.0. Whenever the function is triggered, it is going to read the blob, extract relevant information, and then queries the database to upload the data on it.
However, when all files are uploaded on the blob storage at the same time, the Function is going to process all files at the same time, which results in too many requests for the database to handle. To avoid this, I ensured that the Azure Function would only process one file at a time, by setting the batchSize to 1 in the host.json file :
{
"extensions": {
"queues": {
"batchSize": 1,
"maxDequeueCount": 1,
"newBatchThreshold": 0
}
},
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"version": "2.0"
}
This worked perfectly fine for 20 files at a time.
Now, we are trying to process 300 files at a time, and this feature doesn't seem to work anymore, the Function processes all the files at the same time again, which results in the database not being able to handle all the requests.
What am I missing here ? Is there some scaling issue I'm not aware of ?
From here:
If you want to avoid parallel execution for messages received on one queue, you can set batchSize to 1. However, this setting eliminates concurrency as long as your function app runs only on a single virtual machine (VM). If the function app scales out to multiple VMs, each VM could run one instance of each queue-triggered function.
You need to combine this with the app setting WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT when you run in Consumption plan.
Or, according to the docs, the better way would be through the Function property functionAppScaleLimit: https://learn.microsoft.com/en-us/azure/azure-functions/event-driven-scaling#limit-scale-out
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT would work of course.
You can also scale to multiple Function App instances within one Host then you can have less hosts and more FUNCTIONS_WORKER_PROCESS_COUNT per host. Cost implications would depend on your plan.
Note that all workers within a Host would share resources, so this is recommended for more IO bound workload.
I am updating a QuickSight data source in my aws account.
aws quicksight update-data-source --cli-input-json file://update-stag-data-source-request.json --output json
And I get the following response:
{
"Status": 202,
"Arn": "arn:aws:quicksight:eu-west-1:<my-aws-account-nr>:datasource/099676d0-99e3-44d7-b581-d6e532e72961",
"DataSourceId": "099676d0-99e3-44d7-b581-d6e532e72961",
"UpdateStatus": "UPDATE_IN_PROGRESS",
"RequestId": "1d304a80-e507-46c3-acb3-237a58237e77"
}
So currently the status of this request is "UPDATE_IN_PROGRESS", but how do I track the status afterwards?
I need to do it, because it seems that the update fails eventually, for reasons unknown. I know that, because I still see the old setup of the data source several minutes later. I believe, if I knew the eventual request status it would help me to debug the issue.
Check the command describe-data-source, it will return the DataSource.Status and, in case of any failure, you can check inside DataSource.ErrorInfo.Message.
Getting the following response even when I make one request (concurrency set to 200) to a web service.
{ status: 503, headers: '{"content-length":"174","content-type":"application/json; charset=utf-8","etag":"\"8ce068bf420a485c8096065ea3e4f436\"","server":"Microsoft-HTTPAPI/2.0","x-ms-request-id":"d5c56cdd-644f-48ba-ba2b-6eb444975e4c","date":"Mon, 15 Feb 2016 04:54:01 GMT","connection":"close"}', body: '{"error":{"code":"ServiceUnavailable","message":"Service is temporarily unavailable.","details":[{"code":"NoMoreResources","message":"No resources available for request."}]}}' }
The request-response web service is a recommender retraining web service with the training set containing close to 200k records. The training set is already present in my ML studio dataset, only 10-15 extra records are passed in the request. The same experiment was working flawlessly till 13th Feb 2016. I have already tried increasing the concurrency but still the same issue. I even reduced the size of the training set to 20 records, still didn't work.
I have two web service both doing something similar and both aren't working since 13th Feb 2016.
Finally, I created a really small experiment ( skill.csv --> split row ---> web output ) which doesn't take any input. It just has to return some part of the dataset. Did not work, response code 503.
The logs I got are as follows
{
"version": "2014-10-01",
"diagnostics": [{
.....
{
"type": "GetResourceEndEvent",
"timestamp": 13.1362,
"resourceId": "5e2d653c2b214e4dad2927210af4a436.865467b9e7c5410e9ebe829abd0050cd.v1-default-111",
"status": "Failure",
"error": "The Uri for the target storage location is not specified. Please consider changing the request's location mode."
},
{
"type": "InitializationSummary",
"time": "2016-02-15T04:46:18.3651714Z",
"status": "Failure",
"error": "The Uri for the target storage location is not specified. Please consider changing the request's location mode."
}
]
}
What am I missing? Or am I doing it completely wrong?
Thank you in advance.
PS: Data is stored in mongoDB and then imported as CSV
This was an Azure problem. I quote the Microsoft guy,
We believe we have isolated the issue impacting tour service and we are currently working on a fix. We will be able to deploy this in the next couple of days. The problem is impacting only the ASIA AzureML region at this time, so if this is an option for you, might I suggest using a workspace in either the US or EU region until the fix gets rolled out here.
To view the complete discussion, click here