How to Purges data in an Log Analytics workspace with "between" operator by API? - azure

I want to purges data for my custom log with TimeGenerated range.
Sample Request
POST https://management.azure.com/subscriptions/00000000-0000-0000-0000-00000000000/resourceGroups/OIAutoRest5123/providers/Microsoft.OperationalInsights/workspaces/aztest5048/purge?api-version=2020-08-01
Request Body
{
"table": "MY_CUSTOM_TABLE",
"filters": [
{
"column": "TimeGenerated",
"operator": "between",
"value": ["2022-01-02T04:00:00","2022-01-07T03:59:59"]
}
]
}
Response Code: 400
Response Body:
{
"error": {
"message": "The request had some invalid properties",
"code": "BadArgumentError",
"correlationId": "00000000-0000-0000-0000-00000000000",
"innererror": {
"code": "QueryValidationError",
"message": "The user with object Id '00000000-0000-0000-0000-00000000000' does not have the role 'Data Purger' required to perform purge operation on this resource"
}
}
}

"message": "The user with object Id
'00000000-0000-0000-0000-00000000000' does not have the role 'Data
Purger' required to perform purge operation on this resource"
Looking at the Error Message that you have shared above, it is moreover related to access issue. You need to have DataPurger RBAC role on that resource inorder to perform data purge over a particular table in log analytic workspace.
Here is the reference documentation for more information about DataPurger RBAC role.
We have tested this in our local environment , it is working fine.
In our local environment , we have tried to purge the heartbeat table using the above shared Purge REST API of Azure Log Analytics workspace.
if you are using the between operator in the body of your restAPI , the value should be array of items.
Here is the sample request & body of the Rest API :
https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/purge?api-version=2020-08-01
Body:
{
"table": "Heartbeat",
"filters": [
{
"column": "TimeGenerated",
"operator": "between",
"value": ["2022-01-18T05:40:00","2022-01-18T05:50:59"]
}
]
}
Here is the sample Output for reference:

Related

Azure assign group and device to script using Graph API

I am trying to assign groups and devices to a PowerShell script in Microsoft Intune using the following POST request:
URL: https://graph.microsoft.com/beta/deviceManagement/deviceManagementScripts/%7Bid%7D/Assign
Body:
{
"deviceManagementScriptGroupAssignments": [
{
"#odata.type": "#microsoft.graph.deviceManagementScriptGroupAssignment",
"id": "{ScriptId:GroupId}",
"targetGroupId": "{GroupId}"
}
],
"deviceManagementScriptAssignments": [
{
"#odata.type": "#microsoft.graph.deviceManagementScriptAssignment",
"id": "{ScriptID:DeviceID}",
"target": {
"#odata.type": "#microsoft.graph.allDevicesAssignmentTarget",
"deviceAndAppManagementAssignmentFilterId": null,
"deviceAndAppManagementAssignmentFilterType": "none"
}
}
]
}
However, I am getting a 400 error with the following message:
"{ "_version": 3, "Message": "An error has occurred - Operation ID (for customer support): 00000000-0000-0000-0000-000000000000 - Activity ID: 615c3437-bd99-2193-9d4b-d9a61deb0340 - Url: https://fef.amsua0102.manage.microsoft.com/DeviceFE/StatelessDeviceFEService/deviceManagement/deviceManagementScripts('{id}')/microsoft.management.services.api.assign?api-version=5022-11-08", "CustomApiErrorPhrase": "", "RetryAfter": null, "ErrorSourceService": "", "HttpHeaders": "{}"}"
I base on this document of Microsoft https://learn.microsoft.com/en-us/graph/api/intune-shared-devicemanagementscript-assign?view=graph-rest-beta
I tried to reproduce the same in my environment.
Received the same error:
{
"error": {
"code": "UnknownError",
"message": "{\"ErrorCode\":\"Forbidden\",\"Message\":\"{\\ \\\"_version\\\": 3,\\ \\\"Message\\\": \\\"An error has occurred - Operation ID (for customer support): xxxx –
}
When I followed the url in the error and queried, I got that the api version is not supported.
Make sure the authorization header is given with bearer token which has proper permissions to work with intune and device management
Omit the parameter #odata.type in the post request body.
{
"#odata.type": "#microsoft.graph.deviceManagementScript",
…
"runAs32Bit": true
}
Make sure you have DeviceManagementConfiguration.ReadWrite.All Application and delegated permissions.
Note that Intune requires an active Intune license for the tenant.

How to purge records from Application Insights bucket?

I used to following instructions from https://learn.microsoft.com/en-us/rest/api/application-insights/components/purge, but now it produces the following error:
Purge operation on a workspace-based Application Insights resource is not supported. Instead, please run purge operation directly on the workspace, scoped to specific resource id. See more at https://aka.ms/purgeai"
OK, so https://aka.ms/purgeai are the new instructions, but it does not explain how to "scope the operation to the specific resource id", because the records are still in an AI bucket, not the workspace.
So, how do I purge from AI in the new world?
EDIT 1
I should have been more clear. I do not want to purge all the records. For example, consider the following purge request body, which was enough in the past:
{
"table": "customEvents",
"filters": [
{
"column": "customMeasurements",
"key": "seq",
"operator": "==",
"value": "''"
}
]
}
It could be used to purge all the records with customMeasurements.seq being empty. Now how do I compose it with the _ResourceId field?
When giving the following request body:
{
"table": "customEvents",
"filters": [
{
"column": "customMeasurements",
"key": "seq",
"operator": "==",
"value": "''"
},
{
"column": "_ResourceId",
"operator": "==",
"value": "/subscriptions/.../resourcegroups/.../providers/microsoft.insights/components/..."
}
]
}
The request fails with:
{
"error": {
"message": "The request had some invalid properties",
"code": "BadArgumentError",
"correlationId": "...",
"innererror": {
"code": "QueryValidationError",
"message": "'customEvents' is not a valid table"
}
}
}
EDIT 2
There is still a problem. Indeed, consider the following query:
AppEvents
| where Measurements.seq == '' and _ResourceId == '/subscriptions/d...8/resourcegroups/app508-re-mark-test/providers/microsoft.insights/components/app508-re-mark-test-ai'
| summarize count()
This query works and return some count.
Now I want to use the same condition to purge records in the same workspace:
{
"table": "AppEvents",
"filters": [
{
"column": "Measurements",
"key": "seq",
"operator": "==",
"value": "''"
},
{
"column": "_ResourceId",
"operator": "==",
"value": "/subscriptions/d...8/resourcegroups/app508-re-mark-test/providers/microsoft.insights/components/app508-re-mark-test-ai"
}
]
}
But I get back the following error:
{
"error": {
"message": "The request had some invalid properties",
"code": "BadArgumentError",
"correlationId": "c08bb441-a316-4989-a527-202a101e515d",
"innererror": {
"code": "QueryValidationError",
"message": "Unsupported column '_ResourceId'"
}
}
}
Note, that omitting the _ResourceId works, but I do want to be able to use it.
What am I missing?
Thank you for your question, and feedback.
What you will need to do, is to filter by the specific resource id, in addition to any other required filter.
Ex:
`{
"column": "_ResourceId",
"operator": "==",
"value": “/subscriptions/12341234-1234-1234-1234-123412341234/resourceGroups/SomeResourceGroup/providers/microsoft.insights/components/AppInsightResource”
}`
Please note,
The table names have changed between the Classic Applicaiton insights, and the workspace-based Application insights. For example, in your case the table name should be AppEvents instead of customEvents.
You can review this link for other table names changes.
We will also modify the articles to reflect this information.

Azure Graph API - ClaimsMappingPolicy with ClaimsTransformation

Im trying to automate the configuration of an enterpise application via the Azure Graph API.
Specifically, its the Azure Palo Alto Admin UI - https://learn.microsoft.com/en-us/azure/active-directory/saas-apps/paloaltoadmin-tutorial#configure-azure-ad-sso
Ive managed to get this working via the frontend, but im having trouble configuring the custom claims via the Graph Api.
For now, i just want to use a string claim in the custom claim as the customadmin value with a hardcoded value for the admin role
When creating via the portal, you can easily enter a string value as the source type of the claim.
However, via the Graph API the source type must be user, resource, audience, company or transformation.
https://learn.microsoft.com/en-us/azure/active-directory/develop/active-directory-claims-mapping#claim-schema-entry-elements
It seems that you can create a string type of transformation and then link the transformation into the main ClaimsSchema.
There is a similar example documented here https://learn.microsoft.com/en-us/graph/api/resources/claimsmappingpolicy?view=graph-rest-1.0#example-definition-that-uses-a-claims-transformation
But I cannot get the example to work. Even with a bit of massaging, the example fails. This is what ive been trying:
cat <<- EOF > claims.json
{
"definition": [
"{\"ClaimsMappingPolicy\":{
\"Version\":1,
\"IncludeBasicClaimSet\":\"true\",
\"ClaimsSchema\":[
{\"Source\":\"user\",\"ID\":\"extensionattribute1\"},{\"Source\":\"transformation\",\"ID\":\"DataJoin\",\"TransformationId\":\"JoinTheData\",\"JwtClaimType\":\"JoinedData\"}
],
\"ClaimsTransformation\":[
{\"ID\":\"JoinTheData\",\"TransformationMethod\":\"Join\",\"InputClaims\":[{\"ClaimTypeReferenceId\":\"extensionattribute1\",\"TransformationClaimType\":\"string1\"}], \"InputParameters\": [{\"ID\":\"string2\",\"Value\":\"sandbox\"},{\"ID\":\"separator\",\"Value\":\".\"}],\"OutputClaims\":[{\"ClaimTypeReferenceId\":\"DataJoin\",\"TransformationClaimType\":\"outputClaim\"}]}
]
}}"
],
"displayName": "Azure Reference Claim",
"isOrganizationDefault": false
}
EOF
az rest --method post --headers Content-type="application/json" --url "https://graph.microsoft.com/v1.0/policies/claimsMappingPolicies" --body #claims.json
Ive tried both the v1.0 and beta APIs but they both have the same behaviour
Which returns with the following error:
Bad Request({
"error": {
"code": "Request_BadRequest",
"message": "Property has an invalid value.",
"innerError": {
"date": "2020-09-01T13:03:10",
"request-id": "bc7cf58e-fe6d-47d1-b1e5-cae43326864f"
}
}
})
I was able to get the rest of the Palo Alto claim working (excluding the custom string) with the following:
{
"definition": [
"{\"ClaimsMappingPolicy\":{
\"Version\":1,
\"IncludeBasicClaimSet\":\"true\",
\"ClaimsSchema\": [{\"Source\":\"user\",\"ID\":\"userprincipalname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier\"},{\"Source\":\"user\",\"ID\":\"givenname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname\"},{\"Source\":\"user\",\"ID\":\"displayname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name\"},{\"Source\":\"user\",\"ID\":\"surname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname\"},{\"Source\":\"user\",\"ID\":\"userprincipalname\",\"SamlClaimType\":\"username\"}]
}}"
],
"displayName": "Palo Alto Claims Policy",
"isOrganizationDefault": false
}
And i was able to create a CustomString transformation which isnt linked to anything with the following:
{
"definition": [
"{\"ClaimsMappingPolicy\":{
\"Version\":1,
\"IncludeBasicClaimSet\":\"true\",
\"ClaimsTransformation\":[{\"ID\":\"CreateTermsOfService\",\"TransformationMethod\":\"CreateStringClaim\",\"InputParameters\": [{\"ID\":\"value\",\"DataType\":\"string\", \"Value\":\"sandbox\"}],\"OutputClaims\":[{\"ClaimTypeReferenceId\":\"TOS\",\"TransformationClaimType\":\"createdClaim\"}]}]
}}",
],
"displayName": "sdfa",
"isOrganizationDefault": false
}
However, when i try them together in the format of the example I get an error.
{
"definition": [
"{\"ClaimsMappingPolicy\":{
\"Version\":1,
\"IncludeBasicClaimSet\":\"true\",
\"ClaimsSchema\": [
{\"Source\":\"user\",\"ID\":\"userprincipalname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier\"},{\"Source\":\"user\",\"ID\":\"givenname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname\"},{\"Source\":\"user\",\"ID\":\"displayname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name\"},{\"Source\":\"user\",\"ID\":\"surname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname\"},{\"Source\":\"user\",\"ID\":\"userprincipalname\",\"SamlClaimType\":\"username\"},{\"Source\":\"transformation\",\"TransformationID\":\"xxxxxxxxx\",\"ID\":\"DataJoin\",\"SamlClaimType\":\"test\"}
],
\"ClaimsTransformation\":[
{\"ID\":\"xxxxxxxxx\",\"TransformationMethod\":\"CreateStringClaim\",\"InputParameters\": [{\"ID\":\"value\",\"DataType\":\"string\", \"Value\":\"sandbox\"}],\"OutputClaims\":[{\"ClaimTypeReferenceId\":\"DataJoin\",\"TransformationClaimType\":\"createdClaim\"}]}
]
}}"
],
"displayName": "Palo Alto Claims Policy",
"isOrganizationDefault": false
}
Which returns the same unhelpful error:
Bad Request({
"error": {
"code": "Request_BadRequest",
"message": "Property has an invalid value.",
"innerError": {
"date": "2020-09-01T13:03:10",
"request-id": "bc7cf58e-fe6d-47d1-b1e5-cae43326864f"
}
}
})
Any ideas what i am doing wrong? Im trying to base off of the example, which i cant get working.
I do not want to use powershell, i want to be able to automate via my desktop terminal.
I imagine i can avoid this situation and get the PA to integrate with AAD without a hardcoded value, but i feel that i should be able to get this working this way.
The mandatory encoding of the ClaimMappingPolicy object makes it quite fiddely to develop, so its possible there is a problem there somewhere.
Ive also tried just creating the ClaimsSchema without the ClaimsTransformation and then running a PATCH to amend the object with the transformed object, but it just overwrites the whole ClaimsMappingPolicy object rather than adding just the extra field.
When I remove the transformation source from the ClaimsSchema the request succeeds.
cat <<- EOF > claims.json
{
"definition": [
"{\"ClaimsMappingPolicy\":{
\"Version\":1,
\"IncludeBasicClaimSet\":\"true\",
\"ClaimsSchema\": [
{\"Source\":\"user\",\"ID\":\"userprincipalname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier\"},{\"Source\":\"user\",\"ID\":\"givenname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname\"},{\"Source\":\"user\",\"ID\":\"displayname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name\"},{\"Source\":\"user\",\"ID\":\"surname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname\"},{\"Source\":\"user\",\"ID\":\"userprincipalname\",\"SamlClaimType\":\"username\"}
],
\"ClaimsTransformation\":[
{\"ID\":\"xxxxxxxxx\",\"TransformationMethod\":\"CreateStringClaim\",\"InputParameters\": [{\"ID\":\"value\",\"DataType\":\"string\", \"Value\":\"sandbox\"}],\"OutputClaims\":[{\"ClaimTypeReferenceId\":\"DataJoin\",\"TransformationClaimType\":\"createdClaim\"}]}
]
}}"
],
"displayName": "Palo Alto Claims Policy",
"isOrganizationDefault": false
}
EOF
But there isnt an association between the ClaimsSchema and the ClaimsTransformation. This hints at a problem with the ClaimsSchema object
{\"Source\":\"transformation\",\"TransformationID\":\"xxxxxxxxx\",\"ID\":\"DataJoin\",\"SamlClaimType\":\"test\"}
But this looks suitable when looking at the documentation and the (possibly broken) reference example.
Providing information in answer as its too long to comment it.Please try this below query in Graph explorer
Post https://graph.microsoft.com/beta/policies/claimsMappingPolicies
{"definition":["{\"ClaimsMappingPolicy\":{\"Version\":1,\"IncludeBasicClaimSet\":\"true\", \"ClaimsSchema\":[{\"Source\":\"user\",\"ID\":\"extensionattribute1\"},{\"Source\":\"transformation\",\"ID\":\"DataJoin\",\"TransformationId\":\"JoinTheData\",\"JwtClaimType\":\"JoinedData\"}],\"ClaimsTransformations\":[{\"ID\":\"JoinTheData\",\"TransformationMethod\":\"Join\",\"InputClaims\":[{\"ClaimTypeReferenceId\":\"extensionattribute1\",\"TransformationClaimType\":\"string1\"}], \"InputParameters\": [{\"ID\":\"string2\",\"Value\":\"sandbox\"},{\"ID\":\"separator\",\"Value\":\".\"}],\"OutputClaims\":[{\"ClaimTypeReferenceId\":\"DataJoin\",\"TransformationClaimType\":\"outputClaim\"}]}]}}"],"displayName":"TestclaimsPolicy","isOrganizationDefault":false}
Post https://graph.microsoft.com/beta/policies/claimsMappingPolicies
{"definition":["{\"ClaimsMappingPolicy\":{\"Version\":1,\"IncludeBasicClaimSet\":\"true\",\"ClaimsSchema\": [{\"Source\":\"user\",\"ID\":\"userprincipalname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier\"},{\"Source\":\"user\",\"ID\":\"givenname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname\"},{\"Source\":\"user\",\"ID\":\"displayname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name\"},{\"Source\":\"user\",\"ID\":\"surname\",\"SamlClaimType\":\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname\"},{\"Source\":\"user\",\"ID\":\"userprincipalname\",\"SamlClaimType\":\"username\"}],\"ClaimsTransformation\":[{\"ID\":\"CreateTermsOfService\",\"TransformationMethod\":\"CreateStringClaim\",\"InputParameters\": [{\"ID\":\"value\",\"DataType\":\"string\", \"Value\":\"sandbox\"}],\"OutputClaims\":[{\"ClaimTypeReferenceId\":\"TOS\",\"TransformationClaimType\":\"createdClaim\"}]}]}}"],"displayName":"Test1234","isOrganizationDefault":false}
for more information on CreateTermsOfService please refer to this document

Adding User to Group using Graph API B2C

I am using Azure AD B2C to create users.
After a user is created I would like to add them to a Group
Using Postman I have made the following request (with access token in the header). Note: I have managed to successfully create a group using the Api.
POST
https://graph.windows.net/{azure-tenant}/groups/{group-objectId/members/$ref?api-version=1.6
With Body:
{
"#odata.id": "https://graph.windows.net/{azure-tenant}/directoryObjects/{user-objectId}"
}
Which is what the documentation specifies. No matter the body that is sent I get the following error
{
"odata.error": {
"code": "Request_BadRequest",
"message": {
"lang": "en",
"value": "The request URI is not valid. Since the segment 'members' refers to a collection, this must be the last segment in the request URI. All intermediate segments must refer to a single resource."
},
"requestId": "48cf65f3-20ba-411e-8121-f7ea54252f3a",
"date": "2019-05-27T06:09:25"
}
}
I tried removing the /$ref
POST
https://graph.windows.net/{azure-tenant}/groups/{group-objectId/members?api-version=1.6
Body:
{
"#odata.id": "https://graph.windows.net/{azure-tenant}/directoryObjects/{user-objectId}"
}
as well as
{
"#odata.id": "https://graph.windows.net/{azure-tenant}/users/{user-objectId}"
}
Now the error that gets returned now is:
{
"odata.error": {
"code": "Request_BadRequest",
"message": {
"lang": "en",
"value": "Unsupported resource type 'DirectoryObject' for operation 'Create'."
},
"requestId": "2c36cc6d-383c-44f8-8609-2ac3e3efc862",
"date": "2019-05-27T06:15:26"
}
}
The documentation that I have been using https://learn.microsoft.com/en-us/graph/api/group-post-members?view=graph-rest-1.0&tabs=javascript
You are using graph.windows.net endpoint, so you should refer to azure ad graph api.
POST https://graph.windows.net/myorganization/groups/b4bda672-1fba-4711-8fb1-5383c40b2c14/$links/members?api-version=1.6
{
"url": "https://graph.windows.net/myorganization/directoryObjects/3eb6055a-baeb-44d4-a1ea-2fee86d8891b"
}
The document you referred to is microsoft graph api. The endpoint should be https://graph.microsoft.com.
You can use Microsoft Graph explorer to call these apis.

Azure automation job OData $filter

Background
I have an application which get list of runbooks jobs using REST API
I would like to apply $filter on properties parameters(see JSON in the end). Parameters is of type IDictionary<string,string>
Issue
$filter works fine for most of the properties but fails for IDictionary<string,string>
This is what I am trying https://management.azure.com/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Automation/automationAccounts/XXX/jobs?$filter=properties/parameters/any(keyValue: keyValue/owner eq 'Adam#test.com')&api-version=2015-10-31
Request fails with error {
"code": "BadRequest",
"message": "Could not find a property named 'owner' on type 'System.Collections.Generic.KeyValuePair_2OfString_String'."
}
Question
Is it possible to filter what I am trying?
If yes, then what I am doing wrong?
JSON response on which I want to apply filter
"value": [
{
"id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Automation/automationAccounts/XXX/jobs/XXX",
"properties": {
"jobId": "XXX",
"runbook": {
"name": "HelloWorldRunbook"
},
"schedule": null,
"provisioningState": "Succeeded",
"status": "Completed",
"creationTime": "2018-06-17T05:44:12.197+00:00",
"startTime": "2018-06-17T05:44:21.227+00:00",
"lastModifiedTime": "2018-06-17T05:44:43.43+00:00",
"endTime": "2018-06-17T05:44:43.43+00:00",
"jobScheduleId": "7fc134ac-d8bd-464e-b041-6e6b50f83f0c",
"runOn": null,
"parameters": {
"Owner": "Adam#test.com",
"mailBox": "test_mailbox#test.com"
}
}
............removed for brevity
When I try to run List Job By Automation Account Rest API, there is no properties parameters returned in the response body.
If yes, then what I am doing wrong?
...
"parameters": {
"Owner": "Adam#test.com",
"mailBox": "test_mailbox#test.com"
}
...
But according to you mentioned responsed json format, the parameters properties is an object not a dictionary, so you could use $filter=properties/parameters/Owner eq 'Adam#test.com' to do that if you make sure that parameters is existed.
Get https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Automation/automationAccounts/{automationAccountName}/jobs?api-version=2015-10-31&$filter=properties/parameters/Owner eq 'Adam#test.com'

Resources