I am using PowerShell code to hit public API but there is pagination meaning there are around 400000 pages but I am exporting to csv I am getting duplicate value.
My Code :
$uri = 'https://prices.azure.com/api/retail/prices'
$Headers = #{}
$Items = #()
Do
{
$r = Invoke-WebRequest -Method 'Get' -uri $uri -Header $Headers -UseBasicParsing # -Verbose
$c = $r.Content | ConvertFrom-Json
$Items += $c.Items
$Count += $c.Count
$uri = $c.NextPageLink
$uri
} While ($c.NextPageLink)
$Items | Export-CSV -Path ("$(get-date -format "Retail_yyyy_mm")" +'.csv') -Force
When I am checking the csv file there 100000 records that are duplicate. can someone guide me what I am doing wrong here.
Related
Have a nice day everyone!
I have a VMware Windows Which has permission in key vault and I want to collect all key secrets but the code below when it finished just has ID + Attributes not consist value of Key secrets. Anyone can help me finish the code below to get all key secrets.
Many thanks for your help!
$RresourceUrl = 'dddd.vault.azure.net'
# Compose REST request.
$response = Invoke-WebRequest -Uri 'http://169.254.111.211/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fvault.azure.net' -Method GET -Headers #{Metadata="true"}
$OAuth = $response.Content | ConvertFrom-Json
# Check if authentication was successfull.
if ($OAuth.access_token) {
#Format headers.
$HeaderParams = #{
'Content-Type' = "application\json"
'Authorization' = "Bearer $($OAuth.access_token)"
}
# Create an empty array to store the result.
$QueryResults = #()
$Uri = "https://$RresourceUrl/secrets?api-version=2016-10-01"
# Invoke REST method and fetch data until there are no pages left.
do {
$Results = Invoke-WebRequest -Uri $Uri -Method GET -Headers $HeaderParams | ConvertFrom-Json
$Results.nextLink
if ($Results.value) {
$QueryResults += $Results.value
}
else {
$QueryResults += $Results
}
$Uri = $Results.nextLink
} until (!($Uri))
# Return the result.
$QueryResults | Export-Csv -NoTypeInformatio *\Documents\Tesst.csv
}
else {
Write-Error "No Access Token"
}
This is my final code maybe isn't optimized but it worked.
$RresourceUrl = 'devakv01.vault.azure.net'
# Compose REST request.
$response = Invoke-WebRequest -Uri 'http://169.254.111.211/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fvault.azure.net' -Method GET -Headers #{Metadata="true"}
$OAuth = $response.Content | ConvertFrom-Json
# Check if authentication was successfull.
if ($OAuth.access_token) {
#Format headers.
$HeaderParams = #{
'Content-Type' = "application\json"
'Authorization' = "Bearer $($OAuth.access_token)"
}
# Create an empty array to store the result.
$QueryResults = #()
$Uri = "https://$RresourceUrl/secrets?api-version=2016-10-01"
# Invoke REST method and fetch data until there are no pages left.
do {
$Results = Invoke-WebRequest -Uri $Uri -Method GET -Headers $HeaderParams | ConvertFrom-Json
$Results.nextLink
if ($Results.value) {
$QueryResults += $Results.value
}
else {
$QueryResults += $Results
}
$Uri = $Results.nextLink
} until (!($Uri))
# Return the result.
$QueryResults
}
else {
Write-Error "No Access Token"
}
# get Key after to have secrets name
$output = ForEach ($nameSecret in $QueryResults.id)
{
$Resultskey = Invoke-WebRequest -Uri $($nameSecret+'?api-version=2016-10-01') -Method GET -Headers $HeaderParams | ConvertFrom-Json
$Resultskey
}
$output | Export-Csv -NoTypeInformatio *\$RresourceUrl.csv
I cannot figure this on out. I need a way to assign Endpoint Manager's Scope tags to an Azure AD group using Microsoft Graph and PowerShell.
Under the portal this is done under Endpoint Manager\Tenant Administration\Roles\Scope (Tags). Then clicking on the Tag and tgo to assignments and browse to Azure AD group.
Since its under Roles, I'm assuming it falls under the roleAssignment or roleScopeTag resource types?
I have thoroughly read all documentation for the REST api and I have also attempted to do this via Microsoft.Graph.Intune modules but still cannot find a suitable cmdlet that will do this. Am I missing something?
Here is the current code I have built following this document
FIRST let's assume I have the correct tag id and azure ad group object id.
$ScopeTagId = 2
$TargetGroupIds = #()
$TargetGroupIds += '687c08f1-e78f-4506-b4a6-dfe35a05d138'
$graphApiVersion = "beta"
$Resource = "deviceManagement/roleScopeTags"
$object = New-Object -TypeName PSObject
$object | Add-Member -MemberType NoteProperty -Name 'assignments' -Value #($TargetGroupIds)
$JSON = $object | ConvertTo-Json
$uri = "https://graph.microsoft.com/$graphApiVersion/$($Resource)/$ScopeTagId/assign"
Invoke-RestMethod -Method Post -Uri $uri -Headers $global:authToken -Body $JSON
No success. I then thought that since the create roleScopeTag API doesn't have an assignment property in the request body, this must be done using the update method, but that doesn't have it in there either. The only one I read was to use the assign action and in the documentation example it shows the roleScopeTagAutoAssignment URI, so I went down that rabbit hole:
$ScopeTagId = 2
$TargetGroupIds = #()
$TargetGroupIds += '687c08f1-e78f-4506-b4a6-dfe35a05d138'
$graphApiVersion = "beta"
$Resource = "deviceManagement/roleScopeTags"
$AutoTagObject = #()
foreach ($TargetGroupId in $TargetGroupIds) {
#Build custom object for assignment
$AssignmentProperties = "" | Select '#odata.type',id,target
$AssignmentProperties.'#odata.type' = '#microsoft.graph.roleScopeTagAutoAssignment'
$AssignmentProperties.id = $TargetGroupId
#Build custom object for target
$targetProperties = "" | Select "#odata.type",deviceAndAppManagementAssignmentFilterId,deviceAndAppManagementAssignmentFilterType
$targetProperties."#odata.type" = "microsoft.graph.deviceAndAppManagementAssignmentTarget"
$targetProperties.deviceAndAppManagementAssignmentFilterId = $TargetGroupId
$targetProperties.deviceAndAppManagementAssignmentFilterType = 'include'
#add target object to assignment
$AssignmentProperties.target = $targetProperties
$AutoTagObject += $AssignmentProperties
}
#build body object
$object = New-Object -TypeName PSObject
$object | Add-Member -MemberType NoteProperty -Name 'assignments' -Value #($AutoTagObject)
$JSON = $object | ConvertTo-Json
$uri = "https://graph.microsoft.com/$graphApiVersion/$($Resource)/$ScopeTagId/assign"
Invoke-RestMethod -Method Post -Uri $uri -Headers $global:authToken -Body $JSON
The error I get back is something like: "Property target in payload has a value that does not match schema.","innerError":{"date":"2022-01-27T16:28:34","request-id":"75626c4d-f09b-438e-8b0f-0b2928ac23ce","client-request-id":"75626c4d-f09b-438e-8b0f-0b2928ac23ce" which I assume is the odata.type object i'm calling "microsoft.graph.deviceAndAppManagementAssignmentTarget"
There is a second post method in the documentations via roledefinition URI which seems like an unnecessary step, but I tried that too with no success.
I do not know if any of this is correct. I understand the API calls for others pretty well and I have been able to successfully add Tags for Custom Roles and their assignments using graph; I just can't seem to find the right combination of URI and JSON body for scope tags themselves...if it even exists. :(
Any ideas, please share some code snippets if you can. THANKS!
I found the correct URI and request body
If can be generated like this:
$ScopeTagId = 2
$TargetGroupIds = #()
$TargetGroupIds += '687c08f1-e78f-4506-b4a6-dfe35a05d138'
$graphApiVersion = "beta"
$Resource = "deviceManagement/roleScopeTags"
$AutoTagObject = #()
#TEST $TargetGroupId = $TargetGroupIds[0]
foreach ($TargetGroupId in $TargetGroupIds) {
#Build custom object for assignment
$AssignmentProperties = "" | Select id,target
$AssignmentProperties.id = ($TargetGroupId + '_' + $ScopeTagId)
#Build custom object for target
$targetProperties = "" | Select "#odata.type",deviceAndAppManagementAssignmentFilterId,deviceAndAppManagementAssignmentFilterType,groupId
$targetProperties."#odata.type" = "microsoft.graph.groupAssignmentTarget"
$targetProperties.deviceAndAppManagementAssignmentFilterId = $null
$targetProperties.deviceAndAppManagementAssignmentFilterType = 'none'
$targetProperties.groupId = $TargetGroupId
#add target object to assignment
$AssignmentProperties.target = $targetProperties
$AutoTagObject += $AssignmentProperties
}
#build body object
$object = New-Object -TypeName PSObject
$object | Add-Member -MemberType NoteProperty -Name 'assignments' -Value #($AutoTagObject)
$JSON = $object | ConvertTo-Json -Depth 10
$uri = "https://graph.microsoft.com/$graphApiVersion/$($Resource)/$ScopeTagId/assign"
Invoke-RestMethod -Method Post -Uri $uri -Headers $global:authToken -Body $JSON -ErrorAction Stop
The Json request body would look like this:
{
"assignments": [
{
"id": "b25c80e3-78cc-4b7c-888e-fc50dcc6b582_2",
"target": {
"#odata.type": "microsoft.graph.groupAssignmentTarget",
"deviceAndAppManagementAssignmentFilterId": null,
"deviceAndAppManagementAssignmentFilterType": "none",
"groupId": "b25c80e3-78cc-4b7c-888e-fc50dcc6b582"
}
}
]
}
Nowhere is this documented...
I have an Azure build pipeline with the following powershell task running on the agent to get the commits of my PR:
$id = $(System.PullRequest.PullRequestId)
$uri = "$(System.TeamFoundationCollectionUri)NPI/_apis/git/repositories/$(Build.Repository.ID)/pullRequests/$id/commits?api-version=5.1"
Write-Host "Getting all commits for PR" $id ":" $uri
$J = Invoke-WebRequest -URI $uri -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
} | ConvertFrom-Json
And then loop through the reply like this:
foreach ($c in $J.value)
{
$id = $c.commitId
$uri = "$(System.TeamFoundationCollectionUri)NPI/_apis/git/repositories/$(Build.Repository.ID)/commits/$id/changes?api-version=5.1"
Write-Host "Getting details for commit" $id ":" $uri
$J = Invoke-WebRequest -URI $uri -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
} | ConvertFrom-Json
Write-Host $J
}
The log file of the Azure task contains this:
Getting all commits for PR 1026 : https://dev.azure.com/xxx/xxx/_apis/git/repositories/xxx/pullRequests/1026/commits?api-version=5.1
Getting details for commit xxx : https://dev.azure.com/xxx/xxx/_apis/git/repositories/xxx/commits/xxx/changes?api-version=5.1
#{changeCounts=; changes=System.Object[]}
As you can see, there are no changes listed. However, if I click on the logged url for getting the commit details, I get a non-empty list. What's going on here?
Oops, turns out it does actually work but the Write-Host output is broken.
This runs fine:
foreach ($cc in $J.changes)
{
Write-Host $cc.item.path
}
You need a variable to store it before convert to json. This is work correctly to me.
"$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/git/repositories/$($pipeline.repository.id)/commits/$(Build.BuildId)/changes?api-version=5.0"
Write-Host $commitUrl
$rsCommit = Invoke-RestMethod -Uri $commitUrl -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
Write-Host "CommitInfo = $($rsCommit | ConvertTo-Json -Depth 100)"
Each time the loop executes the file is being replaced in the place of the first file . I want to upload it as a new file without distrubing the existing ones..
foreach ($blob in $blobs)
{
$file=New-TemporaryFile
$file=Get-AzureStorageBlobContent -Container $container_name -Blob $blob.Name -Context $ctx -Destination $localFile -Force
$contents = Get-Content $localFile -Raw -ErrorAction:SilentlyContinue
$f=New-TemporaryFile
Add-Content $f $contents
$Header = #{
"Content-Disposition"="attachment;filename=$($blob.Name)"
"Authorization"=$accessToken
}
Invoke-RestMethod -Uri $apiUrl -Headers $Header -Method put -InFile $f
}
I think you over-did it with the New-TemporaryFile commands in your code.
Why use Get-AzureStorageBlobContent to store blob content in a file and then create another temp file to copy the contents to?
This would be much simpler:
foreach ($blob in $blobs) {
$localFile = New-TemporaryFile
Get-AzureStorageBlobContent -Container $container_name -Blob $blob.Name -Context $ctx -Destination $localFile.FullName -Force
$Header = #{
"Content-Disposition"="attachment;filename=$($blob.Name)"
"Authorization"=$accessToken
}
Invoke-RestMethod -Uri $apiUrl -Headers $Header -Method put -InFile $localFile.FullName
}
Edit
The New-TemporaryFile command returns a System.IO.FileInfo object which you cannot use directly as the InFile parameter. Instead, it expects a full path and file name, so we use $localFile.FullName
I've developed an WebApp (API) which is hosted in Azure & uses VSTS for CI/CD/version control.
I'd like to give the customer (owner) of this API the ability to update various configuration/data files under wwwroot, however I'd like those files to be under version control (source of truth - a separate repository to the API source code). Creating/Updating/Deleting one of those files in the repository should cause the file to be uploaded/removed in the WebApp (in a folder under wwwroot).
Modifying (creating/deleting) one of these files should not trigger a full redeployment (of the WebApp)
How can I achieve this? So far I've thought about a VSTS release pipeline for a GIT artefact however I couldn't see a low-friction way to make the changes in the Azure webapp (KUDU API seems a bit complex and heavy-handed)
**EDIT: ** Sample PowerShell script to sync Configuration files in WebApp w/ a build-artefact (PUT/DELETE only invoked when necessary).
# The idea behind this script is to synchronize the configuration files on the server with what's in the repo, only updating files where necessary
param (
[string]$resourceGroupName = "XXX",
[Parameter(Mandatory=$true)][string]$webAppName,
[Parameter(Mandatory=$true)][string]$latestConfigFilesPath
)
function Get-AzureRmWebAppPublishingCredentials($resourceGroupName, $webAppName, $slotName = $null) {
if ([string]::IsNullOrWhiteSpace($slotName)) {
$resourceType = "Microsoft.Web/sites/config"
$resourceName = "$webAppName/publishingcredentials"
} else {
$resourceType = "Microsoft.Web/sites/slots/config"
$resourceName = "$webAppName/$slotName/publishingcredentials"
}
$publishingCredentials = Invoke-AzureRmResourceAction -ResourceGroupName $resourceGroupName -ResourceType $resourceType -ResourceName $resourceName -Action list -ApiVersion 2015-08-01 -Force
return $publishingCredentials
}
function Get-KuduApiAuthorisationHeaderValue($resourceGroupName, $webAppName, $slotName = $null) {
$publishingCredentials = Get-AzureRmWebAppPublishingCredentials $resourceGroupName $webAppName $slotName
return ("Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $publishingCredentials.Properties.PublishingUserName, $publishingCredentials.Properties.PublishingPassword))))
}
function Get-KuduInode($kuduHref) {
return Invoke-RestMethod -Uri $kuduHref `
-Headers #{"Authorization"=$kuduApiAuthorisationToken;"If-Match"="*"} `
-Method GET `
-ContentType "application/json"
}
function Get-AllFilesUnderKuduHref($kuduHref) {
$result = #()
$inodes = (Get-KuduInode $kuduHref)
Foreach ($inode in $inodes) {
if ($inode.mime -eq "inode/directory") {
$result += (Get-AllFilesUnderKuduHref $inode.href)
} else {
$result += $inode.href
}
}
return $result
}
function Get-LocalPathForUri([System.Uri]$uri) {
$latestConfigFilesUri = [System.Uri]$latestConfigFilesPath
$localFileUri = [System.Uri]::new($latestConfigFilesUri, $uri)
return $localFileUri.LocalPath
}
function Get-RemoteUri([System.Uri]$uri) {
return [System.Uri]::new($configurationHref, $uri)
}
function Files-Identical($uri) {
$localFilePath = Get-LocalPathForUri $uri
$localFileHash = Get-FileHash $localFilePath -Algorithm MD5
# Download the remote file so that we can calculate the hash. It doesn't matter that it doesn't get cleaned up, this is running on a temporary build server anyway.
$temporaryFilePath = "downloded_kudu_file"
$remoteFileUri = [System.Uri]::new($configurationHref, $uri)
Invoke-RestMethod -Uri $remoteFileUri `
-Headers #{"Authorization"=$kuduApiAuthorisationToken;"If-Match"="*"} `
-Method GET `
-OutFile $temporaryFilePath `
-ContentType "multipart/form-data"
$remoteFileHash = Get-FileHash $temporaryFilePath -Algorithm MD5
return $remoteFileHash.Hash -eq $localFileHash.Hash
}
function CalculateRelativePath([System.Uri]$needle, [System.Uri]$haystack) {
return $haystack.MakeRelativeUri($needle).ToString();
}
function Put-File([System.Uri]$uri) {
Write-Host "Uploading file $uri"
$localFilePath = Get-LocalPathForUri $uri
$remoteFileUri = Get-RemoteUri $uri
Invoke-RestMethod -Uri $remoteFileUri `
-Headers #{"Authorization"=$kuduApiAuthorisationToken;"If-Match"="*"} `
-Method PUT `
-InFile $localFilePath `
-ContentType "multipart/form-data"
}
function Delete-File([System.Uri]$uri) {
Write-Host "Deleting file $uri"
$remoteFileUri = Get-RemoteUri $uri
Invoke-RestMethod -Uri $remoteFileUri `
-Headers #{"Authorization"=$kuduApiAuthorisationToken;"If-Match"="*"} `
-Method DELETE `
}
# Script begins here
$configurationHref = [System.Uri]"https://$webAppName.scm.azurewebsites.net/api/vfs/site/wwwroot/Configuration/"
$kuduApiAuthorisationToken = Get-KuduApiAuthorisationHeaderValue -resourceGroupName $resourceGroupName -webAppName $webAppName
$filenamesOnServer = Get-AllFilesUnderKuduHref $configurationHref $kuduApiAuthorisationToken | % { $configurationHref.MakeRelativeUri($_).OriginalString }
Write-Host "Files currently on server" $filenamesOnServer
$filesCurrentlyInRepo = Get-ChildItem -Path $latestConfigFilesPath -Recurse -File
$filenamesInRepo = $filesCurrentlyInRepo | Select-Object -ExpandProperty FullName | % { CalculateRelativePath $_ $latestConfigFilesPath}
Write-Host "Files currently in repo" $filenamesInRepo
$intersection = $filenamesOnServer | ?{$filenamesInRepo -contains $_}
Write-Host "Intersection: " $intersection
$onlyOnServer = $filenamesOnServer | ?{-Not($filenamesInRepo -contains $_)}
$onlyInRepo = $filenamesInRepo | ?{-Not($filenamesOnServer -contains $_)}
Write-Host "Only on server" $onlyOnServer
Write-Host "Only in repo" $onlyInRepo
Write-Host
Foreach ($uri in $onlyInRepo) {
Put-File $uri
}
Foreach ($uri in $onlyOnServer) {
Delete-File $uri
}
Foreach ($uri in $intersection) {
if (-Not (Files-Identical $uri)) {
Write-Host "Configuration file $uri needs updating"
Put-File $uri
} else {
Write-Host "Configuration file $uri is identical, skipping"
}
}
Write-Host "Sync complete"
With Azure App Service deploy task, you can upload files to app service, but can’t delete files (Uncheck Publish using Web Deploy optioin and specify the folder path in Package or folder input box).
So, the better way is using Kudu API to delete/upload files during build/release.
There is the thread and a blog about using Kudu API:
How to access Kudu in Azure using power shell script
Interacting with Azure Web Apps Virtual File System using PowerShell and the Kudu API