I have a following JSON file/blob in the Azure Storage as students.json. Using Powershell, I want to read the contents of the file, student's FirstName and LastName where the MainSubject is Math
{
"college": {
"students": [
{
"FirstName": "abc",
"LastName": "xyz",
"MainSubject": "Computers"
},
{
"FirstName": "lmn",
"LastName": "opq"
"MainSubject": "Math"
}
]
}
}
I have reproduced in my environment and got expected results as below and followed Microsoft-Document and Thanks to # Ivan Yang's SO-thread :
$accountName = "rithwikstor"
$accountKey = "XX"
$containerName = "rithwik"
$blobName = "emo.json"
$context = New-AzStorageContext -StorageAccountName $accountName -StorageAccountKey $accountKey
$container_client = Get-AzStorageContainer -Name $containerName -Context $context
$source_blob_client = $container_client.CloudBlobContainer.GetBlockBlobReference($blobName)
$download_file = $source_blob_client.DownloadText()
$jsonContent = $download_file | ConvertFrom-Json
$ans=$jsonContent.college | ConvertTo-json
$res= $ans | ConvertFrom-json
$res.students | where{$_.Mainsubject -eq 'Math'}
$emo=$res.students | where{$_.Mainsubject -eq 'Math'}
$emo | Select-Object * -ExcludeProperty MainSubject
XX is the Storage account key. Got it from below:
Output:
Related
I am trying to get the registered providers for multiple subscriptions and output everything into one file.
For that, i am getting the subscriptions from a folder full of *.yaml files that contain information about the subscriptions, including their name.
What i cannot achieve is get the result for each subscriptions into an array and output that array into a text file. The script also allows the use of only one item from the folder in case that is the case.
Here is a sample code of a subscription and the code for it:
subscription1.yaml
name: subscription1
emailContact: email.address#domain.com
tags:
- key: "key1"
value: "Value1"
subscription2.yaml
name: subscription2
emailContact: email.address#domain.com
tags:
- key: "key1"
value: "Value1"
Folder structure where the yaml files is: ./landingZones/landingzone1/settings/dev/*.yaml
script:
param (
[Parameter(Mandatory = $false)]
[string]$Environment = 'dev',
[Parameter(Mandatory = $false)]
[string]$LandingZoneType = 'landingzone1',
[Parameter(Mandatory = $false)]
[string]$SingleSubscription
)
$scriptPath = Split-Path -parent $PSCommandPath
$subscriptionsEnvironmentDirectory = Get-ChildItem -Directory $scriptPath -Recurse -Filter "*$Environment*" | Where-Object { $_.parent.parent.Name -eq $LandingZoneType }
$subscriptions = Get-ChildItem -Path $($subscriptionsEnvironmentDirectory.FullName)
foreach ($subscription in ($subscriptions | Where-Object { ([System.String]::IsNullOrEmpty($SingleSubscription)) -or ($_.Name -replace "\.[^\.]+$", '') -eq $SingleSubscription })) {
$landingZone = Get-Content -Path $subscription.FullName | ConvertFrom-Yaml
# Set subscriptionName variable
$subscriptionName = $landingZone.name
$providers = az provider list --subscription $subscriptionName | ConvertFrom-Json
$defaultRegisteredProviders = 'Microsoft.ADHybridHealthService|Microsoft.Authorization|Microsoft.Billing|Microsoft.ClassicSubscription|Microsoft.Commerce|Microsoft.Consumption|Microsoft.CostManagement|Microsoft.Features|Microsoft.MarketplaceOrdering'
$registeredProviders = $providers | Where-Object { ($_.registrationState -eq 'Registered') -and ($_.namespace -notmatch $defaultRegisteredProviders) }
# Outputting result into txt file in the same directory where the command was executed
Write-Host ('{1}# Registered providers for subscription [{0}]' -f $subscriptionName, "`n")
$list = New-Object -TypeName 'System.Collections.ArrayList'
$sortedObjects = $registeredProviders | Sort-Object namespace | `
Format-Table `
#{l = 'Namespace'; e = { $_.namespace } }, `
#{l = "Subscription Id [$subscriptionName]"; e = { $_.id } }, `
#{l = 'Registration State'; e = { $_.registrationState } }, `
#{l = 'Registration Policy'; e = { $_.registrationPolicy } }
foreach ($i in $sortedObjects) {
$list.Add($i) | Out-Null
}
# Alternative to add into array:
# #($sortedObjects).foreach({$list.Add($_)}) | Out-Null
}
$list.Count
$list | Out-File .\registered_providers.txt -Force
The result is a file called 'registered_providers.txt' that contains only the registered providers for the first subscription in the foreach loop. I cannot get the contents of the second, third and so on in the same file, just a replaced text from the $sortedObjects
How do i create the array to contain all the info from all the subscriptions called?
Thanks
$list | Out-File Should be inside the for loop so that all the data you are fetching will be stored in the specified file.
Out-File cmdlet has -append parameter that appends the output to the existing file for every for loop happens. Otherwise, it will clean up the information/value stored in that output file.
Thanks to #jdweng for pointing the user to the right solution.
Refer to this MS Doc on Out-File -Append Parameter usage.
I've been trying to tweak and make a basic simple script just a bit more clean and overall to improve it.
Basically the initial script was iterating and doing a backup for each secret,certificate and key from each keyvault in a subscription.
I am trying to make it better by creating a function and trying to use it as such, unfortunately I'm still missing some thigns and I would like someone to help me sort this out:
function Get-Backup{
[CmdletBinding()]
param (
[Parameter()]
$Item,
[Parameter()]
$VaultName
)
$Items = az keyvault $Item list --vault-name $VaultName | ConvertFrom-Json
foreach ($Item in $Items) {
az keyvault $Item backup --vault-name $VaultName --name $Item.Name --file $Item/$Name.txt
}
}
$Vaults = az keyvault list | ConvertFrom-Json
foreach ($VaultName in $Vaults) {
Get-Backup("secret",$VaultName)
Get-Backup("certificate",$VaultName)
Get-Backup("key",$VaultName)
}
This doesn't work, and I don't understand really what I'm missing or doing wrong. the whole point of this would be to do a script that would automatically pick all the secrets all the keys and all the certificates in a vault, and do this for each vault.
I am trying to compile a function so I could reduce the main code and rely more on functions.
Unfortunatetly I can not post the error as it contains lots of identifiable information of my subscription name resource group etc. Starts with " ERROR: '' is misspelled or not recognized by the system. "
I am looking to do this myself but being stuck for a couple of days, I'd really appreaciate some hints and help.
Could you try solution given here. But please note that here you have
risk of having in-memory variables
Param(
[parameter(mandatory)] [string] $originVault,
[parameter(mandatory)] [string] $originSubscriptionId,
[parameter(mandatory)] [string] $destinationVault,
[parameter(mandatory)] [string] $destinationSubscriptionId,
[string] $disableDestinationSecrets = $true
)
# 1. Set the source subscription id.
Write-Host "Setting origin subscription to: $($originSubscriptionId)..."
az account set -s $originSubscriptionId
# 1.1 Get all secrets
Write-Host "Listing all origin secrets from vault: $($originVault)"
$originSecretKeys = az keyvault secret list --vault-name $originVault -o json --query "[].name" | ConvertFrom-Json
# 1.3 Loop secrets into PSCustomObjects, making it easy to work with later.
$secretObjects = $originSecretKeys | ForEach-Object {
Write-Host " - Getting secret value for '$($_)'"
$secret = az keyvault secret show --name $_ --vault-name $originVault -o json | ConvertFrom-Json
[PSCustomObject]#{
secretName = $_;
secretValue = $secret.value;
}#endcustomobject.
}#endforeach.
Write-Host "Done fetching secrets..."
# 2. Set the destination subscription id.
Write-Host "Setting destination subscription to: $($destinationSubscriptionId)..."
az account set -s $destinationSubscriptionId
# 2.2 Loop secrets objects, and set secrets in destination vault
Write-Host "Writing all destination secrets to vault: $($destinationVault)"
$secretObjects | ForEach-Object {
Write-Host " - Setting secret value for '$($_.secretName)'"
az keyvault secret set --vault-name $destinationVault --name "$($_.secretName)" --value "$($_.secretValue)" --disabled $disableDestinationSecrets -o none
}
# 3. Clean up
Write-Host "Cleaning up and exiting."
Remove-Variable secretObjects
Remove-Variable originSecretKeys
Write-Host "Finished."
or this one which uses powershell instead of azure cli.
$storageAccountTemplateFile = "https://raw.githubusercontent.com/srozemuller/Azure/main/AzureStorageAccount/azuredeploy.json"
$storageAccountTemplateParameters = "https://raw.githubusercontent.com/srozemuller/Azure/main/AzureStorageAccount/azuredeploy.parameters.json"
$backupFolder = "$env:Temp\KeyVaultBackup"
$location = "West Europe"
$backupLocationTag = "BackupLocation"
$backupContainerTag = "BackupContainer"
$global:parameters = #{
resourceGroupName = "RG-PRD-Backups-001"
location = $location
}
function backup-keyVaultItems($keyvaultName) {
#######Parameters
#######Setup backup directory
If ((test-path $backupFolder)) {
Remove-Item $backupFolder -Recurse -Force
}
####### Backup items
New-Item -ItemType Directory -Force -Path "$($backupFolder)\$($keyvaultName)" | Out-Null
Write-Output "Starting backup of KeyVault to a local directory."
###Certificates
$certificates = Get-AzKeyVaultCertificate -VaultName $keyvaultName
foreach ($cert in $certificates) {
Backup-AzKeyVaultCertificate -Name $cert.name -VaultName $keyvaultName -OutputFile "$backupFolder\$keyvaultName\certificate-$($cert.name)" | Out-Null
}
###Secrets
$secrets = Get-AzKeyVaultSecret -VaultName $keyvaultName
foreach ($secret in $secrets) {
#Exclude any secrets automatically generated when creating a cert, as these cannot be backed up
if (! ($certificates.Name -contains $secret.name)) {
Backup-AzKeyVaultSecret -Name $secret.name -VaultName $keyvaultName -OutputFile "$backupFolder\$keyvaultName\secret-$($secret.name)" | Out-Null
}
}
#keys
$keys = Get-AzKeyVaultKey -VaultName $keyvaultName
foreach ($kvkey in $keys) {
#Exclude any keys automatically generated when creating a cert, as these cannot be backed up
if (! ($certificates.Name -contains $kvkey.name)) {
Backup-AzKeyVaultKey -Name $kvkey.name -VaultName $keyvaultName -OutputFile "$backupFolder\$keyvaultName\key-$($kvkey.name)" | Out-Null
}
}
}
$keyvaults = Get-AzKeyVault
if ($keyvaults) {
if ($null -eq (get-AzResourceGroup $global:parameters.resourceGroupName -ErrorAction SilentlyContinue)) {
New-AzResourceGroup #global:parameters
}
if ($null -eq ($keyvaults | ? { $_.Tags.Keys -match $BackupLocationTag })) {
# if no backuplocation tags is available at any of the keyVaults we will create one first
$deployment = New-AzResourceGroupDeployment -ResourceGroupName $global:parameters.resourceGroupName -TemplateUri $storageAccountTemplateFile -TemplateParameterUri $storageAccountTemplateParameters
$backupLocation = $deployment.outputs.Get_Item("storageAccount").value
if ($deployment.ProvisioningState -eq "Succeeded") {
foreach ($keyvault in $keyvaults) {
$containerName = $keyvault.VaultName.Replace("-", $null).ToLower()
if (!(Get-aztag -ResourceId $keyvault.ResourceId | ? { $_.Tags.Keys -match $BackupLocationTag } )) {
Update-AzTag $keyvault.ResourceId -operation Merge -Tag #{BackupLocation = $backupLocation; BackupContainer = $containerName }
}
}
}
}
else {
foreach ($keyvault in $keyvaults) {
$backupLocation = (get-azkeyvault -VaultName $keyvault.vaultname | ? { $_.Tags.Keys -match $BackupLocationTag}).tags.Get_Item($BackupLocationTag)
$storageAccount = get-AzStorageAccount | ? { $_.StorageAccountName -eq $backupLocation }
if ($null -eq (Get-aztag -ResourceId $keyvault.ResourceId | ? { $_.Tags.Keys -match $BackupContainerTag } )) {
$containerName = $keyvault.VaultName.Replace("-", $null).ToLower()
Update-AzTag $keyvault.ResourceId -operation Merge -Tag #{BackupContainer = $containerName }
}
$containerName = (get-azkeyvault -VaultName $keyvault.vaultname | ? { $_.Tags.Keys -match $backupContainerTag }).tags.Get_Item($backupContainerTag)
if ($null -eq (Get-AzStorageContainer -Name $containerName -Context $storageAccount.context)) {
New-AzStorageContainer -Name $containerName -Context $storageAccount.context
}
backup-keyVaultItems -keyvaultName $keyvault.VaultName
foreach ($file in (get-childitem "$($backupFolder)\$($keyvault.VaultName)")) {
Set-AzStorageBlobContent -File $file.FullName -Container $containerName -Blob $file.name -Context $storageAccount.context -Force
}
}
}
}
I am trying to parse the Date time from +00 into UK format.
This is the script I'm using:
$rgName = "product1-website"
$accountName = "product1website"
$dtProvider = New-Object -TypeName System.Globalization.CultureInfo -ArgumentList 'en-GB'
$storageAccount = Get-AzStorageAccount -ResourceGroupName $rgName -Name $accountName
$ctx = $storageAccount.Context
Get-AzStorageContainer -Context $ctx |
Select-Object Name,
PublicAccess,
LastModified,
#{n = 'LastModifiedTime (Local)'; e = { Try { [DateTime]::Parse("$($_.LastModified)", $dtProvider) } Catch { $_.Exception.Message } } }
and this is the error it throws when the date is converted into MM/dd/yyyy:
How can I just parse and convert the UTC +00 into UTC +1 ?
Try this :
Get-AzStorageContainer -Context $ctx |
Select-Object Name,
PublicAccess,
LastModified,
#{n = 'LastModifiedTime (Local)'; e = { Try { [System.TimeZoneInfo]::ConvertTimeBySystemTimeZoneId( $_.LastModified , 'Central European Standard Time') } Catch { $_.Exception.Message } } }
Result(using time zone Central European Standard Time as UTC+1):
I am using below PowerShell script to read JSON data using REST API call from source. Now I want to load the data of $Result to the Azure Blob Storage. Any idea please?
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9
Regarding the issue, you can use the following ways
Save the JSON into one file then upload the file to Azure blob
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9
$Result | Out-File "D:\file.json"
$context=New-AzStorageContext -StorageAccountName "andyprivate" -StorageAccountKey ""
Set-AzStorageBlobContent -File "D:\file.json" `
-Container "" `
-Blob "file.json" `
-Context $context `
-StandardBlobTier Hot
Directly upload to Azure blob
$Params = #{
"URI" = 'https://3ea5e53b-817e-4c41-ae0b-c5afc1610f4e-bluemix.cloudant.com/test/_all_docs?include_docs=true'
}
$Result = Invoke-RestMethod #Params | ConvertTo-Json -Depth 9
Write-Host "the result is :"
$Result
$context=New-AzStorageContext -StorageAccountName "andyprivate" -StorageAccountKey ""
$container=Get-AzStorageContainer -Name "input" -Context $context
$content = [system.Text.Encoding]::UTF8.GetBytes($Result)
$container.CloudBlobContainer.GetBlockBlobReference("my.json").UploadFromByteArray($content,0,$content.Length)
I want to add StorageAccountName and its SKU to my report, this is the code I'm having now:
VMs = Get-AzureRmVM -Status
$vmOutput = $VMs | ForEach-Object {
[PSCustomObject]#{
"VM Name" = $_.name
"Private IP" =""
"VM Type" = $_.StorageProfile.osDisk.osType
"VM Profile" = $_.HardwareProfile.VmSize
"Environment" = $_.Tags.Environment
"Application" = $_.Tags.Application
"Decommission Date" = $_.Tags.Decomission
"OS Disk Size" = $_.StorageProfile.OsDisk.DiskSizeGB
"Data Disks Total Size" = ($_.StorageProfile.DataDisks.DiskSizeGB | Measure -Sum).Sum
"Data Disks Amount" = ($_.StorageProfile.DataDisks | Measure ).Count
"Managed OS Disk" = ($_.StorageProfile.OSDisk.ManagedDisk | Measure).Count
"Managed Data Disks" = ($_.StorageProfile.DataDisks.ManagedDisk | Measure).Count
"Powerstate" = $_.PowerState
}
}
$nics = get-azurermnetworkinterface | where VirtualMachine -NE $null
$ips =#{}
foreach($nic in $nics){
$vm = $VMs | Where-Object -Property id -EQ $nic.VirtualMachine.id
$prv = $nic.IpConfigurations | select-object -ExpandProperty PrivateIpAddress
$ips.Add($vm.Name,$prv)
}
foreach($vm in $vmOutput)
{
if($ips.ContainsKey($vm."VM Name"))
{
$vm."Private IP"=$ips[$vm."VM Name"]
}
}
$vmOutput | sort "Environment", "VM Type", "VM Profile", "Application" | export-csv VMReport.csv -delimiter ";" -force -notypeinformation
I tried to do it more less the same way IP addresses were implemented into this code, but it's much harder than I expected, as Get-AzureRmStorageAccount cmdlet doesn't store VirtualMachine.Id property like Get-AzureRMNetworkInterface cmdlet does.
Any ideas how can I merge it into one table? Is there any key upon which I can join both information? What I want to achieve is to add StorageAccountName and SKUName columns into the $vmOutput table.
Update:
$sat =#{}
$OutFile = "..."
#part1
foreach ($vmdetails in $VMs)
{
$ResourceGroupName=$vmdetails.ResourceGroupName
$VMName=$vmdetails.name
$storage=$vmdetails.StorageProfile.OsDisk.Vhd.Uri
$DataDiks=$vmdetails.StorageProfile.OsDisk.Name
$ss=$storage.split('/')[2]
$OSStorageAccountName=$ss.split('.')[0]
$DiskType="OSDisk"
$StroageAccountLocation=($StroageAccDetail | where {$_.StorageAccountName -eq $OSStorageAccountName }).location
$StorageAccountType=($StroageAccDetail | where {$_.StorageAccountName -eq $OSStorageAccountName }).AccountType
"$sub,$ResourceGroupName,$VMName,$DataDiks,$DiskType,$OSStorageAccountName" | Out-File -FilePath $OutFile -Append
$vmdatadisks=$vmdetails.StorageProfile.DataDisks
if($vmdatadisks -ne $null){
foreach($vmDatadsik in $vmdatadisks)
{
$vmDatadsikss=$vmDatadsik.vhd.uri
$DiskType="DataDisk"
$ss=$vmDatadsikss.split('/')[2]
$DataDiks=$vmDatadsik.Name
$dataStorageAccountName=$ss.split('.')[0]
"$sub,$ResourceGroupName,$VMName,$DataDiks,$DiskType,$OSStorageAccountName" | Out-File -FilePath $OutFile -Append
}
}
}
#part2
$VMs = Get-AzureRmVM -ResourceGroupName "..." -Name "..."
$storageAccountName = $VMs.StorageProfile.OsDisk.Vhd.Uri.Split("/")[2].Split(".")[0]
Get-AzureRmStorageAccount -StorageAccountName $storageAccountName -ResourceGroupName "..."
Please correct me if I'm misunderstanding you.
Assume the sku you refer to the account storage's sku, you can also add placeholders in the $vmOutput for SA Name and SKUName.
$VMs = Get-AzureRmVM -Status
$vmOutput = $VMs | ForEach-Object {
[PSCustomObject]#{
"VM Name" = $_.name
"ResourceGroupName" =$_.ResourceGroupName #you must include this parameter
"storage" = $_.StorageProfile.OsDisk.Vhd.Uri #you must include this parameter
#your other property
#add placeholder for SA Name and SKUName like below
"StorageAccountName" =""
"StorageAccountSKUName" =""
}
}
Then, iterate all the vms in $vmoutput, and find the related storage account and sku, then add value to the $vmoutput:
foreach($v in $vmOutput){
$resourceGroups = $v.ResourceGroupName
$storage=$v.storage
if($storage -ne $null)
{
$ss=$storage.split('/')[2]
$OSStorageAccountName=$ss.split('.')[0]
$s1 = Get-AzureRmStorageAccount -ResourceGroupName $resourceGroups -StorageAccountName $OSStorageAccountName
#add the value to $vmoutput
$v.StorageAccountName =$OSStorageAccountName
$v.StorageAccountSKUName=$s1.Sku.name
}
else
{
$v.StorageAccountName ="no value"
$v.StorageAccountSKUName="no value"
}
}