Azure File Storage deletion calculation - azure

I am trying to delete Files out of the Azure File Storage that are that are 30 + 1 on the first day of the month or older.
I have basic list and remove script that works. My main question is how do I do a calculation for a if older than statement?
$resourceGroupName=""
$storageAccName=""
$fileShareName=""
$directoryPath=""
## Function to Lists directories and files
Function GetFiles
{
Write-Host -ForegroundColor Green "Lists directories and files.."
## Get the storage account context
$ctx=(Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccName).Context
## List directories
$directories=Get-AZStorageFile -Context $ctx -ShareName $fileShareName
## Loop through directories
foreach($directory in $directories)
{
write-host -ForegroundColor Magenta " Directory Name: " $directory.Name
$files=Get-AZStorageFile -Context $ctx -ShareName $fileShareName -Path $directory.Name | Get-AZStorageFile
## Loop through all files and display
foreach ($file in $files)
{
write-host -ForegroundColor Yellow $file.Name
}
}
}
GetFiles
$context = ""
Remove-AzStorageFile -ShareName "name" -Path "path" -Context $context
Get-ChildItem -Path $path -Recurse -Force | Where-Object { !$_.PSIsContainer -and $_.CreationTime -lt $limit } | Remove-Item -Force

We do have AZ CLI command to delete files older than XX days.
// Delete old files block
$filelist = az storage file list -s $myshare --account-name $accountName --account-key $accountKey
$fileArray = $filelist | ConvertFrom-Json
foreach ($file in $fileArray | Where-Object {$_.properties.lastModified.DateTime -lt ((Get-Date).AddDays(-31))})
{
$removefile = $file.name
if ($removefile -ne $null)
{
Write-Host "Removing file $removefile"
az storage file delete -s $myshare -p $removefile
}
}
Reference So Thread: Use Azure Cli to Delete Old Files in Azure file share - Stack Overflow

Related

How to convert JSON file content into Powershell object in Powershell runbook?

I'm trying to convert JSON file which is present in storage account data into Powershell object. But I'm not getting the proper output. Output does not contain proper values.
My code:
$storageAccountKey = "xxxx"
$Context = New-AzStorageContext -StorageAccountName 'xxxx' -StorageAccountKey $storageAccountKey
$b='C:\Temp\Scheduled.json'
Get-AzureStorageFileContent -Path $b -ShareName "file-share-name"
$newScheduledRules = Get-Content -Raw $b | ConvertFrom-Json
Write-Output ("########newScheduledRules::###########" + $newScheduledRules)
Output:
Could not get the storage context. Please pass in a storage context or set the current storage context.
########newScheduledRules::############{Scheduled=System.Object[]; Fusion=System.Object[]; MLBehaviorAnalytics=System.Object[]; MicrosoftSecurityIncidentCreation=System.Object[]}
I have reproduced in my environment and got expected results as below and followed below process and followed Microsoft-Document:
Scheduled.json:
{
"Rithwik":"Hello",
"Chotu":"Bojja"
}
Firstly, I have created Storage account and then added a Scheduled.json in file share as below:
Now i have created a runbook and excuted below script in runbook as below:
$storageAccountKey = "PFHxFbVmAEvwBM6/9kW4nORJYA+AStA2QQ1A=="
$Context = New-AzStorageContext -StorageAccountName 'rithwik' -StorageAccountKey $storageAccountKey
$out = "$($env:TEMP)\$([guid]::NewGuid())"
New-Item -Path $out -ItemType Directory -Force
Get-AzStorageFileContent -ShareName "rithwik" -Context $Context -Path 'Scheduled.json' -Destination $out -Force
$newScheduledRules = Get-Content -Path "$out\Scheduled.json" -Raw | ConvertFrom-Json
Write-Output ("########newScheduledRules::###########" + $newScheduledRules)
Output:
Here $out is the Destination Variable.
-Path should be Only the file name Scheduled.json in
Get-AzStorageFileContent command.
It seems the Get-AzureStorageFileContent is missing -Context parameter. It should be something like this
$OutPath = "$($env:TEMP)\$([guid]::NewGuid())"
New-Item -Path $OutPath -ItemType Directory -Force
$storageContext = (Get-AzStorageAccount -ResourceGroupName xxxx -Name xxxx).Context
Get-AzStorageFileContent -ShareName "file-share-name" -Context $storageContext -Path 'Scheduled.json' -Destination $OutPath -Force
$newScheduledRules = Get-Content -Path "$OutPath\Scheduled.json" -Raw | ConvertFrom-Json
Write-Output ("########newScheduledRules::###########" + $newScheduledRules)

Script for backing up keyvault

I've been trying to tweak and make a basic simple script just a bit more clean and overall to improve it.
Basically the initial script was iterating and doing a backup for each secret,certificate and key from each keyvault in a subscription.
I am trying to make it better by creating a function and trying to use it as such, unfortunately I'm still missing some thigns and I would like someone to help me sort this out:
function Get-Backup{
[CmdletBinding()]
param (
[Parameter()]
$Item,
[Parameter()]
$VaultName
)
$Items = az keyvault $Item list --vault-name $VaultName | ConvertFrom-Json
foreach ($Item in $Items) {
az keyvault $Item backup --vault-name $VaultName --name $Item.Name --file $Item/$Name.txt
}
}
$Vaults = az keyvault list | ConvertFrom-Json
foreach ($VaultName in $Vaults) {
Get-Backup("secret",$VaultName)
Get-Backup("certificate",$VaultName)
Get-Backup("key",$VaultName)
}
This doesn't work, and I don't understand really what I'm missing or doing wrong. the whole point of this would be to do a script that would automatically pick all the secrets all the keys and all the certificates in a vault, and do this for each vault.
I am trying to compile a function so I could reduce the main code and rely more on functions.
Unfortunatetly I can not post the error as it contains lots of identifiable information of my subscription name resource group etc. Starts with " ERROR: '' is misspelled or not recognized by the system. "
I am looking to do this myself but being stuck for a couple of days, I'd really appreaciate some hints and help.
Could you try solution given here. But please note that here you have
risk of having in-memory variables
Param(
[parameter(mandatory)] [string] $originVault,
[parameter(mandatory)] [string] $originSubscriptionId,
[parameter(mandatory)] [string] $destinationVault,
[parameter(mandatory)] [string] $destinationSubscriptionId,
[string] $disableDestinationSecrets = $true
)
# 1. Set the source subscription id.
Write-Host "Setting origin subscription to: $($originSubscriptionId)..."
az account set -s $originSubscriptionId
# 1.1 Get all secrets
Write-Host "Listing all origin secrets from vault: $($originVault)"
$originSecretKeys = az keyvault secret list --vault-name $originVault -o json --query "[].name" | ConvertFrom-Json
# 1.3 Loop secrets into PSCustomObjects, making it easy to work with later.
$secretObjects = $originSecretKeys | ForEach-Object {
Write-Host " - Getting secret value for '$($_)'"
$secret = az keyvault secret show --name $_ --vault-name $originVault -o json | ConvertFrom-Json
[PSCustomObject]#{
secretName = $_;
secretValue = $secret.value;
}#endcustomobject.
}#endforeach.
Write-Host "Done fetching secrets..."
# 2. Set the destination subscription id.
Write-Host "Setting destination subscription to: $($destinationSubscriptionId)..."
az account set -s $destinationSubscriptionId
# 2.2 Loop secrets objects, and set secrets in destination vault
Write-Host "Writing all destination secrets to vault: $($destinationVault)"
$secretObjects | ForEach-Object {
Write-Host " - Setting secret value for '$($_.secretName)'"
az keyvault secret set --vault-name $destinationVault --name "$($_.secretName)" --value "$($_.secretValue)" --disabled $disableDestinationSecrets -o none
}
# 3. Clean up
Write-Host "Cleaning up and exiting."
Remove-Variable secretObjects
Remove-Variable originSecretKeys
Write-Host "Finished."
or this one which uses powershell instead of azure cli.
$storageAccountTemplateFile = "https://raw.githubusercontent.com/srozemuller/Azure/main/AzureStorageAccount/azuredeploy.json"
$storageAccountTemplateParameters = "https://raw.githubusercontent.com/srozemuller/Azure/main/AzureStorageAccount/azuredeploy.parameters.json"
$backupFolder = "$env:Temp\KeyVaultBackup"
$location = "West Europe"
$backupLocationTag = "BackupLocation"
$backupContainerTag = "BackupContainer"
$global:parameters = #{
resourceGroupName = "RG-PRD-Backups-001"
location = $location
}
function backup-keyVaultItems($keyvaultName) {
#######Parameters
#######Setup backup directory
If ((test-path $backupFolder)) {
Remove-Item $backupFolder -Recurse -Force
}
####### Backup items
New-Item -ItemType Directory -Force -Path "$($backupFolder)\$($keyvaultName)" | Out-Null
Write-Output "Starting backup of KeyVault to a local directory."
###Certificates
$certificates = Get-AzKeyVaultCertificate -VaultName $keyvaultName
foreach ($cert in $certificates) {
Backup-AzKeyVaultCertificate -Name $cert.name -VaultName $keyvaultName -OutputFile "$backupFolder\$keyvaultName\certificate-$($cert.name)" | Out-Null
}
###Secrets
$secrets = Get-AzKeyVaultSecret -VaultName $keyvaultName
foreach ($secret in $secrets) {
#Exclude any secrets automatically generated when creating a cert, as these cannot be backed up
if (! ($certificates.Name -contains $secret.name)) {
Backup-AzKeyVaultSecret -Name $secret.name -VaultName $keyvaultName -OutputFile "$backupFolder\$keyvaultName\secret-$($secret.name)" | Out-Null
}
}
#keys
$keys = Get-AzKeyVaultKey -VaultName $keyvaultName
foreach ($kvkey in $keys) {
#Exclude any keys automatically generated when creating a cert, as these cannot be backed up
if (! ($certificates.Name -contains $kvkey.name)) {
Backup-AzKeyVaultKey -Name $kvkey.name -VaultName $keyvaultName -OutputFile "$backupFolder\$keyvaultName\key-$($kvkey.name)" | Out-Null
}
}
}
$keyvaults = Get-AzKeyVault
if ($keyvaults) {
if ($null -eq (get-AzResourceGroup $global:parameters.resourceGroupName -ErrorAction SilentlyContinue)) {
New-AzResourceGroup #global:parameters
}
if ($null -eq ($keyvaults | ? { $_.Tags.Keys -match $BackupLocationTag })) {
# if no backuplocation tags is available at any of the keyVaults we will create one first
$deployment = New-AzResourceGroupDeployment -ResourceGroupName $global:parameters.resourceGroupName -TemplateUri $storageAccountTemplateFile -TemplateParameterUri $storageAccountTemplateParameters
$backupLocation = $deployment.outputs.Get_Item("storageAccount").value
if ($deployment.ProvisioningState -eq "Succeeded") {
foreach ($keyvault in $keyvaults) {
$containerName = $keyvault.VaultName.Replace("-", $null).ToLower()
if (!(Get-aztag -ResourceId $keyvault.ResourceId | ? { $_.Tags.Keys -match $BackupLocationTag } )) {
Update-AzTag $keyvault.ResourceId -operation Merge -Tag #{BackupLocation = $backupLocation; BackupContainer = $containerName }
}
}
}
}
else {
foreach ($keyvault in $keyvaults) {
$backupLocation = (get-azkeyvault -VaultName $keyvault.vaultname | ? { $_.Tags.Keys -match $BackupLocationTag}).tags.Get_Item($BackupLocationTag)
$storageAccount = get-AzStorageAccount | ? { $_.StorageAccountName -eq $backupLocation }
if ($null -eq (Get-aztag -ResourceId $keyvault.ResourceId | ? { $_.Tags.Keys -match $BackupContainerTag } )) {
$containerName = $keyvault.VaultName.Replace("-", $null).ToLower()
Update-AzTag $keyvault.ResourceId -operation Merge -Tag #{BackupContainer = $containerName }
}
$containerName = (get-azkeyvault -VaultName $keyvault.vaultname | ? { $_.Tags.Keys -match $backupContainerTag }).tags.Get_Item($backupContainerTag)
if ($null -eq (Get-AzStorageContainer -Name $containerName -Context $storageAccount.context)) {
New-AzStorageContainer -Name $containerName -Context $storageAccount.context
}
backup-keyVaultItems -keyvaultName $keyvault.VaultName
foreach ($file in (get-childitem "$($backupFolder)\$($keyvault.VaultName)")) {
Set-AzStorageBlobContent -File $file.FullName -Container $containerName -Blob $file.name -Context $storageAccount.context -Force
}
}
}
}

How to create folder structure inside Azure Storage Container using PowerShell

I have created the storage account along container in Azure using ARM templates. But I want to create folder structure inside the container using PowerShell script.
For example:
Folder1>SubFolder1>
Folder2>SUbFolder2>
…… etc
So, can anyone pls suggest me on this.
There are no directories inside Azure Storage Container. You can create a single container, and then, blobs inside of it.
Alternatively, you can use to include '/' in the blob name.
Eg:
account/container/2020/09/24/sample.txt
where "2020/09/24/sample.txt" is the blobname
You can't create the directories inside the Container
All blobs must reside in a blob container, which is simply a logical grouping of blobs. An account can contain an unlimited number of containers, and each container can store an unlimited number of blobs. You can include the / in the container name ("folder/1.txt"). You can create a folder structure
This SO thread may help you in your scenario
Creating an Azure Blob Hierarchy
You can create the folder structure using the below PowerShell Script.
#connecting to Storage account
$storageAccount = Get-AzStorageAccount -ResourceGroupName "<ResourceGroupName>"
-AccountName "<storage account>"
$ctx = $storageAccount.Context
# Passing Container name
$filesystemName = "<Container>"
# directory to be created
$folders1= #('folder1/folder2/folder3','folder4/folder5/folder6')
$FolderArray = ""
for ($i = 0; $i -le ($folders1.length - 1); $i += 1){
$dirname =""
$path =""
$filter=""
$FolderArray =$folders1[$i].Split("/")
for ($j = 0; $j -le ($FolderArray.length - 1); $j += 1){
$dirname = $dirname+$FolderArray[$j]+"/"
#print Directory name
$dirname
$path = $path + $(if ($j -eq 0) {"/"} else { "" })
$filter = $filter + $(if ($j -eq 0) {""} else { "/" })+ $FolderArray[$j]
# Check the directory is exist or not
$present = Get-AzDataLakeGen2ChildItem -Context
$ctx -FileSystem $filesystemName -Path $path |
Where-Object {$_.Name -eq "$($filter)"} -ErrorAction SilentlyContinue
# Create directory
if (! $present)
{
New-AzDataLakeGen2Item -Context $ctx -FileSystem
$filesystemName -Path $dirname -Directory
Write-Host "The directory $dirname is created"
}
# Show the existing folder
else
{
Write-Host "Folder named $dirname already exists"
Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem
$filesystemName -Path $path
}
$path = $(if ($path -eq "/"){$FolderArray[$j]} else {""}) +
$(if ($path -ne "/"){$path +"/"} else {""}) +
$(if ($path -ne "/"){$FolderArray[$j]} else {""})
}
}

Escaping characters on AzureBlobContent

I have got a problem to set my content in AzureBlobStorage.
In local, I have succeeded to replace characters for each files in a directory.
$sourceFolder = "C:\MyDirectory"
$targetFolder = "C:\MyDirectoryEncodeded"
$fileList = Dir $sourceFolder -Filter *.dat
MkDir $targetFolder -ErrorAction Ignore
ForEach($file in $fileList) {
$file | Get-Content | %{$_ -replace '"',''} | %{$_ -replace ',','.'} | Set-Content -Path "tempDirectory\$file"
$newFile = Get-Content "tempDirectory\$file"
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines("targetDirectory\$file" , $newFile,$Utf8NoBomEncoding)
}
exit
But now, I need to do the same in Microsoft Azure.
I get the content into an Azure Blob Storage, I escape characters, I encoding my file in UTF-8NoBom and then I set the encode file into a new Blob Directory.
Nevertheless, I faced an issue when I want to set the new content with escape characters (First line in my loop).
$storageContext = New-AzureStorageContext -ConnectionString "DefaultEndpointsProtocol=https;AccountName=<myAccountName>;AccountKey=<myAccountKey>;"
$sourceFolder = Get-AzureStorageBlob -Container "datablobnotencoded" -Blob "*.dat" -Context $storageContext
$targetFolder = Get-AzureStorageBlob -Container "datablob" -Context $storageContext
MkDir $targetFolder -ErrorAction Ignore
ForEach($file in $sourceFolder) {
Get-AzureStorageBlob -Container "datablobnotencoded" -Blob $file.Name -Context $storageContext | Get-AzureStorageBlobContent | %{$_ -replace '"',''} | %{$_ -replace ',','.'} | Set-AzureStorageBlobContent -File $file.Name -Context $storageContext -CloudBlob $file
$newFile = Get-AzureStorageFileContent -Path $file
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines($file , $newFile, $Utf8NoBomEncoding)
}
I've got this error:
Set-AzureStorageBlobContent : Cannot bind parameter 'CloudBlob'.
Cannot convert the
"Microsoft.WindowsAzure.Commands.Storage.Model.ResourceModel.AzureStorageBlob"
value of type
"Microsoft.WindowsAzure.Commands.Storage.Model.ResourceModel.AzureStorageBlob"
to type "Microsoft.WindowsAzure.Storage.Blob.CloudBlob". At line:7
char:264
+ ... lobContent -File $file.Name -Context $storageContext -CloudBlob $file
+ ~~~~~
+ CategoryInfo : InvalidArgument: (:) [Set-AzureStorageBlobContent], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand
Thank you for your answers!
There are some mistakes in your powershell scripts:
1.You may misunderstand the usage of Get-AzureStorageBlobContent, it's used to download blob to local, you cann't get the content of the blob, more details refer here.
2.In the loop, you used $newFile = Get-AzureStorageFileContent -Path $file, the Get-AzureStorageFileContent cmdlet is for file share storage, not for the blob storage.
You can use Get-AzureStorageBlobContent to download the blobs to a local folder, then operate on the local file which is downloaded from blob storage. After the file is modified, you can use Set-AzureStorageBlobContent to upload the local files to the specified azure blob storage.
Sample code as below, and works fine at my side:
$context = New-AzureStorageContext -ConnectionString "xxxx"
#download the blobs in specified contianers
$sourceFolder_blob = Get-AzureStorageBlob -Container "test-1" -Blob "*.txt" -Context $context
#the target azure container, which you want to upload the modifed blob to
$taget_container="test-2"
#the local path which is used to store the download blobs, and make sure the folders exist before use.
$sourceFolder_local="d:\test\blob1\"
$targetFolder_local="d:\test\blob2\"
foreach($file in $sourceFolder_blob)
{
#download the specified blob to local path
Get-AzureStorageBlobContent -Container "test-1" -Blob $file.name -Destination $sourceFolder_local -Context $context
#get the local file path
$local_file_path=$sourceFolder_local + $file.name
#set content to the file in target local folder
$local_target_file_path = "$targetFolder_local"+$file.name
#since the files are downloaded to local, you can any operation for the local file
Get-Content $local_file_path | %{$_ -replace '-','!'} | %{$_ -replace ',','.'} | Set-Content -Path $local_target_file_path
$newFile = Get-Content -Path $local_target_file_path
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines($local_target_file_path , $newFile,$Utf8NoBomEncoding)
#the last step, upload the modified file to another azure container
Set-AzureStorageBlobContent -File $local_target_file_path -Context $context -Container $taget_container
}

Unable to Download Blob Programmatically

We have multiple blobs in an azure storage container, when we use powershell to download the blobs (files), 8 out of the 9 files download, however the 9th one fails. There is absolutely nothing different about this file, the only thing I've noticed in the blob properties the "content MD5" is blank, however the other 8 have a value. Not sure what this is or if it has anything to do wit it, I was hoping someone could shed some light as to why this is one file is not downloading..
Thanks in advance :)
Try below code for downloading files from Azure Blob
function Get-DLLFile
{
param(
[Parameter(Mandatory=$true)] [string] $connectionString,
[Parameter(Mandatory=$true)] [String[]] $blobsName,
[Parameter(Mandatory=$true)] [string] $container,
[Parameter(Mandatory=$true)] [string] $filePath
)
Try
{
foreach ($blobName in $blobsName)
{
$file = $filePath + $blobName
$fileAvailable = Get-Item -Path $file -ErrorAction SilentlyContinue
if($null -eq $fileAvailable)
{
$ctx = New-AzureStorageContext -ConnectionString $connectionString
New-Item -Path $filePath -ItemType Directory -Force | Out-Null
Get-AzureStorageBlobContent -Blob $blobName -Container $container -Destination $filePath -Context $ctx -Force | out-null
}
}
}
Catch
{
$_.Exception.Message
}
}
Get-DLLFile -blobsName "File1.csv","File2.json" -container "myContainer" -connectionString "$(BlobConnectionString)" -filePath "$(System.DefaultWorkingDirectory)/Download"
Hope this should work, if not Please share the exception you are getting.

Resources