Powershell script to download file with current date as filename from Azure blob - Download log file and Remove the Downloaded content from blob - azure

I have PowerShell script which downloads the file with current date as filename from Azure blob. But How to get the log file of the process and how to remove the file which is downloaded from Azure blob through script. Could someone help me in this, Please.
Example.
app_09102021.txt
app_10102021.txt
app_11102021.txt
Below is the script.
$container_name = '$XXX'
$destination_path = 'D:\test'
$Ctx = New-AzStorageContext $ZZZZ -StorageAccountKey $CCVCVCVCV
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$Listblobs = 'app_{0:ddMMyyyy}.txt' -f (Get-Date)
# Just download that blob
Get-AzStorageBlobContent -Context $Ctx -Container $container_name -Blob $Listblobs -Destination $destination_path

I have tested in my environment to download the blobs and delete by using below cmd and its successfully downloaded and got removed from Azure as well .
$container_name = 'testaj'
$destination_path = 'C:\Users\Desktop\test'
$Ctx = New-AzStorageContext 'accountname' -StorageAccountKey 'accountkeyrA=='
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$Listblobs ='app_{0:ddMMyyyy}.txt' -f (Get-Date)
$blob = Get-AzStorageBlobContent -Context $Ctx -Container $container_name -Blob $Listblobs -Destination $destination_path -ErrorAction SilentlyContinue
if($blob -ne $Null)
{
Write-Host ("The File $Listblobs has been downloaded to $destination_path")
Write-Host ("Proceeding to delete the downloaded file from $container_name in Azure!!!")
Remove-AzStorageBlob -Container $container_name -Blob $Listblobs -Context $Ctx
Write-Host ("deleted file from $Listblobs in Azure!!!")
}
else{
write-Host ("The file does not exit")
}
Here is the Output for downloaded and deleted file from blob:
If the file got deleted then it will be something like below:
For more information please refer this MS DOC: Monitoring Azure Blob Storage
UPDATE:
I am looking for the solution to search & match the date with today's
date in filename, not sorting by last modified date/time.
Tried with the below code to download all the blobs with todays date for e.g abc_04012022,app_04012022 and delete them from Azure
PS Script :-
$container_name = 'test'
$destination_path = 'C:\Users\Desktop\test'
$Ctx = New-AzStorageContext 'accountname' -StorageAccountKey 'accountkey=='
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$latestBlob = Get-Date -UFormat '%d%m%Y'
$bloblist = Get-AzStorageBlob -Container $container_name -Context $Ctx |select -Property Name
foreach($item in $bloblist){
if($item.Name -match $latestBlob){
Write-Output "Here is the blobs"
$blob = Get-AzStorageBlobContent -Context $Ctx -Container $container_name -Blob $item.Name -Destination $destination_path -ErrorAction SilentlyContinue
if($blob -ne $Null)
{
Write-Output ("The File $item.Name has been downloaded to $destination_path")
Write-Output ("Proceeding to delete the downloaded file from $container_name in Azure!!!")
Remove-AzStorageBlob -Container $container_name -Blob $item.Name -Context $Ctx
Write-Output ("deleted file from $item.Name in Azure!!!")
}
else{
write-Output ("The file does not exit")
}
}
}
Screenshot for reference:

Related

Azure PowerShell script to compare files in the blob with local destination and download files which is not available in the destination

Could you please help me with Azure PowerShell script to compare files in the blob with local destination and download files which is not available in the destination.
I tried some, but able to get the answer.
$blobNames = Get-Content
For ($i=0; $i -lt $blobNames.Length; $i++) {
$blob = $blobNames[$i]
Write-Host "Downloading $blob. Please wait."
Get-AzStorageBlobContent -Blob $blob -Container $containerName -Destination $destination -Context $context -Verbos
PowerShell script to compare files in the blob with local destination and download files which is not available in the destination:
I've created a script and it works for me:
$ContainerName ='<containername>'
$destination_path = 'C:\Users\xxxx\Desktop\blobcheck' #pathtobedownloaded
$Ctx = New-AzureStorageContext '<storageaccount>' -StorageAccountKey 'accesskey'
$Blobs = Get-AzureStorageBlob -Container $ContainerName -Context $Ctx
$localfile = get-childitem -LiteralPath C:\Users\xxxxxx
For ($i=0; $i -lt $Blobs.Length; $i++) {
if($Blobs[$i].Name -eq $localfile){
Write-Host "Presented"
}
else{
$blob = $Blobs[$i].Name
Write-Host "Downloading $blob. Please wait."
Get-AzureStorageBlobContent -Blob $blob -Container $ContainerName -Destination $destination_path -Context $Ctx -Verbose
}
}
Compared and downloaded the files which is not present in my local folder:
Files uploaded in My Local folder (destination_path):
Prompting to rewrite if the file is already exists as shown below:
If the prompt is not needed, then you can simply use
DisplayAlerts = FALSE
Note: Get storage access key from Azure Portal:
Goto <Storageaccount> -> Access keys

Powershell script to delete file from subfolder from blob container

I am trying to delete file from specific folder like from full or diff in blob container but unable to do.
Container_name and then there are two folders full and diff and I want to delete file from full only.
Please help.
$context = New-AzureStorageContext -StorageAccountName "storage_name" -StorageAccountKey "key"
$blobs= Get-AzureStorageBlob -Container "container_name" -blob *DIFF*.bak -Context $context
foreach ($blob in $blobs)
{
$modifieddate = $blob.LastModified
Write-Host $modifieddate
if ($modifieddate -ne $null)
{
$howold = ([DateTime]::Now - [DateTime]$modifieddate.LocalDateTime)
if ($howold.TotalDays -ge 5)
{
Remove-AzureStorageBlob -Blob $blob.Name -Container "container_name" -Context $context
Write-Host $blob.Name
}
}
}
If you want to use Azure PowerShell to delete blobs from one subfolder in one container, you can use the following script :
$StorageAccountKey=" "
$StorageAccountName=" "
$ContainerName=" "
$Token = $null
$Total = 0
$MaxCount=5000
$context = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
do
{
$Blobs = Get-AzStorageBlob -Container $ContainerName -MaxCount $MaxCount -ContinuationToken $Token -Context $context -Prefix "your fodler name"
if($Blobs.Length -le 0) { Break;}
$Token = $Blobs[$blobs.Count -1].ContinuationToken;
foreach($blob in $blobs){
Remove-AzStorageBlob -Blob $blob.Name -Container $ContainerName -Context $context
}
}
While ($Token -ne $null)
Update
I use the latest version Azure Az module to do a test.

How to copy files from one container to another containers fits equally in all dest containers according to size using powershell

I have one container in blob of storage account in azure contains different folder having files of different sizes.
In other side, in same storage account, I have 10 different containers.
I have to copy these files from source container to destination 10 containers but the condition is the files should be equally distributed to all the containers.
I have tried below so far
$key = "abcdxyz"
# declaring the azure context
$context = New-AzStorageContext -StorageAccountName abcd -StorageAccountKey $key
#Getting the data from the blob
$bacdata = Get-AzStorageContainer -Name sourcecontainer* -Context $context | Get-AzStorageBlob
$15=$bacdata | where{$_.Name -like "sourcecontainer1*"} | where{$_.LastModified -gt (get-date).adddays(-1)}
Here is the powershell script to do it:
#Server side storage copy
$SourceStorageAccount = "sourceAccountName"
$SourceStorageKey = "sourceAccountAPIKey"
$DestStorageAccount = "destinationAccountName"
$DestStorageKey = "destinationAccountAPIKey"
$SourceStorageContext = New-AzureStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey
$DestStorageContext = New-AzureStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey
$Containers = Get-AzureStorageContainer -Context $SourceStorageContext
foreach($Container in $Containers)
{
$ContainerName = $Container.Name
if (!((Get-AzureStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName }))
{
Write-Output "Creating new container $ContainerName"
New-AzureStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
}
$Blobs = Get-AzureStorageBlob -Context $SourceStorageContext -Container $ContainerName
$BlobCpyAry = #() #Create array of objects
#Do the copy of everything
foreach ($Blob in $Blobs)
{
$BlobName = $Blob.Name
Write-Output "Copying $BlobName from $ContainerName"
$BlobCopy = Start-CopyAzureStorageBlob -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName
$BlobCpyAry += $BlobCopy
}
#Check Status
foreach ($BlobCopy in $BlobCpyAry)
{
#Could ignore all rest and just run $BlobCopy | Get-AzureStorageBlobCopyState but I prefer output with % copied
$CopyState = $BlobCopy | Get-AzureStorageBlobCopyState
$Message = $CopyState.Source.AbsolutePath + " " + $CopyState.Status + " {0:N2}%" -f (($CopyState.BytesCopied/$CopyState.TotalBytes)*100)
Write-Output $Message
}
}
Overall concept would be similar , just change the function aas per the Az CLI.
here is the copy command in Az CLI
az storage blob copy start
You can find more details here.
https://learn.microsoft.com/en-us/cli/azure/storage/blob/copy?view=azure-cli-latest#az-storage-blob-copy-start
Hope it helps.
how to copy files from one folder in container to another folder?
method 1-azuresynapse>integrate>copydata tool>Built-in copy task>new connection>create connection>dataset>browse file>uncheck recursively>next >next>next
method 2-HOme>resource groups>data factory>review create>open azure data factory>ingest>select source and file>select destination >finish>check in container if file is transfered or not

Listing blobs in Loop not being Shown

I've created a script which deletes blobs which are older than a set date, I'm trying to run this using an automation account, however when I test it using the "test pane" it gives the desired output, which is a list of blobs to be deleted, however when it actually runs using the automation account it doesn't display a list of blobs to be deleted.
The code is below:
### delete blobs older than 30 days
param(
[parameter(mandatory=$true)]
[int32]$daysToKeep,
[parameter(mandatory=$true)]
[string]$storageAccount,
[parameter(mandatory=$true)]
[string]$storageContainer,
[parameter(mandatory=$true)]
[string]$storageAccessKey
)
$connectionName = "AzureRunAsConnection"
# Get the connection "AzureRunAsConnection "
$servicePrincipalConnection = Get-AutomationConnection -Name $connectionName
"Logging in to Azure..."
Add-AzureRmAccount `
-ServicePrincipal `
-TenantId $servicePrincipalConnection.TenantId `
-ApplicationId $servicePrincipalConnection.ApplicationId `
-CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint
Write-Host "logged into Azure"
$context = New-AzureStorageContext -StorageAccountName $storageAccount -StorageAccountKey $storageAccessKey
New-AzureStorageContainer -Name $storageContainer -Context $context -Permission Blob -ErrorAction SilentlyContinue
$EGBlobs = Get-AzureStorageBlob -Container $storageContainer -Context $context | sort-object LastModified | select lastmodified, name
foreach($blob in $EGBlobs)
{
if($blob.lastmodified -lt (get-date).AddDays($daysToKeep*-1))
{
$blob_date = [datetime]$blob.LastModified.UtcDateTime
Write-Output "-----------------------------------"
write-output "Purging blob from Storage: " $blob.name
write-output "----------------------------------- "
write-output "Last Modified Date of the Blob: " $blob_date
Write-Output "-----------------------------------"
Remove-AzureStorageBlob -Blob $blob.name -Container $storageContainer -Context $context
}
}
Can't see where I'm going wrong, is this a setting within Azure Automation account.
Thanks in advance
Not sure why it is, but in fact it is.
Just move the line Remove-AzureStorageBlob -Blob $blob.name -Container $storageContainer -Context $context to the top in the loop, it will work fine.
It should be:
foreach($blob in $EGBlobs)
{
if($blob.lastmodified -lt (get-date).AddDays(2))
{
Remove-AzureStorageBlob -Blob $blob.name -Container $storageContainer -Context $context
$blob_date = [datetime]$blob.LastModified.UtcDateTime
Write-Output "-----------------------------------"
write-output "Purging blob from Storage: " $blob.name
write-output "----------------------------------- "
write-output "Last Modified Date of the Blob: " $blob_date
Write-Output "-----------------------------------"
}
}

Azure PowerShell Download blob contents from container

I'm trying to download sme blob files from an azure storage account.
There are a mix of other containers and blockblobs in the parent container, I need to download just the blockblobs and not the other containers, I can't find a way of seperating them out, also I need to download some blobs from a conatiner within a container.
My code will download all the contents in the parent blob, including all sub containers.
$sub = "MySub"
$staccname = "straccname1234"
$key = "sdcsecurekeythinghere"
$ctx = New-AzureStorageContext -StorageAccountName $staccname `
-StorageAccountKey $key
$cont = "data\download\files.001" ##the container includes other cntainers and subcontainers
$dest = "C:\blb-Downloads"
Select-AzureSubscription -SubscriptionName $sub –Default
Set-AzureSubscription -Currentstaccname $staccname -SubscriptionName $sub
Get-AzureStorageBlob -Container $cont -Context $ctx
$blobs = Get-AzureStorageBlob -Container $cont -Context $ctx
$blobs | Get-AzureStorageBlobContent –Destination $dest -Context $ctx
There are approx 75 files in the parent blob and 123 files in data\downloads.
Using the newer Azure PowerShell Az module, you can use Get-AzStorageBlob to list all the block blobs from the container, then use Get-AzStorageBlobContent to download the blobs.
As already shown by #George Wallace, we can use Where-Object or its alias ? to filter block blob types.
Demo:
$resourceGroup = "myResourceGroup"
$storageAccount = "myStorageAccount"
$container = "myContainerName"
$destination = "./blobs"
# Create destination directory if it doesn't exist
if (-not (Test-Path -Path $destination -PathType Container)) {
New-Item -Path $destination -ItemType Directory
}
# Get storage account with container we want to download blobs from
$storageAccount = Get-AzStorageAccount -Name $storageAccount -ResourceGroupName $resourceGroup
# Get all BlockBlobs from container
$blockBlobs = Get-AzStorageBlob -Container $container -Context $storageAccount.Context
| Where-Object {$_.BlobType -eq "BlockBlob"}
# Download each blob from container into destination directory
$blockBlobs | Get-AzStorageBlobContent -Destination $destination -Force
Can you not just run the following and limit it to BlockBlobs only?
Get-AzureStorageBlob -Container $cont -Context $ctx | ? {$_.BlobType -eq "BlockBlob"}
This works for me
$storageaccountname = "jawadtestsaacc"
$sastoken = ""
$ContainerName = "access"
$Ctx = New-AzStorageContext -StorageAccountName $storageaccountname -SasToken $sastoken
Get-AzStorageBlob -Container "$ContainerName" -Blob "Az.json" -Context $Ctx | Get-AzStorageBlobContent
or
Get-AzStorageBlobContent -Container "$ContainerName" -Blob "Az.json" -Context $Ctx

Resources