Could you please help me with Azure PowerShell script to compare files in the blob with local destination and download files which is not available in the destination.
I tried some, but able to get the answer.
$blobNames = Get-Content
For ($i=0; $i -lt $blobNames.Length; $i++) {
$blob = $blobNames[$i]
Write-Host "Downloading $blob. Please wait."
Get-AzStorageBlobContent -Blob $blob -Container $containerName -Destination $destination -Context $context -Verbos
PowerShell script to compare files in the blob with local destination and download files which is not available in the destination:
I've created a script and it works for me:
$ContainerName ='<containername>'
$destination_path = 'C:\Users\xxxx\Desktop\blobcheck' #pathtobedownloaded
$Ctx = New-AzureStorageContext '<storageaccount>' -StorageAccountKey 'accesskey'
$Blobs = Get-AzureStorageBlob -Container $ContainerName -Context $Ctx
$localfile = get-childitem -LiteralPath C:\Users\xxxxxx
For ($i=0; $i -lt $Blobs.Length; $i++) {
if($Blobs[$i].Name -eq $localfile){
Write-Host "Presented"
}
else{
$blob = $Blobs[$i].Name
Write-Host "Downloading $blob. Please wait."
Get-AzureStorageBlobContent -Blob $blob -Container $ContainerName -Destination $destination_path -Context $Ctx -Verbose
}
}
Compared and downloaded the files which is not present in my local folder:
Files uploaded in My Local folder (destination_path):
Prompting to rewrite if the file is already exists as shown below:
If the prompt is not needed, then you can simply use
DisplayAlerts = FALSE
Note: Get storage access key from Azure Portal:
Goto <Storageaccount> -> Access keys
Related
I have PowerShell script which downloads the file with current date as filename from Azure blob. But How to get the log file of the process and how to remove the file which is downloaded from Azure blob through script. Could someone help me in this, Please.
Example.
app_09102021.txt
app_10102021.txt
app_11102021.txt
Below is the script.
$container_name = '$XXX'
$destination_path = 'D:\test'
$Ctx = New-AzStorageContext $ZZZZ -StorageAccountKey $CCVCVCVCV
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$Listblobs = 'app_{0:ddMMyyyy}.txt' -f (Get-Date)
# Just download that blob
Get-AzStorageBlobContent -Context $Ctx -Container $container_name -Blob $Listblobs -Destination $destination_path
I have tested in my environment to download the blobs and delete by using below cmd and its successfully downloaded and got removed from Azure as well .
$container_name = 'testaj'
$destination_path = 'C:\Users\Desktop\test'
$Ctx = New-AzStorageContext 'accountname' -StorageAccountKey 'accountkeyrA=='
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$Listblobs ='app_{0:ddMMyyyy}.txt' -f (Get-Date)
$blob = Get-AzStorageBlobContent -Context $Ctx -Container $container_name -Blob $Listblobs -Destination $destination_path -ErrorAction SilentlyContinue
if($blob -ne $Null)
{
Write-Host ("The File $Listblobs has been downloaded to $destination_path")
Write-Host ("Proceeding to delete the downloaded file from $container_name in Azure!!!")
Remove-AzStorageBlob -Container $container_name -Blob $Listblobs -Context $Ctx
Write-Host ("deleted file from $Listblobs in Azure!!!")
}
else{
write-Host ("The file does not exit")
}
Here is the Output for downloaded and deleted file from blob:
If the file got deleted then it will be something like below:
For more information please refer this MS DOC: Monitoring Azure Blob Storage
UPDATE:
I am looking for the solution to search & match the date with today's
date in filename, not sorting by last modified date/time.
Tried with the below code to download all the blobs with todays date for e.g abc_04012022,app_04012022 and delete them from Azure
PS Script :-
$container_name = 'test'
$destination_path = 'C:\Users\Desktop\test'
$Ctx = New-AzStorageContext 'accountname' -StorageAccountKey 'accountkey=='
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$latestBlob = Get-Date -UFormat '%d%m%Y'
$bloblist = Get-AzStorageBlob -Container $container_name -Context $Ctx |select -Property Name
foreach($item in $bloblist){
if($item.Name -match $latestBlob){
Write-Output "Here is the blobs"
$blob = Get-AzStorageBlobContent -Context $Ctx -Container $container_name -Blob $item.Name -Destination $destination_path -ErrorAction SilentlyContinue
if($blob -ne $Null)
{
Write-Output ("The File $item.Name has been downloaded to $destination_path")
Write-Output ("Proceeding to delete the downloaded file from $container_name in Azure!!!")
Remove-AzStorageBlob -Container $container_name -Blob $item.Name -Context $Ctx
Write-Output ("deleted file from $item.Name in Azure!!!")
}
else{
write-Output ("The file does not exit")
}
}
}
Screenshot for reference:
We have Azure Blob Storage Accounts with 100s of containers. The file structure is something like below:
container_01
|
--somemedia.jpg
--anothermedia.jpg
container_02
|
--secondcontainersmedia.jpg
--andSoOn
--AndSoOnAndSoOn
My client wants to download all of the containers to local storage so that if necessary they can be re-uploaded to Azure. After doing some research I found this blog post. Updating the script from there to suit my needs (just updating from AzureRM to AZ and my personal connection and local path), I came up with the following suitable script for downloading the files.
$destination_path = 'C:\Storage Dump Test'
$connection_string = '[Insert Connection String]'
$storage_account = New-AzStorageContext -ConnectionString $connection_string
$containers = Get-AzStorageContainer -Context $storage_account
Write-Host 'Starting Storage Dump...'
foreach ($container in $containers)
{
Write-Host -NoNewline 'Processing: ' . $container.Name . '...'
$container_path = $destination_path + '\' + $container.Name
if(!(Test-Path -Path $container_path ))
{
New-Item -ItemType directory -Path $container_path
}
$blobs = Get-AzStorageBlob -Container $container.Name -Context $storage_account
Write-Host -NoNewline ' Downloading files...'
foreach ($blob in $blobs)
{
$fileNameCheck = $container_path + '\' + $blob.Name
if(!(Test-Path $fileNameCheck ))
{
Get-AzStorageBlobContent `
-Container $container.Name -Blob $blob.Name -Destination $container_path `
-Context $storage_account
}
}
Write-Host ' Done.'
}
Write-Host 'Download complete.'
So now I have a directory on my local storage with hundreds of folders containing media items. I need to create a PS script (or find some other way) to basically do the opposite-- take all the folders in that directory, create containers using the names of the folders, and upload the items within each folder to the appropriate container.
How should I start going about this?
You'd have a lot more success, quicker, using azcopy instead of working with the azure cmdlets. To copy:
azcopy copy '<local-file-path>' 'https://<storage-account-name>.<blob| dfs>.core.windows.net/<container-name>/<blob-name>'
It can also create containers:
azcopy make 'https://mystorageaccount.blob.core.windows.net/mycontainer'
azcopy can download an entire container without you having to specify each file. Use --recursive
See: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10
I am trying to delete file from specific folder like from full or diff in blob container but unable to do.
Container_name and then there are two folders full and diff and I want to delete file from full only.
Please help.
$context = New-AzureStorageContext -StorageAccountName "storage_name" -StorageAccountKey "key"
$blobs= Get-AzureStorageBlob -Container "container_name" -blob *DIFF*.bak -Context $context
foreach ($blob in $blobs)
{
$modifieddate = $blob.LastModified
Write-Host $modifieddate
if ($modifieddate -ne $null)
{
$howold = ([DateTime]::Now - [DateTime]$modifieddate.LocalDateTime)
if ($howold.TotalDays -ge 5)
{
Remove-AzureStorageBlob -Blob $blob.Name -Container "container_name" -Context $context
Write-Host $blob.Name
}
}
}
If you want to use Azure PowerShell to delete blobs from one subfolder in one container, you can use the following script :
$StorageAccountKey=" "
$StorageAccountName=" "
$ContainerName=" "
$Token = $null
$Total = 0
$MaxCount=5000
$context = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
do
{
$Blobs = Get-AzStorageBlob -Container $ContainerName -MaxCount $MaxCount -ContinuationToken $Token -Context $context -Prefix "your fodler name"
if($Blobs.Length -le 0) { Break;}
$Token = $Blobs[$blobs.Count -1].ContinuationToken;
foreach($blob in $blobs){
Remove-AzStorageBlob -Blob $blob.Name -Container $ContainerName -Context $context
}
}
While ($Token -ne $null)
Update
I use the latest version Azure Az module to do a test.
I have got a problem to set my content in AzureBlobStorage.
In local, I have succeeded to replace characters for each files in a directory.
$sourceFolder = "C:\MyDirectory"
$targetFolder = "C:\MyDirectoryEncodeded"
$fileList = Dir $sourceFolder -Filter *.dat
MkDir $targetFolder -ErrorAction Ignore
ForEach($file in $fileList) {
$file | Get-Content | %{$_ -replace '"',''} | %{$_ -replace ',','.'} | Set-Content -Path "tempDirectory\$file"
$newFile = Get-Content "tempDirectory\$file"
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines("targetDirectory\$file" , $newFile,$Utf8NoBomEncoding)
}
exit
But now, I need to do the same in Microsoft Azure.
I get the content into an Azure Blob Storage, I escape characters, I encoding my file in UTF-8NoBom and then I set the encode file into a new Blob Directory.
Nevertheless, I faced an issue when I want to set the new content with escape characters (First line in my loop).
$storageContext = New-AzureStorageContext -ConnectionString "DefaultEndpointsProtocol=https;AccountName=<myAccountName>;AccountKey=<myAccountKey>;"
$sourceFolder = Get-AzureStorageBlob -Container "datablobnotencoded" -Blob "*.dat" -Context $storageContext
$targetFolder = Get-AzureStorageBlob -Container "datablob" -Context $storageContext
MkDir $targetFolder -ErrorAction Ignore
ForEach($file in $sourceFolder) {
Get-AzureStorageBlob -Container "datablobnotencoded" -Blob $file.Name -Context $storageContext | Get-AzureStorageBlobContent | %{$_ -replace '"',''} | %{$_ -replace ',','.'} | Set-AzureStorageBlobContent -File $file.Name -Context $storageContext -CloudBlob $file
$newFile = Get-AzureStorageFileContent -Path $file
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines($file , $newFile, $Utf8NoBomEncoding)
}
I've got this error:
Set-AzureStorageBlobContent : Cannot bind parameter 'CloudBlob'.
Cannot convert the
"Microsoft.WindowsAzure.Commands.Storage.Model.ResourceModel.AzureStorageBlob"
value of type
"Microsoft.WindowsAzure.Commands.Storage.Model.ResourceModel.AzureStorageBlob"
to type "Microsoft.WindowsAzure.Storage.Blob.CloudBlob". At line:7
char:264
+ ... lobContent -File $file.Name -Context $storageContext -CloudBlob $file
+ ~~~~~
+ CategoryInfo : InvalidArgument: (:) [Set-AzureStorageBlobContent], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand
Thank you for your answers!
There are some mistakes in your powershell scripts:
1.You may misunderstand the usage of Get-AzureStorageBlobContent, it's used to download blob to local, you cann't get the content of the blob, more details refer here.
2.In the loop, you used $newFile = Get-AzureStorageFileContent -Path $file, the Get-AzureStorageFileContent cmdlet is for file share storage, not for the blob storage.
You can use Get-AzureStorageBlobContent to download the blobs to a local folder, then operate on the local file which is downloaded from blob storage. After the file is modified, you can use Set-AzureStorageBlobContent to upload the local files to the specified azure blob storage.
Sample code as below, and works fine at my side:
$context = New-AzureStorageContext -ConnectionString "xxxx"
#download the blobs in specified contianers
$sourceFolder_blob = Get-AzureStorageBlob -Container "test-1" -Blob "*.txt" -Context $context
#the target azure container, which you want to upload the modifed blob to
$taget_container="test-2"
#the local path which is used to store the download blobs, and make sure the folders exist before use.
$sourceFolder_local="d:\test\blob1\"
$targetFolder_local="d:\test\blob2\"
foreach($file in $sourceFolder_blob)
{
#download the specified blob to local path
Get-AzureStorageBlobContent -Container "test-1" -Blob $file.name -Destination $sourceFolder_local -Context $context
#get the local file path
$local_file_path=$sourceFolder_local + $file.name
#set content to the file in target local folder
$local_target_file_path = "$targetFolder_local"+$file.name
#since the files are downloaded to local, you can any operation for the local file
Get-Content $local_file_path | %{$_ -replace '-','!'} | %{$_ -replace ',','.'} | Set-Content -Path $local_target_file_path
$newFile = Get-Content -Path $local_target_file_path
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines($local_target_file_path , $newFile,$Utf8NoBomEncoding)
#the last step, upload the modified file to another azure container
Set-AzureStorageBlobContent -File $local_target_file_path -Context $context -Container $taget_container
}
I'm trying to download sme blob files from an azure storage account.
There are a mix of other containers and blockblobs in the parent container, I need to download just the blockblobs and not the other containers, I can't find a way of seperating them out, also I need to download some blobs from a conatiner within a container.
My code will download all the contents in the parent blob, including all sub containers.
$sub = "MySub"
$staccname = "straccname1234"
$key = "sdcsecurekeythinghere"
$ctx = New-AzureStorageContext -StorageAccountName $staccname `
-StorageAccountKey $key
$cont = "data\download\files.001" ##the container includes other cntainers and subcontainers
$dest = "C:\blb-Downloads"
Select-AzureSubscription -SubscriptionName $sub –Default
Set-AzureSubscription -Currentstaccname $staccname -SubscriptionName $sub
Get-AzureStorageBlob -Container $cont -Context $ctx
$blobs = Get-AzureStorageBlob -Container $cont -Context $ctx
$blobs | Get-AzureStorageBlobContent –Destination $dest -Context $ctx
There are approx 75 files in the parent blob and 123 files in data\downloads.
Using the newer Azure PowerShell Az module, you can use Get-AzStorageBlob to list all the block blobs from the container, then use Get-AzStorageBlobContent to download the blobs.
As already shown by #George Wallace, we can use Where-Object or its alias ? to filter block blob types.
Demo:
$resourceGroup = "myResourceGroup"
$storageAccount = "myStorageAccount"
$container = "myContainerName"
$destination = "./blobs"
# Create destination directory if it doesn't exist
if (-not (Test-Path -Path $destination -PathType Container)) {
New-Item -Path $destination -ItemType Directory
}
# Get storage account with container we want to download blobs from
$storageAccount = Get-AzStorageAccount -Name $storageAccount -ResourceGroupName $resourceGroup
# Get all BlockBlobs from container
$blockBlobs = Get-AzStorageBlob -Container $container -Context $storageAccount.Context
| Where-Object {$_.BlobType -eq "BlockBlob"}
# Download each blob from container into destination directory
$blockBlobs | Get-AzStorageBlobContent -Destination $destination -Force
Can you not just run the following and limit it to BlockBlobs only?
Get-AzureStorageBlob -Container $cont -Context $ctx | ? {$_.BlobType -eq "BlockBlob"}
This works for me
$storageaccountname = "jawadtestsaacc"
$sastoken = ""
$ContainerName = "access"
$Ctx = New-AzStorageContext -StorageAccountName $storageaccountname -SasToken $sastoken
Get-AzStorageBlob -Container "$ContainerName" -Blob "Az.json" -Context $Ctx | Get-AzStorageBlobContent
or
Get-AzStorageBlobContent -Container "$ContainerName" -Blob "Az.json" -Context $Ctx