Powershell function for modifying the ContentType of an object in Azure blob storage - azure

I'm trying to write a function in Powershell that sets blobs in a specific container to a certain type, as they are always written with the type application/octet-stream which causes issues with downstream applications. I have written the below function but it returns the error 'ContentType' is a ReadOnly property.
I was wondering if there was any way around this? I know the property can be set manually in Azure Storage Explorer, however this is a daily task.
Function:
Function Set-ContentType {
Param (
[string]$accountName,
[string]$accessKey,
[string]$storageContainer
)
# Connect to blob storage and get blobs
$context = New-AzureStorageContext -StorageAccountName $accountName -StorageAccountKey $accessKey
$blobs = Get-AzureStorageBlob -Container $storageContainer -Context $context -Blob $fileMask
foreach ($blob in $blobs) {
if ($blob.ContentType -eq $genericMIME) {
$blob.ContentType = $targetMIME
}
}
}

So based on the link from the comments, please give this solution a try.
Function Set-ContentType {
Param (
[string]$accountName,
[string]$accessKey,
[string]$storageContainer
)
# Connect to blob storage and get blobs
$context = New-AzureStorageContext -StorageAccountName $accountName -StorageAccountKey $accessKey
$blobs = Get-AzureStorageBlob -Container $storageContainer -Context $context -Blob $fileMask
foreach ($blob in $blobs) {
if ($blob.ContentType -eq $genericMIME) {
$blob.Properties.ContentType = $targetMIME
$blob.SetProperties()
}
}
}

I have solved my own issue by writing an alternative upload script that defines the ContentType at the time of writing the blob:
Function UploadFile {
Param (
[string]$accountName,
[string]$accessKey
)
$context = New-AzureStorageContext -StorageAccountName $accountName -StorageAccountKey $accessKey
$files = Get-ChildItem $workingDir -Filter $fileMask
foreach ($file in $files) {
Set-AzureStorageBlobContent -File $file.FullName -Container $container -Properties #{"ContentType" = "$targetMIME"} -Context $context -Force
}
}

Related

Encrypt Azure Storage account key in powershell script

I'm developing a new powershell script in order to download any blobs from a specific container and the problem is due to security reasons because I do not want to paste in text plain the azure account key.
So I have implemented a solution using 'ConvertTo-SecureString' command but the problem still exists because when I create a connection string to the blob, there appears a message who said: "Server Failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. HTTP Status Code 403 - HTTP Error".
With the key in plain text I'm able to create the connection string properly and then list and download all blobs from the container.
I tried other solutions for example ' $Credential= New-Object System.Management.Automation.PSCredential ('$ShareUser, $SharePassword)'
but there is other problem related with the input is not valid base64 string.
Do you know how to avoid this issues and create a secure connection string with an Azure Storage Account?
Best regards and thanks in advance
Here a part of my powershell script
$SecurePassword= Read-Host -AsSecureString | ConvertFrom-SecureString
$SecurePassword | Out-File -FilePath C:\test_blob\pass_file.xml
$ConfigFile= 'C:\Users\\config_file.xml'
IF (Test-Path) {
[xml]$Config= Get-Content $ConfigFile
[string] $Server = $Config.Config.Server;
[string] $SharePassword = $Config.Config.SharePassword;
} ELSE
{
write-host "File do not exists: $ConfigFile"
}
#BlobStorageInformation
$StorageAccountName='test_acc'
$Container='test'
$DestinationFolder= 'C:\Users\user1\Blobs'
$Context = New-AzStorageConext -StorageAccountName $StorageAccountName -StorageAccountKey $SharePassword
#List of Blobs
$ListBlob=#()
$ListBlob+= Get-AzStorageBlob -context $Context -container $Container | Where-Object {$_.LastModified -lt (Get-Date).AddDAys(-1)}
Why would you maintain the password files or enter storage key manually when you have az powershell. Just login using az powershell, set the subscription and enjoy !
$ResourceGroupName = "YOURRESOURCEGROUPNAME"
$StorageAccountName = "YOURSTORAGEACCOUNTNAME"
$ContainerName = "YOURCONTAINERNAME"
$LocalPath = "D:\Temp"
Write-Output 'Downloading Content from Azure blob to local...'
$storageKey = (Get-AzStorageAccountKey -ResourceGroupName $ResourceGroupName -AccountName $StorageAccountName).value[0]
$storageContext = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $storageKey
$blobs = Get-AzStorageBlob -Container $ContainerName -Context $storageContext
foreach($blob in $blobs)
{
Get-AzStorageBlobContent -Container $ContainerName -Context $storageContext -Force -Destination $LocalPath -Blob $blob.Name
}
Write-Output 'Content Downloaded Successfully !!!'

Powershell script to delete file from subfolder from blob container

I am trying to delete file from specific folder like from full or diff in blob container but unable to do.
Container_name and then there are two folders full and diff and I want to delete file from full only.
Please help.
$context = New-AzureStorageContext -StorageAccountName "storage_name" -StorageAccountKey "key"
$blobs= Get-AzureStorageBlob -Container "container_name" -blob *DIFF*.bak -Context $context
foreach ($blob in $blobs)
{
$modifieddate = $blob.LastModified
Write-Host $modifieddate
if ($modifieddate -ne $null)
{
$howold = ([DateTime]::Now - [DateTime]$modifieddate.LocalDateTime)
if ($howold.TotalDays -ge 5)
{
Remove-AzureStorageBlob -Blob $blob.Name -Container "container_name" -Context $context
Write-Host $blob.Name
}
}
}
If you want to use Azure PowerShell to delete blobs from one subfolder in one container, you can use the following script :
$StorageAccountKey=" "
$StorageAccountName=" "
$ContainerName=" "
$Token = $null
$Total = 0
$MaxCount=5000
$context = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
do
{
$Blobs = Get-AzStorageBlob -Container $ContainerName -MaxCount $MaxCount -ContinuationToken $Token -Context $context -Prefix "your fodler name"
if($Blobs.Length -le 0) { Break;}
$Token = $Blobs[$blobs.Count -1].ContinuationToken;
foreach($blob in $blobs){
Remove-AzStorageBlob -Blob $blob.Name -Container $ContainerName -Context $context
}
}
While ($Token -ne $null)
Update
I use the latest version Azure Az module to do a test.

How to copy files from one container to another containers fits equally in all dest containers according to size using powershell

I have one container in blob of storage account in azure contains different folder having files of different sizes.
In other side, in same storage account, I have 10 different containers.
I have to copy these files from source container to destination 10 containers but the condition is the files should be equally distributed to all the containers.
I have tried below so far
$key = "abcdxyz"
# declaring the azure context
$context = New-AzStorageContext -StorageAccountName abcd -StorageAccountKey $key
#Getting the data from the blob
$bacdata = Get-AzStorageContainer -Name sourcecontainer* -Context $context | Get-AzStorageBlob
$15=$bacdata | where{$_.Name -like "sourcecontainer1*"} | where{$_.LastModified -gt (get-date).adddays(-1)}
Here is the powershell script to do it:
#Server side storage copy
$SourceStorageAccount = "sourceAccountName"
$SourceStorageKey = "sourceAccountAPIKey"
$DestStorageAccount = "destinationAccountName"
$DestStorageKey = "destinationAccountAPIKey"
$SourceStorageContext = New-AzureStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey
$DestStorageContext = New-AzureStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey
$Containers = Get-AzureStorageContainer -Context $SourceStorageContext
foreach($Container in $Containers)
{
$ContainerName = $Container.Name
if (!((Get-AzureStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName }))
{
Write-Output "Creating new container $ContainerName"
New-AzureStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
}
$Blobs = Get-AzureStorageBlob -Context $SourceStorageContext -Container $ContainerName
$BlobCpyAry = #() #Create array of objects
#Do the copy of everything
foreach ($Blob in $Blobs)
{
$BlobName = $Blob.Name
Write-Output "Copying $BlobName from $ContainerName"
$BlobCopy = Start-CopyAzureStorageBlob -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName
$BlobCpyAry += $BlobCopy
}
#Check Status
foreach ($BlobCopy in $BlobCpyAry)
{
#Could ignore all rest and just run $BlobCopy | Get-AzureStorageBlobCopyState but I prefer output with % copied
$CopyState = $BlobCopy | Get-AzureStorageBlobCopyState
$Message = $CopyState.Source.AbsolutePath + " " + $CopyState.Status + " {0:N2}%" -f (($CopyState.BytesCopied/$CopyState.TotalBytes)*100)
Write-Output $Message
}
}
Overall concept would be similar , just change the function aas per the Az CLI.
here is the copy command in Az CLI
az storage blob copy start
You can find more details here.
https://learn.microsoft.com/en-us/cli/azure/storage/blob/copy?view=azure-cli-latest#az-storage-blob-copy-start
Hope it helps.
how to copy files from one folder in container to another folder?
method 1-azuresynapse>integrate>copydata tool>Built-in copy task>new connection>create connection>dataset>browse file>uncheck recursively>next >next>next
method 2-HOme>resource groups>data factory>review create>open azure data factory>ingest>select source and file>select destination >finish>check in container if file is transfered or not

Foreach loop for listing unmanaged disks in Azure

It's very easy to list azure managed disks in PS, but unmanaged ones are very tricky to list, as they're not objects from azure POV. I tried to wrote foreach loop to list me all unamanged disks (i.e. *.vhd files) for each storage account. This is the code I wrote:
$StorageAccounts = Get-AzureRmStorageAccount
$sa = $StorageAccounts | foreach-object {
#Get the Management key for the storage account
$key1 = (Get-AzureRmStorageAccountKey -ResourceGroupName $_.ResourceGroupName -name $_.StorageAccountName)[0].value
#Get the Storage Context to access the Storage Container
$storageContext = New-AzureStorageContext -StorageAccountName $_.StorageAccountName -StorageAccountKey $key1
#Get the Storage Container in the Variable
$storageContainer = Get-AzureStorageContainer -Context $storageContext
$blob = Get-AzureStorageBlob -Container $storageContainer.name -Context $storageContext
[PSCustomObject]#{
"Name" = $blob.Name
"Length" = $blob.Length
"Storage Account Name" = $_.StorageAccountName
}
}
I want the loop to fetch all the vhd's for each storageaccount and parse it into pscustomobject to list me all vhd*s from all storage accounts, but I get an error:
Get-AzureStorageBlob : Cannot validate argument on parameter
'Container'. The argument is null or empty. Provide an argument that
is not null or empty, and then try the command again. At line:13
char:41
Get-AzureStorageBlob : Cannot convert 'System.Object[]' to the type
'System.String' required by parameter 'Container'. Specified method is not > supported.
At line:13 char:41
Why loop is not parsing data to $storageContainer in line 11? I can see what's inside other two vars like $key1 and $storageContext.
you can rewrite your script in this fashion:
$StorageAccounts = Get-AzureRmStorageAccount
$StorageAccounts.foreach{
$ctx = $_.Context
$containers = Get-AzureStorageContainer -Context $ctx
$containers.foreach{
$blobs = Get-AzureStorageBlob -Container $_.name -Context $ctx
$blobs.foreach{
do_something
}
}
}
you dont need to get keys to construct context, because storage account variable contains the context. and then you need to iterate containers and blobs

Azure PowerShell Download blob contents from container

I'm trying to download sme blob files from an azure storage account.
There are a mix of other containers and blockblobs in the parent container, I need to download just the blockblobs and not the other containers, I can't find a way of seperating them out, also I need to download some blobs from a conatiner within a container.
My code will download all the contents in the parent blob, including all sub containers.
$sub = "MySub"
$staccname = "straccname1234"
$key = "sdcsecurekeythinghere"
$ctx = New-AzureStorageContext -StorageAccountName $staccname `
-StorageAccountKey $key
$cont = "data\download\files.001" ##the container includes other cntainers and subcontainers
$dest = "C:\blb-Downloads"
Select-AzureSubscription -SubscriptionName $sub –Default
Set-AzureSubscription -Currentstaccname $staccname -SubscriptionName $sub
Get-AzureStorageBlob -Container $cont -Context $ctx
$blobs = Get-AzureStorageBlob -Container $cont -Context $ctx
$blobs | Get-AzureStorageBlobContent –Destination $dest -Context $ctx
There are approx 75 files in the parent blob and 123 files in data\downloads.
Using the newer Azure PowerShell Az module, you can use Get-AzStorageBlob to list all the block blobs from the container, then use Get-AzStorageBlobContent to download the blobs.
As already shown by #George Wallace, we can use Where-Object or its alias ? to filter block blob types.
Demo:
$resourceGroup = "myResourceGroup"
$storageAccount = "myStorageAccount"
$container = "myContainerName"
$destination = "./blobs"
# Create destination directory if it doesn't exist
if (-not (Test-Path -Path $destination -PathType Container)) {
New-Item -Path $destination -ItemType Directory
}
# Get storage account with container we want to download blobs from
$storageAccount = Get-AzStorageAccount -Name $storageAccount -ResourceGroupName $resourceGroup
# Get all BlockBlobs from container
$blockBlobs = Get-AzStorageBlob -Container $container -Context $storageAccount.Context
| Where-Object {$_.BlobType -eq "BlockBlob"}
# Download each blob from container into destination directory
$blockBlobs | Get-AzStorageBlobContent -Destination $destination -Force
Can you not just run the following and limit it to BlockBlobs only?
Get-AzureStorageBlob -Container $cont -Context $ctx | ? {$_.BlobType -eq "BlockBlob"}
This works for me
$storageaccountname = "jawadtestsaacc"
$sastoken = ""
$ContainerName = "access"
$Ctx = New-AzStorageContext -StorageAccountName $storageaccountname -SasToken $sastoken
Get-AzStorageBlob -Container "$ContainerName" -Blob "Az.json" -Context $Ctx | Get-AzStorageBlobContent
or
Get-AzStorageBlobContent -Container "$ContainerName" -Blob "Az.json" -Context $Ctx

Resources