Upload blob with Set-AzStorageBlobContent via pipeline and set ContentType property - azure

Using the Az PowerShell module, I'm trying to enumerate a directory on disk and pipe the output to Set-AzStorageBlobContent to upload to Azure, while preserving the folder structure. This works great, except the ContentType property of all blobs is set to application/octet-stream. I'd like to set it dynamically based on the file extension of the blob being uploaded.
Here's example code for the base case:
Get-ChildItem $SourceRoot -Recurse -File |
Set-AzStorageBlobContent -Container $ContainerName -Context $context -Force
To set the ContentType, I need to add a Properties parameter to Set-AzStorageBlobContent with a value like #{ "ContentType" = "<content type>" }. The content type should be determined from the specific file extension being uploaded. I've written a separate pipelined function that can add a MimeType property to the file object, but I can't figure out how to reference that for the parameter in the pipeline. Example:
function Add-MimeType{
[cmdletbinding()]
param(
[parameter(
Mandatory = $true,
ValueFromPipeline = $true)]
$pipelineInput
)
Process {
$mimeType = Get-MimeType $pipelineInput.Extension
Add-Member -InputObject $pipelineInput -NotePropertyName "MimeType" -NotePropertyValue $mimeType
return $pipelineInput
}
}
function Get-MimeType(
[string]$FileExtension
)
{
switch ($FileExtension.ToLowerInvariant())
{
'.txt' { return 'text/plain' }
'.xml' { return 'text/xml' }
default { return 'application/octet-stream' }
}
}
Get-ChildItem $SourceRoot -Recurse -File |
Add-MimeType |
Set-AzStorageBlobContent -Container $ContainerName -Properties #{"ContentType" = "$($_.MimeType)"} -Context $context -Force
It seems that $_ isn't usable in this context. Is there another way to accomplish this?
The reason I'd like to continue using pipelining is that it appears to work much faster than using a ForEach-Object loop to call the function (where $_ does work).

If you are open to completely different solutions, you can also use AzCopy.
You can upload your whole folder with one command, and AzCopy can also automatically guess the correct mime type based on the file extension. There is also support for Azure Pipelines, if that is part of your setup.
Command could look something like this:
# AzCopy v10 will automatically guess the content type unless you pass --no-guess-mime-type
azcopy copy 'C:\myDirectory' 'https://mystorageaccount.blob.core.windows.net/mycontainer' --recursive
# AzCopy V8
azcopy copy 'C:\myDirectory' 'https://mystorageaccount.blob.core.windows.net/mycontainer' /s /SetContentType
Taken from the output of AzCopy.exe copy --help:
AzCopy automatically detects the content type of the files when uploading from the local disk, based on the file extension or content (if no extension is specified).
The built-in lookup table is small, but on Unix, it is augmented by the local system's mime.types file(s) if available under one or more of these names:
/etc/mime.types
/etc/apache2/mime.types
/etc/apache/mime.types
On Windows, MIME types are extracted from the registry. This feature can be turned off with the help of a flag. Please refer to the flag section.

Related

Download a list of specific files from Azure Blob

I've got an issue with the files download from Azure Blob container. This is not that trivial as it may seem. I saw many examples of how you can download one file, but there's a problem if you need to download a bulk of them.
Issue definition:
I have an Azure Blob container that has ~30k files in it
At the same time, I've got a list of exact file names locally (around 300 files) that I want to download from that Azure Blob container (i.e., I need to download a whole bunch of files selectively, by their names)
I know that all these files exist in the given blob. So, I need a way to iterate over the list of the files and download them from the blob.
What I tried:
I tried 'azcopy copy' command. It works alright if you need to copy one or several files from the blob to your local disk, but there's no way you can pass a huge list of files as a parameter to download those
Tried to search for PowerShell examples that can do the same, but no luck
Please advice.
Please try something like the following. It makes use of Get-AzStorageBlobContent Cmdlet.
The idea is to have an array of blobs you wish to download and then loop over that array and call this Cmdlet for each item.
$accountName = "account-name"
$accountKey = "account-key"
$containerName = "container-name"
$context = New-AzStorageContext -StorageAccountName $accountName -StorageAccountKey $accountKey
$destination = "C:\temp"
$blobNames = #("blob1.txt", "blob2.txt", "blob3.txt", "blob4.txt")
For ($i=0; $i -lt $blobNames.Length; $i++) {
$blob = $blobNames[$i]
Write-Host "Downloading $blob. Please wait."
Get-AzStorageBlobContent -Blob $blob -Container $containerName -Destination $destination -Context $context -Verbose
}
Have you tried "Azure Storage Explorer" soft ?
I was able to download a whole folder from a blob storage thanks to it.
If data in the blob container is in a folder just right click on the folder > Download.
If the files are directly at the root of the container (not stored in a subfolder) you'll have to select all files with the "Select All > Select all files in cache" option then click on "Download".
$accountName = "account-name"
$accountKey = "account-key"
$containerName = "container-name"
$context = New-AzStorageContext -StorageAccountName $accountName -StorageAccountKey $accountKey
$destination = "C:\temp"
$blobNames = Get-Content $destination\list.txt
For ($i=0; $i -lt $blobNames.Length; $i++) {
$blob = $blobNames[$i]
Write-Host "Downloading $blob. Please wait."
Get-AzStorageBlobContent -Blob $blob -Container $containerName -Destination $destination -Context $context -Verbose
}
It'll work like a charm, given that the text file that holds a list of file names for the blobs you need to download is located in the '$destination' directory (could be any directory on your PC, though).
p.s., the files just need to be listed as one column (separated by a caret return, i.e., "\n" at the end of each file name).
Thanks #Gaurav Mantri for the solution.

How to save the file received as response to powershell web request within an Azure function app to an Azure storage container?

I have an Azure function app that makes the following web request:
Invoke-RestMethod -Uri $uri -Method 'GET' -Headers $headers -Body $body -SkipCertificateCheck
On my local machine, I can save the returned XML file as OutFile to local disk. However, in the actual environment this is on an Azure function app and therefore I don't think that I can save the file to disk. Instead, I want to redirect it to a storage container in Azure.
I have tried Azure function output bindings in order to redirect the response there but failed to write the actual file:
Push-OutputBinding -Name outputBlob -Value $response.content
All this write to the storage container is a string value. So how do I write the actual file received as a response to InvokeRestMethod within the Azure function app to the Azure storage container in this environment?
Unless the file is huge, you can indeed save it to the local file system, even when running on Azure. Just select the proper location and don't forget to clean up:
$f = New-TemporaryFile
try {
Invoke-RestMethod ... -OutFile $f.FullName
# Do whatever you want with this file
...
} finally {
Remove-Item $f.FullName
}
Please read your file into byte array and pass it as argument.
using namespace System.Net
# Input bindings are passed in via param block.
param($Request, $TriggerMetadata)
# Write to the Azure Functions log stream.
Write-Host "PowerShell HTTP trigger function processed a request."
New-Item -Path 'test.txt' -ItemType File
Set-Content 'test.txt' 'Welcome to www.thecodemanual.pl'
$file = [System.IO.File]::ReadAllBytes('test.txt')
Push-OutputBinding -Name outputBlob -Value $file
# Associate values to output bindings by calling 'Push-OutputBinding'.
Push-OutputBinding -Name Response -Value ([HttpResponseContext]#{
StatusCode = [HttpStatusCode]::OK
Body = $body
})
and then you will get:

Downloading a file from Adls gen1 using powershell is not working

I am trying a download a file from azure data lake store using powershell script. The code is triggered from a runbook in azure cloud. Looks like Export-AdlStoreItem is not working as expected.I dont get any error messages or compilation errors. In fact when this command is executed a zero byte file is generated in the destination.The name of that file is TemporaryFile2020-06-02_14-56-57.datc18371c2-d39c-4588-9af0-93aa3e136b01Segments
what is happening?.Please help!.
$LocalDwldPath = "T:\ICE_PROD_DATA_SOURCING\FILE_DOWNLOAD_PATH\TemporaryFile$($TimeinSec).dat"
$SourcePath = "Dataproviders/Landing/GCW/HPIndirect/Orders/AMS/gcw_hp_indirect_orders_ams_745_20200601_04_34_01.dat"
$PRODAdlsName = "itgdls01"
Export-AdlStoreItem -Account $PRODAdlsName -Path $("/" + $SourcePath.Trim()) -Destination $LocalDwldPath -Force -ErrorAction Stop
if( Test-Path $LocalDwldPath.Trim() )
{
Get-Content -Path $LocalDwldPath.Trim() -ReadCount 1000 |% { $FileCount += $_.Count }
Remove-Item $LocalDwldPath.Trim()
Set-Content -Path $cntCaptureFile -Value $FileCount
$TimeinSec = TimeStamp2
Add-Content -Value "$TimeinSec Log: Identified file for getting count is $($SourcePath.Trim()) and the count is $FileCount" -Path $logfile
}
else
{
$TimeinSec = TimeStamp2
Add-Content -Value "$TimeinSec Error: Identified file for getting count is $($SourcePath.Trim()) and the count capture failed as local file is not found!" -Path $logfile
}
According to my research, if you want to download a file from azure data lake with PowerShell, we can use the PowerShell command Export-AzDataLakeStoreItem.
For example
Export-AzDataLakeStoreItem -Account <> -Path '/test/test.csv' -Destination 'D:\myDirectory\test.csv'
For more details, please refer to the document
The issue was with the local download path/destination
("T:\ICE_PROD_DATA_SOURCING\FILE_DOWNLOAD_PATH\TemporaryFile$($TimeinSec).dat").
The T:\ drive is a virtual drive/network drive connected as Azure Fileshare.
Instead of T:\ , I have pointed the destination location to a local drive ("F:\ICE_PROD_DATA_SOURCING\FILE_DOWNLOAD_PATH\TemporaryFile$($TimeinSec).dat") and it worked fine.
Its surprising that Powershell didn't give any error messages when it was not able to save the file in a network path pointed to azure fileshare.

Using Set-AzStorageBlobContent to upload only new content without prompts

I'm enumerating a local folder and uploading to Azure storage. I want to only upload new content to my Azure storage. If I use Set-AzStorageBlobContent with -Force, it'll overwrite everything. If I use it without -Force, it'll prompt on items that already exist. I can use Get-AzStorageBlob to check if the item already exists, but it prints red errors if the item does not exist. I can't find a combination of these items that gracefully uploads only new content without printing any errors or prompting. Am I using the wrong approach?
FINAL EDIT: adding working solution based on suggestions from Ivan Yang. Now only new files are uploaded, without any error messages. The key was to use -ErrorAction Stop to convert the error message into an exception, and then catch the exception.
# In my code this is part of a Test-Blob function that returns $blobFound
$blobFound = $false
try
{
$blobInfo = Get-AzStorageBlob `
-Container $containerName `
-Context $storageContext `
-Blob $blobPath `
-ErrorAction Stop
$blobFound = ($null -ne $blobInfo)
}
catch [Microsoft.WindowsAzure.Commands.Storage.Common.ResourceNotFoundException]
{
# Eat the error that'd otherwise be printed
}
# Note in my code this is actually a call to my Test-Blob function
if ($false -eq $blobFound)
{
Set-AzStorageBlobContent `
-Container $containerName `
-Context $storageContext `
-File $sourcePath `
-Blob $blobPath `
-Force # -Force is unnecessary but just being paranoid to avoid prompts
}
I see you have mentioned trying Get-AzStorageBlob, why not use it continually?
The trick here is that you can use try-catch-finally, which can properly handle the error if the blob does not exist in azure.
The sample code works at my side for uploading a single file, and you can modify it to upload multi-files:
$account_name ="xxx"
$account_key ="xxx"
$context = New-AzStorageContext -StorageAccountName $account_name -StorageAccountKey $account_key
#use this flag to determine if a blob exists or not in azure. And assume it exists at first.
$is_exist = $true
try
{
Get-AzStorageBlob -Container test3 -Blob a.txt -Context $context -ErrorAction Stop
}
catch [Microsoft.WindowsAzure.Commands.Storage.Common.ResourceNotFoundException]
{
#if the blob does not exist in azure, do the following
$is_exist = $false
Write-Output "the blob DOES NOT exists."
}
finally
{
#only execute the code when the blob does not exist in azure blob storage.
if(!$is_exist)
{
Set-AzStorageBlobContent -Container test3 -File "d:\myfolder\a.txt" -Blob a.txt -Context $context
Write-Output "uploaded!"
}
}
Not a PowerShell solution but I would suggest that you take a look at AzCopy. It's like RoboCopy but for Azure storage. A command line tool which allows you to synch, copy, move and more. It's free, works on macOS, Linux and Windows. And also, it is fast!
I use AzCopy from PowerShell scripts and it makes lie a lot easier (I'm managing millions of files and the stability and speed of AzCopy really helps)
This command is not smart enough to detect which files are new. You need to keep in the folder just the files you want to upload.
Simply use Set-AzStorageBlobContent -Force all the time.
The alternative is to check for existing file, download the file content, compare the files, and upload if different. The amount of processing/IO will only increase this way.

Azure Runbook Powershell move zip file from Blob Container to FileShare and Unzip the file once in destination

I am trying to move zip files from an Azure Blob container to an Azure fileShare. I want to add in the option to unzip the file on the destination side. I am using Powershell in an automation runbook. The following is a snippit from my function. I have not been able to figure out the Expand-Archive cmdlet where all activity is taking place in storage accounts (no local path c:\, etc)
if ($blob.name.StartsWith($blobMatch)) {
$destFile
$blobName = $blob.name
#Get-AzureStorageFileContent
#Expand-Archive -Path $blob -DestinationPath $destFile
Start-AzureStorageFileCopy `
-SrcBlobName $blob.name `
-SrcContainerName $sourceContainer `
-DestShareName $destinationShare `
-DestFilePath $destFile `
-Context $ctx `
-DestContext $ctx `
-Force
}
I think you won't be able to unzip the file in storage file share, you can put there zipped content, but you can't make the Storage Service unzip it. The workaround is downloading the file first, then unzip it.
Here is a similar issue, you could refer to it.

Resources