Download parquet file from ADL Gen2 using Get-AzureStorageBlobContent - azure

Question

I am trying below powershell command to download parquet file from ADL Gen 2 to local system.
Below is the code snippet
#this appid has access to ADL
[string] $AppID = "bbb88818-aaaa-44fb-q2345678901y"
[string] $TenantId = "ttt88888-xxxx-yyyy-q2345678901y"
[string] $SubscriptionName = "Sub Sample"
[string] $LocalTargetFilePathName = "D:\MoveToModern"
Write-Host "AppID = " $AppID
Write-Host "TenantId = " $TenantId
Write-Host "SubscriptionName = " $SubscriptionName
Write-Host "AzureDataLakeAccountName = " AzureDataLakeAccountName
Write-Host "AzureDataLakeSrcFilePath = " $AzureDataLakeSrcFilePath
Write-Host "LocalTargetFilePathName = " $LocalTargetFilePathName
#this is the access key of the appid
$AccessKeyValue = "1234567=u-r.testabcdefaORYsw5AN5"
$azurePassword = ConvertTo-SecureString $AccessKeyValue -AsPlainText -Force
$psCred = New-Object System.Management.Automation.PSCredential($AppID, $azurePassword)
Login-AzureRmAccount -Credential $psCred -ServicePrincipal -Tenant $TenantId
Get-AzureRmSubscription
Get-AzureRmSubscription -SubscriptionName $SubscriptionName | Set-AzureRmContext
Get-AzureStorageBlobContent -Container "/Test/Partner/Account/" -Blob "Account.parquet" -Destination "D:\MoveToModern"
But I am getting below error
May be we have to set the storage context. Can you please let me know how to set the storage context with the service principal. (I have app id & app key of service principal. W.r.t ADL Gen2 source, I just have the path details. Source team has provided access to service principal)

Answer 1

If you want to download files from Azure Data Lake Gen2, I suggest you use PowerShell module Az.Storage. Meanwhile, regarding how to implement it with a service principal, you have two choices.
1. Use Azure RABC Role
If you use Azure RABC Role, you need to assign the special role(Storage Blob Data Reader) to the sp.
For example
$AppID = ""
$AccessKeyValue = ""
$TenantId=""
$SubscriptionName = ""
#1. Assign role at the storage account level
#please use owner account to login
Connect-AzAccount -Tenant $TenantId -Subscription $SubscriptionName
New-AzRoleAssignment -ApplicationId $AppID -RoleDefinitionName "Storage Blob Data Reader" `
-Scope "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>"
# download
$azurePassword = ConvertTo-SecureString $AccessKeyValue -AsPlainText -Force
$psCred = New-Object System.Management.Automation.PSCredential($AppID, $azurePassword)
Connect-AzAccount -Credential $psCred -ServicePrincipal -Tenant $TenantId -Subscription $SubscriptionName
$AzureDataLakeAccountName = "testadls05"
$ctx =New-AzStorageContext -StorageAccountName $AzureDataLakeAccountName -UseConnectedAccount
$LocalTargetFilePathName = "D:\test.parquet"
$filesystemName="test"
$path="2020/10/28/test.parquet"
Get-AzDataLakeGen2ItemContent -Context $ctx -FileSystem $filesystemName -Path $path -Destination $LocalTargetFilePathName
Use Access control lists
If you use the method, to grant a security principal read access to a file, you'll need to give the security principal Execute permissions to the container, and to each folder in the hierarchy of folders that lead to the file.
For example
$AppID = ""
$AccessKeyValue = ""
$TenantId=""
$azurePassword = ConvertTo-SecureString $AccessKeyValue -AsPlainText -Force
$psCred = New-Object System.Management.Automation.PSCredential($AppID, $azurePassword)
Connect-AzAccount -Credential $psCred -ServicePrincipal -Tenant $TenantId
$AzureDataLakeAccountName = "testadls05"
$ctx =New-AzStorageContext -StorageAccountName $AzureDataLakeAccountName -UseConnectedAccount
$filesystemName="test"
$path="2020/10/28/test.parquet"
$LocalTargetFilePathName = "D:\test1.parquet"
Get-AzDataLakeGen2ItemContent -Context $ctx -FileSystem $filesystemName -Path $path -Destination $LocalTargetFilePathName
For more details, please refer to
https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control
https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control-model
https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-powershell