How to download all the PowerShell files from the Azure storage
account container to a zip folder through power shell cmdlets?
As of now, the below cmdlet helps to download a specific blob by its name
$blob = Get-AzureStorageBlobContent -Container hemac -Blob "CreateAndDeploy-Eventhubs.ps1" -Context $ctx -Destination $f -Force
First set a folder to download all the blobs
1.Provide the full path to a directory you wish to use for downloaded blob
$DestinationFolder = "<C:\DownloadedBlobs>"
Create the destination directory and download the blob
New-Item -Path $DestinationFolder -ItemType Directory -Force
$blob | Get-AzureStorageBlobContent –Destination $DestinationFolder
Now zip the entire folder.
$folderToZip = "C:\DownloadedBlobs"
$rootfolder = Split-Path -Path $folderToZip
$zipFile = Join-Path -Path $rootfolder -ChildPath "ZippedFile.zip"
Then you should use this - docs
Compress-Archive -Path $folderToZip -DestinationPath $zipFile -Verbose
The zipped file will be in the same directory as the download folder
Related
The code below works manually but what I'm trying to do is find a way I can run it in an automated way through a job and log in to my azure account also not expose the account key in the code.
any advice?
{Connect-AzAccount
Select-AzSubscription -SubscriptionId <subId>
$ctx = New-AzStorageContext -StorageAccountName <Accounttest> -UseConnectedAccount
$ctx = New-AzStorageContext -StorageAccountName <Accounttest>
-StorageAccountKey <accoutnKey>
$filesystemName = "DataFiles"
New-AzStorageContainer -Context $ctx -Name $filesystemName
# folder directory
$filesystemName = " DataFiles "
$dirname = "my-directory/"
New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname -Directory
#Upload a file to a directory
$localSrcFile = "upload2.txt"
$filesystemName = " DataFiles "
$dirname = "my-directory/"
$destPath = $dirname + (Get-Item $localSrcFile).Name
New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $destPath -Source $localSrcFile -Force
}
I think it will be better to put your PowerShell script in a DevOps pipeline in Azure Pipelines for instance. A Service principal would be needed. You would need to configure the service connection in Azure DevOps and your service principal should be Contributor on the resource group where the Data Lake Storage is located.
See link: Manage service connections
Below is the runbook code I am using to save the file to azure fileshare. But unable to save in subdirectory.
#Set the context using the storage account name and key
$context = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey
$s = Get-AzureStorageShare "X1" –Context $context
$ErrorLogFileName = "Test.csv"
$LogItem = New-Item -ItemType File -Name $ErrorLogFileName
$_ | Out-File -FilePath $ErrorLogFileName -Append
Set-AzureStorageFileContent –Share $s –Source $ErrorLogFileName
Here I have a folder structure like X1/X2. But unable to get there and save the Test.csv. infact able to save it X1 folder on the Azure fileshare. Any idea ?
Thanks in Advance
You can specify the -Path parameter for Set-AzureStorageFileContent.
For example, the file share is X1, and there is a folder X2 inside X1. Then you can use the command below:
Set-AzureStorageFileContent –Share "X1" –Source $ErrorLogFileName -Path "X2" -Context $context
By the way, the command you're using is old, please try to consider using the latest az powershell command. For example, you can use Set-AzStorageFileContent instead of Set-AzureStorageFileContent(if you try to use the az module, then please replace all the old commands with the new ones).
I am trying to convert .CSV file to .ZIP using Powershell and then store the ZIP file in Azure blob storage.
Invoke-Sqlcmd -ServerInstance $SvrName -Database $dataname -Username $UserName -Password $Password -Query $getTABLEdetails | Export-CSV $filename1 -Append -NoTypeInformation
With the above command I am storing the output in .CSV file. Now I want to convert the .CSV file .ZIP file. And store the ZIP file in Azure blob storage.
$context = New-AzureStorageContext -StorageAccountName 'Accountname' -StorageAccountKey 'storagekey'
Set-AzureStorageBlobContent -Container zipfiles -File $filename1 -Blob $filename1 -Context $context -Force
With the above command I can store it in .CSV file. But I want to store the file in .ZIP format.
Is their any way I can achieve this?
Please help
Use Compress-Archive to place your CSV file in a ZIP archive file before uploading to Blob storage.
To extract the filename from the CSV file, I use -LeafBase from Split-Path, then format this filename into a ZIP file path using the Format operator -f. Could also just concatenate with (Split-Path -Path $csvFile -LeafBase) + '.zip'.
$storageAccountName = 'Accountname'
$storageAccountKey = 'storagekey'
$containerName = 'zipfiles'
$csvFile = 'data.csv'
# Format into data.zip
$zipFile = '{0}.zip' -f (Split-Path -Path $csvFile -LeafBase)
# Create archive file
# Use -Update to overwrite if necessary
Compress-Archive -Path $csvFile -DestinationPath $zipFile -Update
# Get current storage account context
$context = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey
# Upload .zip file to Blob storage container
# Use -Force to skip prompts
Set-AzureStorageBlobContent -Container $containerName -Blob $zipFile -File $zipFile -Context $context -Force
To use the newer Az modules, we can replace New-AzureStorageContext with New-AzStorageContext and Set-AzureStorageBlobContent with Set-AzStorageBlobContent.
We are trying to grant read/write access to many folders in our Azure data Lake gen 2 containers and although we can do this through the UI, it's quite tedious and has to be repeated for all environments. Has anyone used a better way using Powershell to automate or at least parameterize this process of granted access to Azure Data Lake gen 2 containers and avoid granting access manually?
Unfortunately I couldn't get this to work using the following link or other documentation as it's for Gen 1 but it's very similar to what I need to do for gen 2.
https://www.sqlchick.com/entries/2018/3/17/assigning-data-permissions-for-azure-data-lake-store-part-3
According to my test, we can use the PowerShell to manage Azure Data Lake Gen2 permissions. For more details, please refer to the document
Install the required module
install-Module PowerShellGet –Repository PSGallery –Force
install-Module Az.Storage -Repository PSGallery -RequiredVersion 1.9.1-preview –AllowPrerelease –AllowClobber –Force
Besides, please note that if you want to install the module, you need to meet some conditions
.NET Framework is 4.7.2 or greater installed
PowerShell is 5.1 or higher
Script
Connect-AzAccount
$groupName=""
$accountName=""
$account= Get-AzStorageAccount -ResourceGroupName $groupName -Name $accountName
$ctx = $account.Context
$filesystemName = "test"
$dirname="template/"
$Id = "<the Object ID of user, group or service principal>"
$dir=Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname
$acl = New-AzDataLakeGen2ItemAclObject -AccessControlType user -EntityId $id -Permission "rw-" -InputObject $dir.ACL
Update-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname -Acl $acl
$dir=Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname
$dir.ACL
Thanks Jim Xu for providing the script above. I'm just complementing the code with the following items :
Get all folders from the container
Assign ACL for all folders
Propagate ACL to all subfolders
$groupName="resource group name"
$accountName="storage account name"
$account= Get-AzStorageAccount -ResourceGroupName $groupName -Name $accountName
$ctx = $account.Context
$filesystemName = "container name"
$Id = (Get-AzADGroup -DisplayName '<type user / group name here>').Id
$items = Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $filesystemName
foreach ( $item in $items) {
$dir = Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path "$($item.Path)/"
$acl = New-AzDataLakeGen2ItemAclObject -AccessControlType group -EntityId $id -Permission "rwx" -InputObject $dir.ACL -DefaultScope
# Update ACL on blob item
Update-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path "$($item.Path)/" -Acl $acl
# Propagate ACL to child blob items
Set-AzDataLakeGen2AclRecursive -Context $ctx -FileSystem $filesystemName -Path "$($item.Path)/" -Acl $acl
}
I have storage that's where I host a static website. I want to create a task that transfers my build files into that storage.
$context = New-AzureStorageContext -StorageAccountName $env:prSourceBranchName -StorageAccountKey "e4Nt0i1pcsYWnzLKw4PRdwu+************/qMj7fXyJK6lS4YTlPCOdbFlEG2LN9g2i5/yQ=="
Get-ChildItem -Path $env:System_DefaultWorkingDirectory/_ClientWeb-Build-CI/ShellArtifact/out/build
Get-ChildItem -Path $env:System_DefaultWorkingDirectory/_ClientWeb-Build-CI/ShellArtifact/out/build ls -File -Recurse | Set-AzureStorageBlobContent -Container $env:prSourceBranchName -Context $context
When I run the code:
I can see the files that need to be uploaded but when I check my $web blob (created by the Static Website) it's empty.
https://learn.microsoft.com/en-us/powershell/module/azure.storage/set-azurestorageblobcontent?view=azurermps-6.13.0 check out the second example.
Can someone explain why nothing is happening?
I want to do this > https://learn.microsoft.com/en-us/cli/azure/storage/blob?view=azure-cli-latest#az-storage-blob-upload-batch but through AzureRM.
Remove ls from your last line of code which is used to upload to blob storage:
Change:
Get-ChildItem -Path $env:System_DefaultWorkingDirectory/_ClientWeb-Build-CI/ShellArtifact/out/build ls -File -Recurse | Set-AzureStorageBlobContent -Container $env:prSourceBranchName -Context $context
To:
Get-ChildItem -Path $env:System_DefaultWorkingDirectory/_ClientWeb-Build-CI/ShellArtifact/out/build -File -Recurse | Set-AzureStorageBlobContent -Container $env:prSourceBranchName -Context $context