Azure Powershell with storage - azure

I want to copy from Azure file share to blob and I am following below refs, and I wanted a incremental copy which is available in azcopy with not in Start-AzureStorageBlobCopy. And ofcourse Start-AzureStorageBlobIncrementalCopy wont do copy from file share to blob.
https://learn.microsoft.com/en-us/powershell/module/azure.storage/start-azurestorageblobincrementalcopy?view=azurermps-6.8.1
https://learn.microsoft.com/en-us/powershell/module/azure.storage/start-azurestorageblobcopy?view=azurermps-6.8.1
I wrote small commands,
$StorageContext = New-AzureStorageContext -StorageAccountName 'neverdelete' -StorageAccountKey 'XXX'
$Srcsh = Get-AzureStorageFile -ShareName "filetest" -Context $StorageContext
$DestBlob = Get-AzureStorageBlob -Container "client1" -Context $StorageContext
Start-AzureStorageBlobCopy -SrcShare $Srcsh --DestContainer $DestBlob
But this throws an error below:
Start-AzureStorageBlobCopy : Cannot convert 'System.Object[]' to the type 'Microsoft.WindowsAzure.Storage.File.CloudFileShare' required by parameter 'SrcShare'. Specified method
is not supported.
At line:4 char:38
+ Start-AzureStorageBlobCopy -SrcShare $Srcsh --DestContainer $DestBlob
+ ~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Start-AzureStorageBlobCopy], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgument,Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.StartAzureStorageBlobCopy
I dont know what I am doing wrong here and how do I script it so that it could be incremental.
Thanks,
Akshay

You get the error because the
-SrcShare parameter expects a CloudFileShare and you pass a list of AzureStorageFile / AzureStorageDirectory. Also your -DestContainer has two -- and you are missing the context.
So you could e. g. iterate over your AzureStorageFiles and invoke the Start-AzureStorabeBlobCopy cmdlet for each of them:
$Srcsh | ForEach-Object {
Start-AzureStorageBlobCopy -SrcShare ($Srcsh[0] | Select-Object -ExpandProperty Share) -SrcFilePath $_.Name -DestContainer "client1" -Context $StorageContext
}

I suppose that you want to copy the files in the storage fileshare to storage container, you could try the command below.
$StorageContext = New-AzureStorageContext -StorageAccountName 'StorageAccountName' -StorageAccountKey 'xxxxx'
$Srcsh = Get-AzureStorageFile -ShareName "filesharename" -Context $StorageContext | Get-AzureStorageFile
$DestBlob = Get-AzureStorageContainer -Container "containername" -Context $StorageContext
foreach ($item in $Srcsh)
{
Start-AzureStorageBlobCopy -SrcFile $item -DestContainer "DestContainername" -Context $StorageContext
}
My fileshare:
Result in the container:

For Start-AzureStorageBlobCopy (Start-AzureStorageFileCopy), the cmdlets can only copy single File/Blob, can't copy a share/container, and the copy can't be resume if fail.
The incremental copy (https://learn.microsoft.com/en-us/rest/api/storageservices/incremental-copy-blob) can only copy page blob snapshot, and currently File not support it. As you say AzCopy support it, what do you means about "incremental copy"?
If you want to copy all files in the root directory in a share, you can use following command. (Please note, Get-AzureStorageFile will only get the first level file/dir in a share or dir, it won't get the files in subdir.)
Get-AzureStorageFile -ShareName $shareName -Context $StorageContext
| where {$_.GetType().Name -eq "CloudFile"} |Start-AzureStorageBlobCopy -DestContainer $containerName -DestContext $StorageContext
-context $StorageContext
BTW, for any azure powershell problem, the formal way is to open an issue in https://github.com/Azure/azure-powershell/issues, and the proper team will follow up it.

Related

Encrypt Azure Storage account key in powershell script

I'm developing a new powershell script in order to download any blobs from a specific container and the problem is due to security reasons because I do not want to paste in text plain the azure account key.
So I have implemented a solution using 'ConvertTo-SecureString' command but the problem still exists because when I create a connection string to the blob, there appears a message who said: "Server Failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. HTTP Status Code 403 - HTTP Error".
With the key in plain text I'm able to create the connection string properly and then list and download all blobs from the container.
I tried other solutions for example ' $Credential= New-Object System.Management.Automation.PSCredential ('$ShareUser, $SharePassword)'
but there is other problem related with the input is not valid base64 string.
Do you know how to avoid this issues and create a secure connection string with an Azure Storage Account?
Best regards and thanks in advance
Here a part of my powershell script
$SecurePassword= Read-Host -AsSecureString | ConvertFrom-SecureString
$SecurePassword | Out-File -FilePath C:\test_blob\pass_file.xml
$ConfigFile= 'C:\Users\\config_file.xml'
IF (Test-Path) {
[xml]$Config= Get-Content $ConfigFile
[string] $Server = $Config.Config.Server;
[string] $SharePassword = $Config.Config.SharePassword;
} ELSE
{
write-host "File do not exists: $ConfigFile"
}
#BlobStorageInformation
$StorageAccountName='test_acc'
$Container='test'
$DestinationFolder= 'C:\Users\user1\Blobs'
$Context = New-AzStorageConext -StorageAccountName $StorageAccountName -StorageAccountKey $SharePassword
#List of Blobs
$ListBlob=#()
$ListBlob+= Get-AzStorageBlob -context $Context -container $Container | Where-Object {$_.LastModified -lt (Get-Date).AddDAys(-1)}
Why would you maintain the password files or enter storage key manually when you have az powershell. Just login using az powershell, set the subscription and enjoy !
$ResourceGroupName = "YOURRESOURCEGROUPNAME"
$StorageAccountName = "YOURSTORAGEACCOUNTNAME"
$ContainerName = "YOURCONTAINERNAME"
$LocalPath = "D:\Temp"
Write-Output 'Downloading Content from Azure blob to local...'
$storageKey = (Get-AzStorageAccountKey -ResourceGroupName $ResourceGroupName -AccountName $StorageAccountName).value[0]
$storageContext = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $storageKey
$blobs = Get-AzStorageBlob -Container $ContainerName -Context $storageContext
foreach($blob in $blobs)
{
Get-AzStorageBlobContent -Container $ContainerName -Context $storageContext -Force -Destination $LocalPath -Blob $blob.Name
}
Write-Output 'Content Downloaded Successfully !!!'

Powershell - remove all blobs in a container

I need to remove all of the blobs in a specific container before doing a backup to that container. The following does not work. What is the proper way to do this?
Get-AzStorageBlob -Container $ContainerName -Context $ctx | Remove-AzStorageBlob -DeleteSnapshot -Force
I get the following error message for each blob returned by Get-AzStorageBlob:
Remove-AzStorageBlob : The specified blob does not exist. HTTP Status Code: 404 - HTTP Error Message: The specified blob does not exist.
ErrorCode: BlobNotFound
ErrorMessage: The specified blob does not exist.
RequestId:eb2612f4-f01e-0067-5471-3e9f69000000
Time:2019-07-19T20:33:50.1261168Z
At line:15 char:62
+ ... inerName -Context $ctx | Remove-AzStorageBlob -DeleteSnapshot -Force
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Remove-AzStorageBlob], StorageException
+ FullyQualifiedErrorId : StorageException,Microsoft.WindowsAzure.Commands.Storage.Blob.RemoveStorageAzureBlobCommand
I test with your ps script, it's caused by the -DeleteSnapshot -Force, delete it and then it works well.
Get-AzureStorageBlob -Container $ContainerName -Context $ctx | Remove-AzureStorageBlob
And if you want to delete snapshot, just add -Force. Here is the Parameters description, -Force indicates that this cmdlet removes the blob and its snapshot without confirmation.
You could have a try, hope this could help you.
Note: If you are using the newer Az Module, the above code will look like:
Get-AzStorageBlob -Container $ContainerName -Context $ctx | Remove-AzStorageBlob
I don't think the Remove-AzStorageBlob can delete blobs in bulk, so you will have to loop through the blobs you get returned from Get-AzureStorageBlob.
Get-AzStorageBlob -Container $ContainerName -Context $ctx | ForEach-Object {
$_ | Remove-AzureStorageBlob # or: Remove-AzureStorageBlob -ICloudBlob $_.ICloudBlob -Context $ctx
}

How to copy files from one container to another containers fits equally in all dest containers according to size using powershell

I have one container in blob of storage account in azure contains different folder having files of different sizes.
In other side, in same storage account, I have 10 different containers.
I have to copy these files from source container to destination 10 containers but the condition is the files should be equally distributed to all the containers.
I have tried below so far
$key = "abcdxyz"
# declaring the azure context
$context = New-AzStorageContext -StorageAccountName abcd -StorageAccountKey $key
#Getting the data from the blob
$bacdata = Get-AzStorageContainer -Name sourcecontainer* -Context $context | Get-AzStorageBlob
$15=$bacdata | where{$_.Name -like "sourcecontainer1*"} | where{$_.LastModified -gt (get-date).adddays(-1)}
Here is the powershell script to do it:
#Server side storage copy
$SourceStorageAccount = "sourceAccountName"
$SourceStorageKey = "sourceAccountAPIKey"
$DestStorageAccount = "destinationAccountName"
$DestStorageKey = "destinationAccountAPIKey"
$SourceStorageContext = New-AzureStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey
$DestStorageContext = New-AzureStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey
$Containers = Get-AzureStorageContainer -Context $SourceStorageContext
foreach($Container in $Containers)
{
$ContainerName = $Container.Name
if (!((Get-AzureStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName }))
{
Write-Output "Creating new container $ContainerName"
New-AzureStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
}
$Blobs = Get-AzureStorageBlob -Context $SourceStorageContext -Container $ContainerName
$BlobCpyAry = #() #Create array of objects
#Do the copy of everything
foreach ($Blob in $Blobs)
{
$BlobName = $Blob.Name
Write-Output "Copying $BlobName from $ContainerName"
$BlobCopy = Start-CopyAzureStorageBlob -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName
$BlobCpyAry += $BlobCopy
}
#Check Status
foreach ($BlobCopy in $BlobCpyAry)
{
#Could ignore all rest and just run $BlobCopy | Get-AzureStorageBlobCopyState but I prefer output with % copied
$CopyState = $BlobCopy | Get-AzureStorageBlobCopyState
$Message = $CopyState.Source.AbsolutePath + " " + $CopyState.Status + " {0:N2}%" -f (($CopyState.BytesCopied/$CopyState.TotalBytes)*100)
Write-Output $Message
}
}
Overall concept would be similar , just change the function aas per the Az CLI.
here is the copy command in Az CLI
az storage blob copy start
You can find more details here.
https://learn.microsoft.com/en-us/cli/azure/storage/blob/copy?view=azure-cli-latest#az-storage-blob-copy-start
Hope it helps.
how to copy files from one folder in container to another folder?
method 1-azuresynapse>integrate>copydata tool>Built-in copy task>new connection>create connection>dataset>browse file>uncheck recursively>next >next>next
method 2-HOme>resource groups>data factory>review create>open azure data factory>ingest>select source and file>select destination >finish>check in container if file is transfered or not

Azure PowerShell Download blob contents from container

I'm trying to download sme blob files from an azure storage account.
There are a mix of other containers and blockblobs in the parent container, I need to download just the blockblobs and not the other containers, I can't find a way of seperating them out, also I need to download some blobs from a conatiner within a container.
My code will download all the contents in the parent blob, including all sub containers.
$sub = "MySub"
$staccname = "straccname1234"
$key = "sdcsecurekeythinghere"
$ctx = New-AzureStorageContext -StorageAccountName $staccname `
-StorageAccountKey $key
$cont = "data\download\files.001" ##the container includes other cntainers and subcontainers
$dest = "C:\blb-Downloads"
Select-AzureSubscription -SubscriptionName $sub –Default
Set-AzureSubscription -Currentstaccname $staccname -SubscriptionName $sub
Get-AzureStorageBlob -Container $cont -Context $ctx
$blobs = Get-AzureStorageBlob -Container $cont -Context $ctx
$blobs | Get-AzureStorageBlobContent –Destination $dest -Context $ctx
There are approx 75 files in the parent blob and 123 files in data\downloads.
Using the newer Azure PowerShell Az module, you can use Get-AzStorageBlob to list all the block blobs from the container, then use Get-AzStorageBlobContent to download the blobs.
As already shown by #George Wallace, we can use Where-Object or its alias ? to filter block blob types.
Demo:
$resourceGroup = "myResourceGroup"
$storageAccount = "myStorageAccount"
$container = "myContainerName"
$destination = "./blobs"
# Create destination directory if it doesn't exist
if (-not (Test-Path -Path $destination -PathType Container)) {
New-Item -Path $destination -ItemType Directory
}
# Get storage account with container we want to download blobs from
$storageAccount = Get-AzStorageAccount -Name $storageAccount -ResourceGroupName $resourceGroup
# Get all BlockBlobs from container
$blockBlobs = Get-AzStorageBlob -Container $container -Context $storageAccount.Context
| Where-Object {$_.BlobType -eq "BlockBlob"}
# Download each blob from container into destination directory
$blockBlobs | Get-AzStorageBlobContent -Destination $destination -Force
Can you not just run the following and limit it to BlockBlobs only?
Get-AzureStorageBlob -Container $cont -Context $ctx | ? {$_.BlobType -eq "BlockBlob"}
This works for me
$storageaccountname = "jawadtestsaacc"
$sastoken = ""
$ContainerName = "access"
$Ctx = New-AzStorageContext -StorageAccountName $storageaccountname -SasToken $sastoken
Get-AzStorageBlob -Container "$ContainerName" -Blob "Az.json" -Context $Ctx | Get-AzStorageBlobContent
or
Get-AzStorageBlobContent -Container "$ContainerName" -Blob "Az.json" -Context $Ctx

How to break a lease on Blob Storage in Azure with PowerShell?

How do I break a lease on an item in Blob Storage utilizing PowerShell?
I'm receiving the following when trying to upload something over the current image:
Add-AzureRmVhd : The remote server returned an error: (412) There is currently a lease on the blob and no lease ID was specified in the request..
At line:1 char:1
+ Add-AzureRmVhd -Destination $osDiskUri -LocalFilePath $localFileName ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Add-AzureRmVhd], StorageException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Compute.StorageServices.AddAzureVhdCommand
Login to the old portal and navigate to the Virtual Machines then the Images tab the url will be https://manage.windowsazure.com/#yourname.onmicrosoft.com#Workspaces/VirtualMachineExtension/images. Select the image and choose Delete on the bottom.
After that go to your storage and delete it.
You can also try the following which will remove blobs for a given container and then remove the container.
Add-AzureAccount
Get-AzureSubscription | Format-Table SubscriptionName, IsDefault, IsCurrent, CurrentStorageAccountName
$SubscriptionName = 'Your subsscription name'
Select-AzureSubscription -SubscriptionName $SubscriptionName
Get-AzureSubscription -Default
Get-AzureStorageAccount | Format-Table -Property StorageAccountName, Location, AccountType, StorageAccountStatus
$StorageAccountName = "Your storage account"
$StorageAccountKey = (Get-AzureStorageKey -StorageAccountName $StorageAccountName).Primary
$ContainerName = "Your container name"
$Context = New-AzureStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
#Get a reference to all the blobs in the container.
$blobs = Get-AzureStorageBlob -Container $ContainerName -Context $Context
#Remove lease on each Blob
$blobs | %{$_.ICloudBlob.BreakLease()}
#Delete blobs in a specified container.
$blobs| Remove-AzureStorageBlob
Remove-AzureStorageContainer -Container $ContainerName -Context $Context
If you want to break a seal on a blob you can use the How to break the locked lease of blob storage in Microsoft Azure (PowerShell)
$key = (Get-AzureRmStorageAccountKey -ResourceGroupName $selectedStorageAccount.ResourceGroupName -name $selectedStorageAccount.StorageAccountName -ErrorAction Stop)[0].value
$storageContext = New-AzureStorageContext -StorageAccountName $selectedStorageAccount.StorageAccountName -StorageAccountKey $key -ErrorAction Stop
$storageContainer = Get-AzureStorageContainer -Context $storageContext -Name $ContainerName -ErrorAction Stop
$blob = Get-AzureStorageBlob -Context $storageContext -Container $ContainerName -Blob $BlobName -ErrorAction Stop
$leaseStatus = $blob.ICloudBlob.Properties.LeaseStatus;
If($leaseStatus -eq "Locked")
{
$blob.ICloudBlob.BreakLease()
Write-Host "Successfully broken lease on '$BlobName' blob."
}
Else
{
#$blob.ICloudBlob.AcquireLease($null, $null, $null, $null, $null)
Write-Host "The '$BlobName' blob's lease status is unlocked."
}
If you want to a script for ARM resources you can use the
How to break the locked lease of blob storage by ARM in Microsoft Azure(PowerShell)
The lease is likely from something like a VM, or something else using the Blog Storage. As a result manually releasing the lease could cause problems.
With that said, the following PowerShell command should do the trick:
Get-AzureRmStorageAccount -Name "STORAGE_ACCOUNT_NAME" | Get-AzureStorageBlob -name "CONTAINER_NAME").ICloudBlob.BreakLease()
If its a VM, you should see the following post on removing the disk:
Cannot delete blob: There is currently a lease on the blob and no lease ID was specified in the request
However, if you simply want to replace the drive used by every machine that uses the given blob, stopping the VM, releasing the lease, uploading a new image, and starting the VM appears to work.

Resources