I'm getting the error "The specified blob does not exist" running the Start-AzStorageBlobCopy that copies a blob into another new blob in the same folder, the problem looks to be with that destination blob.
$sasKey = 'sv=2019-02-02&ss=b&srt=sco&sp=rwdlac&se=2020-06-07T20:40:57Z&st=2020-01-07T13:40:57Z&spr=https&sig=...'
$storageContext = New-AzStorageContext -StorageAccountName 'my-storage-account' -SasToken $sasKey
$context = $context.Context
Start-AzStorageBlobCopy `
-Context $context `
-SrcContainer 'labo' `
-SrcBlob 'folder/document.txt' `
-DestContainer 'labo' `
-DestBlob 'folder/document2.txt' `
-Force -Verbose
Start-AzStorageBlobCopy: The specified blob does not exist. HTTP Status Code: 404 - HTTP Error Message: The specified blob does not exist.
ErrorCode: CannotVerifyCopySource
ErrorMessage: The specified blob does not exist.
RequestId:5ba3de1b-a01e-008f-4f35-f3644c000000
Time:2020-03-05T21:32:23.8622305Z
At line:5 char:1
+ Start-AzStorageBlobCopy `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Start-AzStorageBlobCopy], StorageException
+ FullyQualifiedErrorId : StorageException,Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.StartAzureStorageBlobCopy
Just find an error in your script:
$storageContext = New-AzStorageContext -StorageAccountName 'my-storage-account' -SasToken $sasKey
# Should be $storageContext, not $context
$context = $storageContext .Context
Then, I test with the following script:
$sasKey = '?sv=2019-02-02&ss=bfqt&srt=sco&sp=rwdlacup&se=2020-03-06T10:02:26Z&st=2020-03-06T02:02:26Z&spr=https,http&sig=tj1c***************************D'
$storageContext = New-AzStorageContext -StorageAccountName 'storagetest789' -SasToken $sasKey
$context = $storageContext.Context
Start-AzStorageBlobCopy `
-Context $context `
-SrcContainer 'test' `
-SrcBlob 'hello/world.json' `
-DestContainer 'pub' `
-DestBlob 'hello/world.json' `
-Force -Verbose
Result:
And, by checking in the storage account, the blob is successfully copied to destination container:
Related
Above hyperlink is the output image.
Hi All,
I want to convert the whole output in a single line result in powershell.Below is my code,
$containername = "testoutput"
$storageAccKey = (Get-AzStorageAccountKey -ResourceGroupName
$rgname -AccountName $storageAccountName)[0].value
$storagecontext = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccKey
New-AzStorageContainer -Name $containername -Context $storagecontext -Permission Off
Write-Output "Container $($containername) created"
Assuming that this is what you're looking for:
$containername = "testoutput"
$storageAccountName = "teststorage001"
$storageAccKey = (Get-AzStorageAccountKey -ResourceGroupName $rgname -AccountName $storageAccountName)[0].value
$storagecontext = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccKey
$result = New-AzStorageContainer -Name $containername -Context $storagecontext -Permission Off
Write-Host "Container $($containername) created in Storage Account-"$($result.Name)",PublicAccess-"$($result.PublicAccess)",LastModified-"$($result.LastModified)""
Either Write-Host or Write-Output can be used. The multiple lines you're seeing is the result of New-AzStorageContainer. Now, I have assigned it to a variable called $result and used in Write-Output.
One way to achieve your requirement is to use if-else condition and check New-AzStorageContainer -Name $containername -Context $storagecontext -Permission Off doesn't result null. Below is the complete code that worked for me.
$rgname = "<YOUR_RESOURCE_GROUP>"
$containername = "<YOUR_CONTAINER_NAME>"
$storageAccountName = "<YOUR_STORAGE_ACCOUNT>"
$storageAccKey = (Get-AzStorageAccountKey -ResourceGroupName $rgname -AccountName $storageAccountName)[0].value
$storagecontext = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccKey
if((New-AzStorageContainer -Name $containername -Context $storagecontext -Permission Off) -ne $null){
Write-Output "Container $($containername) created"
}else{
Write-Output "Container $($containername) Not created"
}
RESULTS:
I am trying to list all the blobs from various containers that i have in an Azure storage account using powershell
So i run the below commands :
$storageAccountName = "contoso"
$resourceGroup = "Contoso-Rg"
$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroup -Name $storageAccountName).Value[0]
$context=New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountkey
$containerName = Get-AzStorageContainer -Context $context
$storageAccount = Get-AzStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccountName
$ctx = $storageAccount.Context
$containerName | ForEach-Object { Get-AzStorageBlob -Container $_ -Context $ctx }
When i run the last command it throws me the error as shown in the screenshot. Any idea what i am doing wrong and how to fix this
Instead of passing $_ to '-contianer' flag you need to pass $_.Name to container parameter in foreach-object.
Posting making the above changes to the shared script we are able to pull all the bolbs inside the container.
Here is the modified script :
$storageAccountName = "<strgAccountName>"
$resourceGroup = "<rgName>"
$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroup -Name $storageAccountName).Value[0]
$context=New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountkey
$containerName = Get-AzStorageContainer -Context $context
$storageAccount = Get-AzStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccountName
$containerName | ForEach-Object { Get-AzStorageBlob -Container $_.Name -Context $context }
Here is the sample output for reference:
I need to remove all of the blobs in a specific container before doing a backup to that container. The following does not work. What is the proper way to do this?
Get-AzStorageBlob -Container $ContainerName -Context $ctx | Remove-AzStorageBlob -DeleteSnapshot -Force
I get the following error message for each blob returned by Get-AzStorageBlob:
Remove-AzStorageBlob : The specified blob does not exist. HTTP Status Code: 404 - HTTP Error Message: The specified blob does not exist.
ErrorCode: BlobNotFound
ErrorMessage: The specified blob does not exist.
RequestId:eb2612f4-f01e-0067-5471-3e9f69000000
Time:2019-07-19T20:33:50.1261168Z
At line:15 char:62
+ ... inerName -Context $ctx | Remove-AzStorageBlob -DeleteSnapshot -Force
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Remove-AzStorageBlob], StorageException
+ FullyQualifiedErrorId : StorageException,Microsoft.WindowsAzure.Commands.Storage.Blob.RemoveStorageAzureBlobCommand
I test with your ps script, it's caused by the -DeleteSnapshot -Force, delete it and then it works well.
Get-AzureStorageBlob -Container $ContainerName -Context $ctx | Remove-AzureStorageBlob
And if you want to delete snapshot, just add -Force. Here is the Parameters description, -Force indicates that this cmdlet removes the blob and its snapshot without confirmation.
You could have a try, hope this could help you.
Note: If you are using the newer Az Module, the above code will look like:
Get-AzStorageBlob -Container $ContainerName -Context $ctx | Remove-AzStorageBlob
I don't think the Remove-AzStorageBlob can delete blobs in bulk, so you will have to loop through the blobs you get returned from Get-AzureStorageBlob.
Get-AzStorageBlob -Container $ContainerName -Context $ctx | ForEach-Object {
$_ | Remove-AzureStorageBlob # or: Remove-AzureStorageBlob -ICloudBlob $_.ICloudBlob -Context $ctx
}
I want to copy from Azure file share to blob and I am following below refs, and I wanted a incremental copy which is available in azcopy with not in Start-AzureStorageBlobCopy. And ofcourse Start-AzureStorageBlobIncrementalCopy wont do copy from file share to blob.
https://learn.microsoft.com/en-us/powershell/module/azure.storage/start-azurestorageblobincrementalcopy?view=azurermps-6.8.1
https://learn.microsoft.com/en-us/powershell/module/azure.storage/start-azurestorageblobcopy?view=azurermps-6.8.1
I wrote small commands,
$StorageContext = New-AzureStorageContext -StorageAccountName 'neverdelete' -StorageAccountKey 'XXX'
$Srcsh = Get-AzureStorageFile -ShareName "filetest" -Context $StorageContext
$DestBlob = Get-AzureStorageBlob -Container "client1" -Context $StorageContext
Start-AzureStorageBlobCopy -SrcShare $Srcsh --DestContainer $DestBlob
But this throws an error below:
Start-AzureStorageBlobCopy : Cannot convert 'System.Object[]' to the type 'Microsoft.WindowsAzure.Storage.File.CloudFileShare' required by parameter 'SrcShare'. Specified method
is not supported.
At line:4 char:38
+ Start-AzureStorageBlobCopy -SrcShare $Srcsh --DestContainer $DestBlob
+ ~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Start-AzureStorageBlobCopy], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgument,Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.StartAzureStorageBlobCopy
I dont know what I am doing wrong here and how do I script it so that it could be incremental.
Thanks,
Akshay
You get the error because the
-SrcShare parameter expects a CloudFileShare and you pass a list of AzureStorageFile / AzureStorageDirectory. Also your -DestContainer has two -- and you are missing the context.
So you could e. g. iterate over your AzureStorageFiles and invoke the Start-AzureStorabeBlobCopy cmdlet for each of them:
$Srcsh | ForEach-Object {
Start-AzureStorageBlobCopy -SrcShare ($Srcsh[0] | Select-Object -ExpandProperty Share) -SrcFilePath $_.Name -DestContainer "client1" -Context $StorageContext
}
I suppose that you want to copy the files in the storage fileshare to storage container, you could try the command below.
$StorageContext = New-AzureStorageContext -StorageAccountName 'StorageAccountName' -StorageAccountKey 'xxxxx'
$Srcsh = Get-AzureStorageFile -ShareName "filesharename" -Context $StorageContext | Get-AzureStorageFile
$DestBlob = Get-AzureStorageContainer -Container "containername" -Context $StorageContext
foreach ($item in $Srcsh)
{
Start-AzureStorageBlobCopy -SrcFile $item -DestContainer "DestContainername" -Context $StorageContext
}
My fileshare:
Result in the container:
For Start-AzureStorageBlobCopy (Start-AzureStorageFileCopy), the cmdlets can only copy single File/Blob, can't copy a share/container, and the copy can't be resume if fail.
The incremental copy (https://learn.microsoft.com/en-us/rest/api/storageservices/incremental-copy-blob) can only copy page blob snapshot, and currently File not support it. As you say AzCopy support it, what do you means about "incremental copy"?
If you want to copy all files in the root directory in a share, you can use following command. (Please note, Get-AzureStorageFile will only get the first level file/dir in a share or dir, it won't get the files in subdir.)
Get-AzureStorageFile -ShareName $shareName -Context $StorageContext
| where {$_.GetType().Name -eq "CloudFile"} |Start-AzureStorageBlobCopy -DestContainer $containerName -DestContext $StorageContext
-context $StorageContext
BTW, for any azure powershell problem, the formal way is to open an issue in https://github.com/Azure/azure-powershell/issues, and the proper team will follow up it.
i have write a script to copy one blob container data into another blob container
it copying only blob name and showing completed inside VHDs are not copying can any one please help in script,,,
what exactly i am trying is
ex:- storage account 1 contain 3 vhds
i have created new storage account in the same log and trying to copy all vhds into new storage account
In the script i have passed few parameters to copy the vhds with date
when i am running that script it only creating name but Vhds are not copying
Thanks in Advance
According to your description, you want to copy all blobs to another storage accounts, we can use this script to do it:
Add this to a new runbook:
$connectionName = "AzureRunAsConnection"
try
{
# Get the connection "AzureRunAsConnection "
$servicePrincipalConnection=Get-AutomationConnection -Name $connectionName
"Logging in to Azure..."
Add-AzureRmAccount `
-ServicePrincipal `
-TenantId $servicePrincipalConnection.TenantId `
-ApplicationId $servicePrincipalConnection.ApplicationId `
-CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint
}
catch {
if (!$servicePrincipalConnection)
{
$ErrorMessage = "Connection $connectionName not found."
throw $ErrorMessage
} else{
Write-Error -Message $_.Exception
throw $_.Exception
}
}
$RGName = "vm"
$SAName = "jasondisk321"
$ConName = "vhd"
$key = "UUzQRoWeIMHzwzwJW9LxtgmwaJJS/Ac3DoXnPMHFIbUmupDpQ+KXCWG8ISJ4E20zjq7ugPtgN4vtVIv3A4m2Pg=="
$Ctx = New-AzureStorageContext -StorageAccountName $SAName -StorageAccountKey $Key
$List = Get-AzureStorageBlob -Blob * -Container $ConName -Context $Ctx
$List = $List.ICloudBlob.Uri.AbsoluteUri
$storageAccount = "jasondisk322"
$storageKey = "6lRJq6hTS1aHfVF4/iWskq/QS+tu4Jm/2zdz7Mo6AINGZOQKUiHtOAKmdZhBAWbcNEcBQq0YxZjXHgHha/iUKw=="
$destContext = New-AzureStorageContext –StorageAccountName $storageAccount -StorageAccountKey $storageKey
$containerName = "vhd"
foreach ( $l in $list ){
$bn = ($l -split '/')[4]
Start-AzureStorageBlobCopy -srcUri $l -context $Ctx -DestContainer $containerName -DestBlob $bn -DestContext $destContext
}
If you want to use powershell on your local PC, we can use this script:
Login-AzureRmAccount
$RGName = "vm" #source storage account resource group name
$SAName = "jasondisk321" #source storage account name
$ConName = "vhd" #source container name
$key = "UUzQRoWeIMHzwzwJW9LxtgmwaJJS/Ac3DoXnPMHFIbUmupDpQ+KXCWG8ISJ4E20zjq7ugPtgN4vtVIv3A4m2Pg=="#source storage account key
$Ctx = New-AzureStorageContext -StorageAccountName $SAName -StorageAccountKey $Key
$List = Get-AzureStorageBlob -Blob * -Container $ConName -Context $Ctx
$List = $List.ICloudBlob.Uri.AbsoluteUri
$storageAccount = "jasondisk322"#destination storage account name
$storageKey = "6lRJq6hTS1aHfVF4/iWskq/QS+tu4Jm/2zdz7Mo6AINGZOQKUiHtOAKmdZhBAWbcNEcBQq0YxZjXHgHha/iUKw=="#destination storage account key
$destContext = New-AzureStorageContext –StorageAccountName $storageAccount -StorageAccountKey $storageKey
$containerName = "vhd"#destination container name
foreach ( $l in $list ){
$bn = ($l -split '/')[4]
Start-AzureStorageBlobCopy -srcUri $l -context $Ctx -DestContainer $containerName -DestBlob $bn -DestContext $destContext
}
You can do it with AzCopy:
AzCopy /source:https://[SourceStorageAccountName].blob.core.windows.net/vhds /dest:https://[DestStprageAccountName].blob.core.windows.net/vhds /sourcekey:<here-is-source-key> /destkey:<here-is-destination-key> /Pattern:[vhd-name].vhd
more info: https://learn.microsoft.com/en-us/azure/storage/storage-use-azcopy
If you want to automate, then you can use powershell using Start-AzureStorageBlobCopy:
Select-AzureSubscription "my subscription"
### Source VHD (West US) - anonymous access container ###
$srcUri = "http://mwwestus1.blob.core.windows.net/source/testcopy1.vhd"
### Target Storage Account (East US) ###
$storageAccount = "mweastus1"
$storageKey = "STORAGEACCOUNTKEY"
### Create the destination context for authenticating the copy
$destContext = New-AzureStorageContext –StorageAccountName $storageAccount `
-StorageAccountKey $storageKey
### Target Container Name
$containerName = "copiedvhds"
### Create the target container in storage
New-AzureStorageContainer -Name $containerName -Context $destContext
### Start the Asynchronous Copy ###
$blob1 = Start-AzureStorageBlobCopy -srcUri $srcUri `
-DestContainer $containerName `
-DestBlob "testcopy1.vhd" `
-DestContext $destContext
more info:https://www.opsgility.com/blog/windows-azure-powershell-reference-guide/copying-vhds-blobs-between-storage-accounts/
UPDATE:
AzCopy /Source:https://myaccount1.blob.core.windows.net/myContainer/ /Dest:https://myaccount2.blob.core.windows.net/myContainer/ /SourceKey:key1 /DestKey:key2 /Pattern:ab /SyncCopy