Suppose I do this operation between storage accounts:
AzCopy /Source:https://sourceaccount.blob.core.windows.net/mycontainer1 /Dest:https://destaccount.blob.core.windows.net/mycontainer2 /SourceKey:key1 /DestKey:key2 /Pattern:abc.txt
In mycontainer1 I have permission "Blob ..", but in mycontainer2 the permission becomes "Private ..." . Is there a way to prevent this to happen / force the same permission on the "new" container?
AzCopy doesn't support this. It's designed for transferring blobs/files. You need to reconfigure the permission of your destination container by yourself.
Related
Our CI pipeline needs to back up some files to Azure Blob Storage. I'm using the Azure CLI like this: az storage blob upload-batch -s . -d container/directory --account-name myaccount
When giving the service principal contributor access, it works as expected. However, I would like to lock down permissions so that the service principal is allowed to add files, but not delete, for example. What are the permissions required for this?
I've created a custom role giving it the same permissions as Storage Blob Data Contributor minus delete. This (and also just using the Storage Blob Data Contributor role directly) fails with a Storage account ... not found. Ok, I then proceeded to add more read permissions to the blob service. Not enough, now I'm at a point where it wants to do Microsoft.Storage/storageAccounts/listKeys/action. But if I give it access to the storage keys, then what's the point? With the storage keys the SP will have full access to the account, which I want to avoid in the first place. Why is az storage blob upload-batch requesting keys and can I prevent this from happening?
I've created a custom role giving it the same permissions as Storage Blob Data Contributor minus delete. This (and also just using the Storage Blob Data Contributor role directly) fails with a Storage account ... not found.
I can also reproduce your issue, actually what you did will work. The trick is the --auth-mode parameter of the command, if you did not specify it, it will use key by default, then the command will list all the storage accounts in your subscription, when it found your storage account, it will list the keys of the account and use the key to upload blobs.
However, the Storage Blob Data Contributor minus delete has no permission to list storage accounts, then you will get the error.
To solve the issue, just specify the --auth-mode login in your command, then it will use the credential of your service principal to get the access token, then use the token to call the REST API - Put Blob to upload blobs, principle see Authorize access to blobs and queues using Azure Active Directory.
az storage blob upload-batch -s . -d container/directory --account-name myaccount --auth-mode login
i am writing az copy script which captures linux 18.04 system log using azcopy and store it into storage account container, but this whole steps I am doing with terraform automation. i have created machine code and I integrate shell script file with terraform extension.
so the issue is when azcopy copy the file from system and pass to a storage account need azcopy login to authenticate this process but these steps we can't perform through automation.
using following azcopy script and version is v10 please help me on this
AzCopy /Source:/var/log/syslog/Dest:https://testingwt.blob.core.windows.net/insights-operational-logs//SourceKey:y/bUACOu/wogikUT1EG0XeaPC4Y6spHcZly2d26QeENKwMiRpjFu5PwmXrThRbNGS3PiPfqEX8WsYC3dg== /S
updated error of azcopy using linux machine in azure
To upload files to the Storage Blob with a shell script automatically, you can use the SAS token of the storage, or use the azcopy login with a service principal or the VM managed identity.
For the SAS token:
azcopy copy "/path/to/file" "https://account.blob.core.windows.net/mycontainer1/?sv=2018-03-28&ss=bjqt&srt=sco&sp=rwddgcup&se=2019-05-01T05:01:17Z&st=2019-04-30T21:01:17Z&spr=https&sig=MGCXiyEzbtttkr3ewJIh2AR8KrghSy1DGM9ovN734bQF4%3D" --recursive=true
For the Service Principal, you need to set the environment variable AZCOPY_SPA_CLIENT_SECRET with the secret of the service principal as value and assign the role Storage Blob Data Contributor or role Storage Blob Data Owner of the storage Blob:
azcopy login --service-principal --application-id <application-id> --tenant-id=<tenant-id>
azcopy copy "/path/to/file" "https://account.blob.core.windows.net/mycontainer1/" --recursive=true
For the VM managed identity, you need also to assign the VM managed identity with the role Storage Blob Data Contributor or role Storage Blob Data Owner of the storage Blob:
azcopy login --identity
azcopy copy "/path/to/file" "https://account.blob.core.windows.net/mycontainer1/" --recursive=true
But when you use the VM managed identity, you need to execute the shell script in the Azure VM, it means you need to deploy the Terraform in the Azure VM. So the best way is that use a service principal, you can execute the shell script in other Linux OS, for example, your local Linux machine. The SAS token is also a good way without assigning the role. For more details, see the Use Azcopy for the Azure Storage Blob.
I am trying to copy all files from one container to another. I am using AzCopy to accomplish this task.
AzCopy command as below:
azcopy copy "https://xxxxxxx.blob.core.windows.net/customers" "https://xxxxxxx.blob.core.windows.net/archive" --recursive
Error:
Alternatively is it possible to Move files between containers?
Please follow this doc to grant your user account the RBAC role Storage Blob Data Contributor in your account or your containers.
Besides, there isn't a "move" operation for Azure Blob Storage, you need to delete the original container after copying it.
I have Owner permissions on a storage account in Azure. To make a blob container immutable I can use this command in PowerShell:
Set-AzRmStorageContainerImmutabilityPolicy -ResourceGroupName 'xxxxx' -StorageAccountName 'xxxxx' -ContainerName 'TestContainer' -ImmutabilityPeriod 1 #1 day
When I try this on the analytics logs container ("$logs"), I get this error:
Set-AzRmStorageContainerImmutabilityPolicy : The account being accessed
does not have sufficient permissions to execute this operation.
What permissions do I need to do this?
Considering $logs container is a system defined blob container, I don't think you can change the immutability policy for that container. You can't even delete that blob container.
If you're trying to set the blobs in this blob container to auto-expire after a certain period of time, you may want to try Blob Lifecycle Management. You can read more about it here: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-lifecycle-management-concepts.
I am trying to copy data from one azure blob location to another blob location using hadoop distcp command(running this in spark scala). from destination location, users will query the data. during copy transition, if users query the data maybe they will get duplicate data. so, I can acquire lock on azure destination location. is there any fastest way/best way to copy data with transaction.
I recommend you to use AzCopy.
1.Copy a single blob from one container to another within the same storage account
AzCopy /Source:https://myaccount.blob.core.windows.net/mycontainer1 /Dest:https://myaccount.blob.core.windows.net/mycontainer2 /SourceKey:key /DestKey:key /Pattern:abc.txt
2.Copy a single blob from one storage account to another
AzCopy /Source:https://sourceaccount.blob.core.windows.net/mycontainer1 /Dest:https://destaccount.blob.core.windows.net/mycontainer2 /SourceKey:key1 /DestKey:key2 /Pattern:abc.txt
3.Copy all blobs in a container to another storage account
AzCopy /Source:https://sourceaccount.blob.core.windows.net/mycontainer1
/Dest:https://destaccount.blob.core.windows.net/mycontainer2 /SourceKey:key1 /DestKey:key2 /S
For more details, you could refer to this article.