Azure cross-account copy using AzCopy and Shared Access Key - azure

Question

I want to use AzCopy to copy a blob from account A to account B. But instead of using access key for the source, I only have access to the Shared Access Key. I've tried appending the SAS after the URL, but it throws a 404 error. This is the syntax I tried
AzCopy "https://source-blob-object-url?sv=blah-blah-blah-source-sas" "https://dest-blob-object-url" /destkey:base64-dest-access-key
The error I got was
Error parsing source location "https://source-blob-object-url?sv=blah-blah-blah-source-sas":
The remote server returned an error: (404) Not Found.
How can I get AzCopy to use the SAS URL? Or that it doesn't support SAS?
Update:
With the SourceSAS and FilePattern options, I'm still getting the 404 error. This is the command I use:
AzCopy [source-container-url] [destination-container-url] [file-pattern] /SourceSAS:"?sv=2013-08-15&sr=c&si=ReadOnlyPolicy&sig=[signature-removed]" /DestKey:[destination-access-key]
This will get me a 404 Not Found. If I change the signature to make it invalid, AzCopy will throw a 403 Forbidden instead.

Answer 1

You're correct. Copy operation using SAS on both source and destination blobs is only supported when source and destination blobs are in same storage account. Copying across storage accounts using SAS is still not supported by Windows Azure Storage. This has been covered (though one liner only) in this blog post from storage team: http://blogs.msdn.com/b/windowsazurestorage/archive/2013/11/27/windows-azure-storage-release-introducing-cors-json-minute-metrics-and-more.aspx. From the post:
Copy blob now allows Shared Access Signature (SAS) to be used for the
destination blob if the copy is within the same storage account.
UPDATE
So I tried it and one thing I realized is that it is meant for copying all blobs from one container to another. Based on my trial/error, a few things you would need to keep in mind are:
Source SAS is for source container and not the blob. Also ensure that you have both Read and List permission on the blob container in the SAS.
If you want to copy a single file, please ensure that it is defined as "filepattern" parameter.
Based on these, can you please try the following:
AzCopy "https://<source account>.blob.core.windows.net/<source container>?<source container sas with read/list permission>" "https://<destination account>.blob.core.windows.net/<destination container>" "<source blob name to copy>" /DestKey:"destination account key"
UPDATE 2
Error parsing source location [container-location]: Object reference
not set to an instance of an object.
I was able to recreate the error. I believe the reason for this error is the version of storage client library (and thus the REST API) which is used to create SAS token. If I try to list contents of a blob container using a SAS token created by using version 3.x of the library, this is the output I get:
<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults ServiceEndpoint="https://cynapta.blob.core.windows.net/" ContainerName="vhds">
<Blobs>
<Blob>
<Name>test.vhd</Name>
<Properties>
<Last-Modified>Fri, 17 May 2013 15:23:39 GMT</Last-Modified>
<Etag>0x8D02129A4ACFFD7</Etag>
<Content-Length>10486272</Content-Length>
<Content-Type>application/octet-stream</Content-Type>
<Content-Encoding />
<Content-Language />
<Content-MD5>uflK5qFmBmek/zyqad7/WQ==</Content-MD5>
<Cache-Control />
<Content-Disposition />
<x-ms-blob-sequence-number>0</x-ms-blob-sequence-number>
<BlobType>PageBlob</BlobType>
<LeaseStatus>unlocked</LeaseStatus>
<LeaseState>available</LeaseState>
</Properties>
</Blob>
</Blobs>
<NextMarker />
</EnumerationResults>
However if I try to list contents of a blob container using a SAS token created by using version 2.x of the library, this is the output I get:
<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults ContainerName="https://cynapta.blob.core.windows.net/vhds">
<Blobs>
<Blob>
<Name>test.vhd</Name>
<Url>https://cynapta.blob.core.windows.net/vhds/test.vhd</Url>
<Properties>
<Last-Modified>Fri, 17 May 2013 15:23:39 GMT</Last-Modified>
<Etag>0x8D02129A4ACFFD7</Etag>
<Content-Length>10486272</Content-Length>
<Content-Type>application/octet-stream</Content-Type>
<Content-Encoding />
<Content-Language />
<Content-MD5>uflK5qFmBmek/zyqad7/WQ==</Content-MD5>
<Cache-Control />
<x-ms-blob-sequence-number>0</x-ms-blob-sequence-number>
<BlobType>PageBlob</BlobType>
<LeaseStatus>unlocked</LeaseStatus>
<LeaseState>available</LeaseState>
</Properties>
</Blob>
</Blobs>
<NextMarker />
</EnumerationResults>
Notice the difference in <EnumerationResults> XElement.
Now AzCopy uses version 2.1.0.4 version of the storage client library. As a part of copying operation it first lists the blobs in source container using the SAS token. Now as we saw above the XML returned is different in both versions so storage client library 2.1.0.4 fails to parse the XML returned by storage service. Because it fails to parse the XML, it is not able to create a Blob object and thus you get the NullReferenceException.
Solution:
One possible solution to this problem is to create a SAS token using version 2.1.0.4 version of the library. I tried doing that and was able to successfully copy the blob. Do give it a try. That should fix the problem you're facing.

Answer 2

Make sure you are using the latest version of the AzCopy and
check this http://blogs.msdn.com/b/windowsazurestorage/archive/2013/09/07/azcopy-transfer-data-with-re-startable-mode-and-sas-token.aspx
/DestSAS and /SourceSAS: This option allows access to storage containers and blobs with a SAS (Shared Access Signature) token. SAS token, which is generated by the storage account owner, grants access to specific containers and blobs with specifc permissions and for a specified period of time.
Example: Upload all files from a local directory to a container using SAS token which offers permits for list and write
AzCopy C:\blobData https://xyzaccount.blob.core.windows.net/xyzcontainer /DestSAS:”?sr=c&si=mypolicy&sig=XXXXX” /s
/DestSAS here is for you to specify the SAS token to access storage container, it should be enclosed in quotes.

Answer 3

You can use IaaS Management Studio to generate the powershell script for you. It is a commercial tool, but you can to that in the trial version. It does not use AzCopy though, but the classic blob API in powershell.
Just "Share the VHD" to get the SAS link. Then "Import from shared link", copy the SAS link you got earlier. Check at the bottom, you'll see a script icon. Put your cursor on it and it shows up.
However, in the trial, you can't copy the script, you'll need to type it by hand, but it is not very long to do so.