I'm attempting to upload 550K files from my local hard drive to Azure Blob Storage using the following command (AzCopy 5.1.1) -
AzCopy /Source:d:\processed /Dest:https://ContainerX.file.core.windows.net/fec-data/Reports/ /DestKey:SomethingSomething== /S
It starts churning right away.
But it's actually creating a new Azure File Storage folder called fec-data/reports rather than creating new blobs in the Azure Blob folder fec-data/reports I've already created.
What am I missing?
Also, is there anyway to keep the date created (or similar) values of the old files?
Thanks,
But it's actually creating a new Azure File Storage folder called
fec-data/reports rather than creating new blobs in the Azure Blob
folder fec-data/reports I've already created.
What am I missing?
The reason you're seeing this behavior is because you're uploading to File storage instead of Blob storage. To upload the files to Blob storage, you need to specify blob service endpoint (blob.core.windows.net). So your command would be:
AzCopy /Source:d:\processed /Dest:https://ContainerX.blob.core.windows.net/fec-data/Reports/ /DestKey:SomethingSomething== /S
Also, is there anyway to keep the date created (or similar) values of
the old files?
Assuming you want to keep the date created of the blob same as that of the desktop file, then it is not possible. Blob's Last Modified Date/Time is a system property that gets assigned when a blob is created and is updated every time that blob is changed. You could however make use of blob's metadata and store file's creation date/time there.
I think you have to get the instance of the bob where you want to deploy the file
like :
AzCopy /Source:d:\processed /Dest:https://ContainerX.blob.core.windows.net/fec-data/Reports/ /DestKey:SomethingSomething== /S
Blob: Upload
Upload single file
AzCopy /Source:C:\myfolder/Dest:https://myaccount.blob.core.windows.net/mycontainer /DestKey:key /Pattern:"abc.txt"
If the specified destination container does not exist, AzCopy will create it and upload the file into it.
Upload single file to virtual directory
AzCopy /Source:C:\myfolder /Dest:https://myaccount.blob.core.windows.net/mycontainer/vd /DestKey:key /Pattern:abc.txt
If the specified virtual directory does not exist, AzCopy will upload the file to include the virtual directory in its name (e.g., vd/abc.txt in the example above).
please refer the link :https://learn.microsoft.com/en-us/azure/storage/storage-use-azcopy
Related
I've set up a logic app to move my new files on my FTP server to my azure storage container, which has blobs for my files. I found a way to create new folders using the storage explorer, but is there a way I can automate this using logic apps? For example, if a new folder is created in my FTP and files are added to it, I want to create a blob folder and move those files into that blob.
First of all, Azure blob storage doesn't support folders. There is only your storage account and a list of container containing blobs.
What you can do is to simulate a directory by adding a name that contains a slash, e. g. uploading the following file:
/myVirtualFolder/test.txt
Will upload the file to your desired container and tools like storage explorer will parse the slashes and display them as a folder:
But if you check the metadata for test.txt, you will see that the actual file name is /myVirtualFolder/test.txt:
So all you have to do is to upload all your files from your target directory to the container by adding the virtual directory to its name. You can`t and don't have to create a folder first.
I have an Azure storage account where I have created a folder to upload & download a file in it.I am also performing the rename operation on it e.g when I perform rename operation and upload the file into the blob all Blob metadata get updated successfully.
Please suggest the changes.
How to download the Azure blob content with the same name of File
As Gaurav Mantri said that you could specify the ContentDisposition property for your blob. Use the Azure Storage Explorer, you could quick set the ContentDisposition property as follows:
But when I downloading the image, the ContentDisposition seems not working at all. Then I found a similar issue, you need to set the DefaultServiceVersion for your blob storage service. And you need to write your code and set the DefaultServiceVersion, more details you could refer to here and choose your development language.
Test:
Additionally, if you upload/download your blob files by programming, you could refer to issue1 and issue2.
I am using LogicApp to import a set of files which are inside the directory(/devcontainer/sample1/abc.csv).
The problem here is that,I could not even located to the azure file from my LogicApp, I am getting the following error as:
verify that the path exists and does not contain the blob name.List Folder is not allowed on blobs.
Screenshots for reference
The problem here is that,I could not even located to the azure file from my LogicApp,
The file explorer will show all the contains and blobs when you choose blob path. And it will cache the data for a period of time to ensure the smoothness of the operation. If a blob is added to the container recently, it will not be seen and chosen from the file explorer. The workaround is by clicking the change connection link and using a new connection to retrieve the data.
Does your blob connection pointing to the correct storage account? one thing you can try to do is instead of providing the path try to browse the path so that you can what are the containers and the blobs that are present in the storage account that you are trying to access.
I want to create following folder structure on Azure:
mycontainer
-images
--2007
---img001.jpg
---img002.jpg
Now, one way is to use PUT Blob request and upload img001.jpg specifying the whole path as
PUT "mycontainer/images/2007/img001.jpg"
But, I want to first create the folders images and 2007 and then in a different request upload the blob img001.jpg.
Right now when I tried to doing this using PUT BLOB request:
StringToSign:
PUT
x-ms-blob-type:BlockBlob
x-ms-date:Tue, 07 Feb 2017 23:35:12 GMT
x-ms-version:2016-05-31
/account/mycontainer/images/
HTTP URL
sun.net.www.protocol.http.HttpURLConnection:http://account.blob.core.windows.net/mycontainer/images/
It is creating a folder but its not empty. By, default its creating an
empty blob file without name.
Now, a lot of people say we can't create a empty folder. But, then how come, we can make it using the azure portal as the browser must be sending some type of rest request to create the folder.
I think it has to do something with Content-Type i.e. x-ms-blob-content-type, which should be specified in order to tell azure that its a folder not a blob.
But, I am confused.
I want to first create the folders images and 2007 and then in a different request upload the blob img001.jpg
I agree with Brendan Green, currently, Azure blob storage just enable us to create virtual directory structure by naming blobs with path information in their names.
I think it has to do something with Content-Type i.e. x-ms-blob-content-type, which should be specified in order to tell azure that its a folder not a blob. But, I am confused.
You could check the description of Request Headers that could be set for Put Blob operation and you will find it does not support creating an empty folder by specifying some request headers.
Besides, as Gaurav Mantri said, if you really want to create an empty folder structure without content, you could try to use Azure File storage and it also enables us to use REST API to access Azure File storage. And the Create Directory operation cloud be used to create a new directory under the specified share or parent directory.
PUT https://myaccount.file.core.windows.net/myshare/myparentdirectorypath/mydirectory?restype=directory
This is not possible - the folder structure is virtual only.
See Get started with Azure Blob storage using .NET. You can only create a container, and everything else held in that container is a blob.
Excerpt:
As shown above, you can name blobs with path information in their
names. This creates a virtual directory structure that you can
organize and traverse as you would a traditional file system. Note
that the directory structure is virtual only - the only resources
available in Blob storage are containers and blobs.
I am trying to copy some blob files from one storage account to another one. I am using AzCopy in order to fulfill this goal.
The process works for copying files between containers within the same storage account, but not between different storage accounts.
The command I am issuing is:
AzCopy /Source:https://<storage_account1>.blob.core.windows.net/<container_name1>/<path_to_desired_blobs> /Dest:https://<storage_account2>.blob.core.windows.net/<container_name2>/<path_to_store>/ /SourceKey:<source_key> /DestKey:<dest_key> /Pattern:<some_pattern> /S
The error I am getting is the following:
The remote server returned an error: (400) Bad Request.
Could not verify the copy source within the specified time.
RequestId:
Time:2016-04-01T19:33:01.0527460Z
The only difference between the two storage accounts is that one is Standard, whereas the other one is Premium.
Any help will be appreciated!
From your description, you're trying to copy Block Blob from source account to Page Blob in destination account, which is not supported in Azure Storage Service and AzCopy.
To work around it, you can firstly use AzCopy to download the Block Blobs from source account to local file system, and then upload them from local file system to destination account with option /BlobType:Page (this option is only valid when uploading from local to blob).
Premium Storage only supports page blobs. Please confirm that you are copying page blobs from standard to premium storage account. Also, specify the BlobType parameter to "page" in order to copy the data as page blobs into destination premium storage account.
From the description, I am assuming your source blob is a block blob. Azure's "Async Copy Blob" process (which is used by AzCopy as the default method) preserves the blob type. That is, you cannot convert a blob type from Block to Page through async copy blob.
Instead, can you try AzCopy again with "/SyncCopy" option along with "/BlobType:page" parameter? That might help change the destination blob type to Page.
(If that doesn't work, only other solution would be to first download the blob, and then upload it with "/BlobType:page")