I've set up a logic app to move my new files on my FTP server to my azure storage container, which has blobs for my files. I found a way to create new folders using the storage explorer, but is there a way I can automate this using logic apps? For example, if a new folder is created in my FTP and files are added to it, I want to create a blob folder and move those files into that blob.
First of all, Azure blob storage doesn't support folders. There is only your storage account and a list of container containing blobs.
What you can do is to simulate a directory by adding a name that contains a slash, e. g. uploading the following file:
/myVirtualFolder/test.txt
Will upload the file to your desired container and tools like storage explorer will parse the slashes and display them as a folder:
But if you check the metadata for test.txt, you will see that the actual file name is /myVirtualFolder/test.txt:
So all you have to do is to upload all your files from your target directory to the container by adding the virtual directory to its name. You can`t and don't have to create a folder first.
Related
I have created an Azure Logic app that replicates the file structure in Azure blob storage. It works fine until I add a folder and files under the folder in SharePoint. When I try to read the contents through "Get file content", I get the error "BadRequest" and in the output Body, I get
status : 400
message : Route did not match\r\nclientRequestID
Is this error because of the folder path or maybe because of audio files in the folder?
Second issue - How can recursive functionality be achieved in the logic app, so that if its folder is within the folder then its read all the files and store them in the same hierarchy in the blob container?
You are receiving this because of the folder path, I assume you are trying to copy the blob according to its path and hence you receive that error.
Second issue - How can recursive functionality be achieved in the logic app, so that if its folder is within the folder then its read all the files and store them in the same hierarchy in the blob container?
you can use for-each loop in order to list all the files which are present inside subfolders too. Below is my logic app where I'm trying to create folders according to the hierarchical structure present in SharePoint.
Below is the Expression in Extracting folder excluding file name to create sub folders in blob storage
slice(outputs('Path_of_subfolder'),0,lastIndexOf(outputs('Path_of_subfolder'),'/'))
Below is the Expression in Path of subfolder
#{items('For_each_3')?['{FullPath}']}
I am using LogicApp to import a set of files which are inside the directory(/devcontainer/sample1/abc.csv).
The problem here is that,I could not even located to the azure file from my LogicApp, I am getting the following error as:
verify that the path exists and does not contain the blob name.List Folder is not allowed on blobs.
Screenshots for reference
The problem here is that,I could not even located to the azure file from my LogicApp,
The file explorer will show all the contains and blobs when you choose blob path. And it will cache the data for a period of time to ensure the smoothness of the operation. If a blob is added to the container recently, it will not be seen and chosen from the file explorer. The workaround is by clicking the change connection link and using a new connection to retrieve the data.
Does your blob connection pointing to the correct storage account? one thing you can try to do is instead of providing the path try to browse the path so that you can what are the containers and the blobs that are present in the storage account that you are trying to access.
I want to create following folder structure on Azure:
mycontainer
-images
--2007
---img001.jpg
---img002.jpg
Now, one way is to use PUT Blob request and upload img001.jpg specifying the whole path as
PUT "mycontainer/images/2007/img001.jpg"
But, I want to first create the folders images and 2007 and then in a different request upload the blob img001.jpg.
Right now when I tried to doing this using PUT BLOB request:
StringToSign:
PUT
x-ms-blob-type:BlockBlob
x-ms-date:Tue, 07 Feb 2017 23:35:12 GMT
x-ms-version:2016-05-31
/account/mycontainer/images/
HTTP URL
sun.net.www.protocol.http.HttpURLConnection:http://account.blob.core.windows.net/mycontainer/images/
It is creating a folder but its not empty. By, default its creating an
empty blob file without name.
Now, a lot of people say we can't create a empty folder. But, then how come, we can make it using the azure portal as the browser must be sending some type of rest request to create the folder.
I think it has to do something with Content-Type i.e. x-ms-blob-content-type, which should be specified in order to tell azure that its a folder not a blob.
But, I am confused.
I want to first create the folders images and 2007 and then in a different request upload the blob img001.jpg
I agree with Brendan Green, currently, Azure blob storage just enable us to create virtual directory structure by naming blobs with path information in their names.
I think it has to do something with Content-Type i.e. x-ms-blob-content-type, which should be specified in order to tell azure that its a folder not a blob. But, I am confused.
You could check the description of Request Headers that could be set for Put Blob operation and you will find it does not support creating an empty folder by specifying some request headers.
Besides, as Gaurav Mantri said, if you really want to create an empty folder structure without content, you could try to use Azure File storage and it also enables us to use REST API to access Azure File storage. And the Create Directory operation cloud be used to create a new directory under the specified share or parent directory.
PUT https://myaccount.file.core.windows.net/myshare/myparentdirectorypath/mydirectory?restype=directory
This is not possible - the folder structure is virtual only.
See Get started with Azure Blob storage using .NET. You can only create a container, and everything else held in that container is a blob.
Excerpt:
As shown above, you can name blobs with path information in their
names. This creates a virtual directory structure that you can
organize and traverse as you would a traditional file system. Note
that the directory structure is virtual only - the only resources
available in Blob storage are containers and blobs.
I'm attempting to upload 550K files from my local hard drive to Azure Blob Storage using the following command (AzCopy 5.1.1) -
AzCopy /Source:d:\processed /Dest:https://ContainerX.file.core.windows.net/fec-data/Reports/ /DestKey:SomethingSomething== /S
It starts churning right away.
But it's actually creating a new Azure File Storage folder called fec-data/reports rather than creating new blobs in the Azure Blob folder fec-data/reports I've already created.
What am I missing?
Also, is there anyway to keep the date created (or similar) values of the old files?
Thanks,
But it's actually creating a new Azure File Storage folder called
fec-data/reports rather than creating new blobs in the Azure Blob
folder fec-data/reports I've already created.
What am I missing?
The reason you're seeing this behavior is because you're uploading to File storage instead of Blob storage. To upload the files to Blob storage, you need to specify blob service endpoint (blob.core.windows.net). So your command would be:
AzCopy /Source:d:\processed /Dest:https://ContainerX.blob.core.windows.net/fec-data/Reports/ /DestKey:SomethingSomething== /S
Also, is there anyway to keep the date created (or similar) values of
the old files?
Assuming you want to keep the date created of the blob same as that of the desktop file, then it is not possible. Blob's Last Modified Date/Time is a system property that gets assigned when a blob is created and is updated every time that blob is changed. You could however make use of blob's metadata and store file's creation date/time there.
I think you have to get the instance of the bob where you want to deploy the file
like :
AzCopy /Source:d:\processed /Dest:https://ContainerX.blob.core.windows.net/fec-data/Reports/ /DestKey:SomethingSomething== /S
Blob: Upload
Upload single file
AzCopy /Source:C:\myfolder/Dest:https://myaccount.blob.core.windows.net/mycontainer /DestKey:key /Pattern:"abc.txt"
If the specified destination container does not exist, AzCopy will create it and upload the file into it.
Upload single file to virtual directory
AzCopy /Source:C:\myfolder /Dest:https://myaccount.blob.core.windows.net/mycontainer/vd /DestKey:key /Pattern:abc.txt
If the specified virtual directory does not exist, AzCopy will upload the file to include the virtual directory in its name (e.g., vd/abc.txt in the example above).
please refer the link :https://learn.microsoft.com/en-us/azure/storage/storage-use-azcopy
How to create sub directory in Windows Azure blob container?
I know we can use blob name like "content/2015/images/mypicture.jpg".
But we need to change parent directory name. Sub directory may contain many files. Change every blob name in sub directory will take time. How to do this?
I have read below links. But they don't mention how to handle rename parent directory. Thanks
Windows Azure: How to create sub directory in a blob container
How to create a sub container in azure storage location
But we need to change parent directory name. Sub directory may contain
many files. Change every blob name in sub directory will take time.
How to do this?
So, there are two things in blob storage:
As you mentioned, there is no concept of folders there. To create an illusion of folders, you prefix the blob name with the folder you want.
Blob storage does not support rename functionality natively. In order to accomplish rename, you would need to perform 2 operations on a blob: Copy+Delete.
Here's how you can rename a folder in blob storage:
First, you would need to list all blobs in that folder. You can use ListBlobs method and pass in the name of the folder as prefix parameter. See my answer here: How to load list of Azure blob files recursively?.
Next, you need to copy these blobs since you're just renaming the folder, you would basically create a new name of the blob by replacing the old folder name with new folder name.
Once the blobs are copied, you can again iterate over your blobs and delete them one by one.
You can also combine step 2 and 3 and do copy + delete operation on each blob. So you iterate over your list of blobs, first copy it with new name, delete it and then move on to the next blobs.
The bottom line is that you would need to do Copy+Delete on each blob in the folder to rename that folder.