Im learning logic apps and im trying to create a simple flow from azure blob storage, perform a liguid parsing and then save parsed file to another blob container.
How should it work:
1. Whenever new file is added to blob container ("from") [containing xml files]
2.Liquid action takes place (XML -> JSON)
3.New file .json is saved to blob container ("too") :)
What i have learned:
1. I manage to write a liguid template for xml files - tested - working
2. I know how to copy file between blob containers - tested - working
For each loop:
https://i.imgur.com/ImaT3tf.jpg "FE loop"
Completed:
https://i.imgur.com/g6M9eLJ.jpg "Completed..."
Current LA:
https://i.imgur.com/ImaT3tf.jpg "Current"
What I dont know how to do:
1. How to "insert" current file content in for each into liquid action? It looks like logic apps is skipping that step?
The main problem is you could not use Current item as the xml content, you need to get the content with Get blob content action in For_each, then parse xml to json. After this create the blob in another container with json value.
You could refer to my workflow.
Related
I have a blob storage that has 2 Containers called input and output. when a file gets uploaded to input then a Function app (Blobtrigger) would work on it and save the result in output folder.
right now i need to trigger a workflow in Azure logic app. i didn't created any event grid outside of this workflow and now i'm trying to trigger it when a file get's uploaded (Created) in the output container.
However my app won't trigger. what should i do?
I have reproduced in my environment and triggered an event when blob get uploaded and it got triggered:
Please find the below approach to fix your issue :
Then uploaded a blob like below:
Output:
EDIT:
I too Uploaded into SubFolder:
Then in Ouput Subfolder:
i solved it.
Make sure your Storage Account is Version 2.(it's really important check it)
mine was V1 so i had to change it here:
use this as a filter for your specific Container. (for more Info Check Microsoft Docs)
/blobServices/default/containers/MyContainer/
in my Case it would be:
/blobServices/default/containers/output/
I am accessing a website that allows me to download CSV file. I would like to store the CSV file directly to the blob container. I know that one way is to download the file locally and then upload the file, but I would like to skip the step of downloading the file locally. Is there a way in which I could achieve this.
i tried the following:
block_blob_service.create_blob_from_path('containername','blobname','https://*****.blob.core.windows.net/containername/FlightStats',content_settings=ContentSettings(content_type='application/CSV'))
but I keep getting errors stating path is not found.
Any help is appreciated. Thanks!
The file_path in create_blob_from_path is the path of your local file, looks like "C:\xxx\xxx". This path('https://*****.blob.core.windows.net/containername/FlightStats') is Blob URL.
You could download your file to byte array or stream, then use create_blob_from_bytes or create_blob_from_stream method.
Other answer uses the so called "Azure SDK for Python legacy".
I recommend that if it's fresh implementation then use Gen2 Storage Account (instead of Gen1 or Blob storage).
For Gen2 storage account, see example here:
from azure.storage.filedatalake import DataLakeFileClient
data = b"abc"
file = DataLakeFileClient.from_connection_string("my_connection_string",
file_system_name="myfilesystem", file_path="myfile")
file.append_data(data, offset=0, length=len(data))
file.flush_data(len(data))
It's painful, if you're appending multiple times then you'll have to keep track of offset on client side.
Hi I am working in Azure Logic app. I am trying to get multiple files from azure data Lake gen v2 and attach these multiple files in an email. As a first step I have added http request and I am giving required information along with file path. It works fine for one file. but I am trying to input folder path and inside that folder, all the files I want to get and attach in email.
Logic app Flow Diagram
Added sample screenshot for attachment
tried to add attchment
In the above diagram, Get blob content step which works fine for one file but I am finding difficult to attach multiple files in email. Can some one help me to figure out the solution. Any help would be appreciated. Thank you
You can use List blobs action to list all blobs in the folder you want:
Then you can define a variable to append the attachments array.
Use For Each to loop the blobs from List Blobs action. Within For Each you can use Get blob content to get blob content, and then use Append to array variable to append attachments.
The expressions of Path, DisplayName and File Content are as follows:
Path : items('For_each')?['Path']
DisplayName : items('For_each')?['DisplayName']
File Content : body('Get_blob_content')
Finally, please fill in the attachment in the email:
==========================update===================
If you send an email with 400 response, please use expression in Append to array variable as below:
base64(body('Get_blob_content'))
I have multiple JSON format files which is being pushed to the Azure storage account under a specific container. There are n number of files in the container.
And 4 to 8 nodes which will be accessing the Azure storage container to downloaded the files locally, the download code is written in java.
Since there are n number of files and multiple file accessing the container at the same time, how to avoid the situation that the same file is downloaded by another server?
Example:
Azure container has 1.json, 2.json, 3.json, etc which are > 35 MB size.
batch-process-node1 -> starts downloading 1.json
batch-process-node2 -> starts downloading 2.json
batch-process-node3 -> should not start downloading the 1.json
Is there any logic to be built for each node which has the java process to download the file uniquely?
Is there any setting that can be set in the Azure storage container?
--
Trying to use the Camel Azure-bolb component, using the block blob (blobType).
New to Azure storage blob, any help is appreciated.
Since we are already using Apache camel in the code, we tried to use camel azure-blob component to address the issue. Below is the approach we used, still the race condition is acceptable for our scenario.
Camel route started with timer consumer, and producer to get the list of blob from container using below endpoint,
azure-blob://<account>/<container>?credentials=#storagecredentials&blobType=blockBlob&operation=listBlobs
Note: storagecredential is a bean of type StorageCredentialsAccountAndKey class.
Created a java class implementing the Processor of camel, and in process() method, using the exchange.getIn().getBody() => which provides an iterable object with has ListBlobItem.
first i set the meta data of the blob using below endpoint
azure-blob://<account>/<container>/*<blobName>*?credentials=#storagecredentials&blobType=blockBlob&operation=updateBlockBlob&blobMetadata=#blobMetaData1
Note: blobMetaData1 is bean created in the context file.
<util:map id="blobMetaData1" map-class="java.util.HashMap">
<entry key="someKey" value="someValue"/>
</util:map>
Key thing: In this class process method
validate the metadata is being set or not, if set then the
process is already picked the blob. so it won't be picked again
assuming if the process executed in different server.
got the blob name from the ListBlobItem individual blob item.
using getURI() and forming the endpoint within this processor class.
in order to invoke the custom endpoint, used to set it an customer
header value of In message.
using the recipientList camel option which invokes the metadata endpoint to update the specific blob.
Then used another processor to form the download blob endpoint
azure-blob://<account>/<container>/*<blobName>*?credentials=#storagecredentials&blobType=blockBlob&operation=getBlob
and using the recipientList to get the processor endpoint from message header.
finally forming another delete endpoint which will delete once its downloaded.
I've created a file system abstraction, where I store files with a relative path, e.g /uploads/images/img1.jpg.
These can then be saved both on local file system (relative to folder), or Azure. Then, I can also ask a method to give me the url to access that relative path.
In Azure, currently this is being done similar to the below:
public string GetWebPathForRelativePathOnUserContentStorage(string relativeFileFullPath)
{
var container = getCloudBlobContainer();
CloudBlockBlob blob = container.GetBlockBlobReference(relativeFileFullPath);
return blob.Uri.ToString();
}
On a normal website, there might be say 40 images in one page - So this get's called like 40 times. Is this first of all slow? I've noticed there is a particular pattern in the generated URL:
https://[storageAccountName].blob.core.windows.net/[container_name]/[relative_path]
Can I safely generate that URL without using the Azure storage API?
On a normal website, there might be say 40 images in one page - So
this get's called like 40 times. Is this first of all slow?
Not at all. The code you wrote above does not make any calls to storage. It just creates an instance of CloudBlockBlob object. If you were using GetBlockBlobReferenceFromServer method, then it would have been a different story because that method makes a call to storage.
`I've noticed there is a particular pattern in the generated URL:
_https://[storageAccountName].blob.core.windows.net/[container_name]/[relative_path]
Can I safely generate that URL without using the Azure storage API?
Absolutely yes. Assuming you're using just standard stuff that would be perfectly fine. Non standard stuff would include things like using a custom domain for your blob storage or connecting to geo-secondary location of your storage account.