I´ve created an Azure Synapse Analytics Pipeline that must be triggered by the creation of a file within a Azure Gen2 storage account.
Somehow the blob creation event (i.e. when I upload the file in the corresponding container and folder) doesn´t fire anything and the pipeline does not start. I´ve registered the Microsoft.EventGrid and Microsoft.Synapse resource providers in the subscription, as suggested by the Microsoft official documentation.
Am I missing anything? As far as I know, and according to the Microsoft documentation and the many tutorials I've read, I don´t need any Event Topic/Event subscription...
Can you please check the content type of the file :
usually when that is blank, event trigger is not initiated
I tried to reproduce your scenario in my environment, and it works for me (i.e., when I upload the file in the corresponding container and folder). Let me share my implementation and then you can compare with yours.
This is the setup for the trigger
The trigger is firing as expected.
Files uploaded date time
Trigger firing date time
I still didn´t figure out what is not working, so I implemented a workaround: a simple ADF pipeline looping for files in the landing zone. The pipeline is associated with a normal schedule trigger (it runs 3 times a day) and it calls in turn the pipeline I originally wanted to be triggered by the file creation trigger.
Related
I was looking for other question which could be similar to mine, but wasn't able to find something like this.
Question: Pictures are uploaded to a platform, which I can access using an API, sometimes 3 times a day, something once a week. Instead of running a scheduled pipeline, we want to trigger the pipeline when new data is uploaded to the platform. I've used a timer function (every 5 minutes) at the moment, but cannot find how to trigger a specific pipeline. So how can I do this?
Good to know: The pipeline starts a job in Azure ML in a compute cluster.
Does anyone know the solution to my question or know where I can find more information about this?
If your Azure ML job (or pipeline) can send the picture to Azure Blob Storage, then your Azure Function can be triggered based whenever a new image is uploaded. Here are tutorials on how to do this in C# and Javascript.
I've successfully integrated Snowpipe with a container inside the Azure storage and loaded data into my target table, but now I can't exactly figure out how does Snowpipe actually works. Also, please let me know if there is already a good resource that answers this question, I'd be very grateful.
In my example, I tested a Snowpipe mechanism that uses cloud messaging. So, from my understanding, when a file is uploaded into an Azure container, Azure Event Grid sends an event message to an Azure queue, from which Snowpipe is notified that a new file is uploaded into the container. Then, Snowpipe in the background starts its loading process and imports the data into a target table.
If this is correct, I don't understand how does Azure queue informs Snowpipe about uploaded files. Is this connected to the "notification integration" inside Snowflake? Also, I don't understand what does it mean when they say on the Snowflake page that "Snowpipe copies the files into a queue, from which they are loaded into the target table...". Is this an Azure queue or some Snowflake queue?
I hope this question makes sense, any help or detailed explanation of the whole process is appreciated!
You've pretty much nailed it. to answer your specific questions... (and don't feel bad about them, this is definitely confusing)
how does Azure queue informs Snowpipe about uploaded files? Is this connected to the "notification integration" inside Snowflake?
Yes, this is the notification integration. But Azure is not "informing" the Snowpipe, it's the other way around. The Azure queue creates a notification that various other applications can subscribe to (this has no awareness of Snowflake). The notification integration on the snowflake side is snowflake's way to integrate with these external notifications
Snowpipe's queueing
Once snowflake recieves one of these notifications it puts that notification into a snowflake-side queue (or according to that page, the file itself. I was surprised by this, but the end result is the same). Snowpipes are wired up to that notification integration (as part of the create statement). The files are directed to the appropriate snowpipe based on the information in the "Stage" (also as part of the pipe create statement. I'm actually not certain if this part is a push or a pull). Then it runs the COPY INTO on that file.
I have an Azure Logic app that dynamically gets Blob contents from my azure storage account and sends an email with the attachment. I want to set a schedule for my logic app to run once a week.
Any idea how I can achieve this?
Here's my current workflow:
It depends on what you are trying to do. If you want to get an email every time your blob is updated, your current Logic App is the way to go. If you change the trigger to a Reccurrence trigger as Rob Ert stated than you could potentially lose updates (the blob could have many updates in a week). If you don't care about the individual updates, then Reccurrence is the proper trigger.
I think you are looking for Recurrence trigger's.
It's possible to set something like time triggers from regular Azure Functions.
Here's instruction how to create one in your logic app.
https://learn.microsoft.com/en-us/azure/connectors/connectors-native-recurrence
I am referring to this feedback:
Azure Storage Blob Trigger to fire when files added in Sub Folders
I have a Azure Logic App that is fired every time a blob is added in a container. This works fine when all the documents are at the root of the container.
Inside my container, I have a dynamic number of (virtual) sub-folders.
When I add a new document in a subfolder (path = mysubfolder/mynewdocument.txt), the logic app is not triggered.
This does not really make sense for me as sub-folders in the blob container are virtual. Does anyone find a workaround except putting all the files at the root level ?
I've opened an issue on Github:
https://github.com/Azure/logicapps/issues/20
This is the expected behavior. None of the Logic App Triggers that work with 'files' support subfolders.
This has been the case with BizTalk Server as well since 2000 so I would not expect a change anytime soon :(.
Please create or vote on a User Voice for this issue: User Voice - Logic Apps
This does not really make sense for me as sub-folders in the blob container are virtual. Does anyone find a workaround except putting all the files at the root level ?
I also can reproduce it on my side. I recommend that you could use the Azure function app blob trigger to instead of Azure Logic App blob trigger. Azure blob trigger function could be fired when you add a new document in a subfolder(virtual).
At the time I was developing this feature (early 2018), EventGrid was still in preview so I've ended up using Azure Function - Blob trigger.
I would definitely use EventGrid - Blob Event now and it works for Logic App / Function App or with any Http endpoint.
I have a time consuming custom activity running in a Azure data factory pipeline.
It copies files from Blob to FTP server recursively.
The entire activity take 3-4 hours based on the number of files in the folder.
But when I am running the pipeline, it shows in progress 0%.
How update pipeline progress from custom activity?
In short, I doubt you will be able to. The services are very discounted from each other.
You might be better off writing out to the Azure generic activity log and monitoring directly from the custom activity method. This is an assumption though.
Hope this helps.