Is there a way for the Azure event grid to trigger when an AZ copy command completes?
We have clients which use az copy to transfer hundreds of files and sub folders into our Azure storage. The number of files is variable. And the az copy command is of a single root folder on their local containing those files and sub folders.
We want to raise an event grid notification when the az copy is complete and successful.
An alternative would be to have a second az copy command in a batch file that transfers a single flag file once the initial command is fully executed successfully. We would then monitor for this single file as the flag to proceed with further processing.
Perhaps if az copy cannot raise the event, then it can add a verification file signaling the end of the transfer?
You can have event grid notifications on an individual blob (or directory, when using ADLS). azcopy is essentially creating individual blobs, so you'd get individual notifications. Azure storage doesn't provide a transactional batch of uploads, so you can't get a single notifiction.
If you wanted a single notification, you'd have to manage this yourself. You mentioned a "flag" file, but you can also create custom topics, use an Azure Function, service bus message, etc. How you ultimately implement this is up to you (and your clients that are uploading content), but tl;dr no, you can't get a single completion event for a batch of uploads.
I have tried to reproduce in my environment get notification successfully
For event grid notification the endpoint which will receive notification in function app or logic apps...I created endpoint in function app
In your function app -> Function -> create ->select azure event grid trigger ->create
once created, in storage account create subscription as below
please make changes as below once you select endpoint by default it shows function in right side as below and confirm selection and create subscription
once azcopy is complete and files uploaded in container you will get a notification like below
Related
I have an azure storage account.
Inside the container, with a client-specific folder structure, every morning, some files get pushed.
I have a function app which processes and converts these files and calls some external service to work upon on these processed files.
I have got a file-share as well, which is basically mounted on a vm.
The external service, after processing the files (#3), generates the resultant success/failure files inside this file-share(#4).
Now the ask is:
Create a simple dashboard which will monitor the storage account(and in effect the container and the file-shares),it should capture & show basic information's, and should look like below table structure(with 3 simple variations of data):
FileName|ReceivedDateTime|NumberOfRecords
Original_file.csv20221011 5:21 AM|10
Original_file_Success.csv20221011 5:31 AM|9
Original_file_Failure.csv20221011 5:32 AM|1
In here the first record is captured from the Container and the second and third - both are generated in the file-share.
Also, whenever a new failure file is generated, i.e., Original file_Failure, it should send email with a predefines template adding the file name to a predefined recipient list.
Any guidance on the azure service to use?
I have seen Azure Monitor,workbook and other stuffs, but I feel that would be an overkill for such simple requirement.
Thanks in advance.
I have the below scenario:
I have a Logic App, which gets triggered once in every day(24hours).
It basically looks at a SFTP location, if there is file dropped in there, pulls it and pushes it into a BLOB storage and then deletes it from the source(SFTP).
I need to trigger an email in the events of:
If the Trigger is "Skipped", i.e. it ran but could not find any file in the SFTP.
If it failed to Upload to the BLOB Storage.
Is it possible to enable Email Trigger in the above scenarios?(1&2)
Any guidance will be appreciated as I am new in the IAC space.
Thanks in advance.
Firstly, you can List the files in FTP and pass the name to get the content of the files using List files in folder and Get file content actions of SFTP connector.
If the Trigger is "Skipped", i.e. it ran but could not find any file in the SFTP.
For this, in the next step you can use a condition action to check if the file has been uploaded for that day by comparing the last modified time with the current date. If yes, then create a file in the blob storage with the file contents from Get file content step. Below is the flow of my logic app.
If it failed to Upload to the BLOB Storage.
For this you can create another condition action and check if the file is been created or not by using actions('Create_blob_(V2)')['outputs']['statusCode']. Below is the complete code of my Logic app
I am trying to copy the databricks logs from one folder to another, Since I am sending databricks logs to storage account which is append blob. My objective as any new blob/any files get appended I need to run the copy activity.
I tired storage events trigger but it is not running if any logs get appended to the same files. Is there any way to run the pipeline immediately if any files appended or new folder dd/mm/yyy format get created.
Thanks
Anuj gupta
There is no out-of-the-box method to trigger when a blob is appended, there is a similar ask here, you can log a more precise one to get an official response.
Or you can use Create a custom event trigger to run a pipeline in Azure Data Factory with Azure Blob Storage as an Event Grid source where event Microsoft.Storage.BlobCreated is "triggered when a blob is created or replaced." (Append Block succeeds only if the blob already exists.)
Also, perhaps with Microsoft.Storage.BlobRenamed, Microsoft.Storage.DirectoryCreated & Microsoft.Storage.DirectoryRenamed
Azure Function Storage Account Blob Container Trigger
In one of our use case, i am looking for Azure function trigger for any activity in Storage account containers with following conditions
Container with a specific naming convention (name like xxxx-input)
It should automatically detect if a new container(with specific naming convention) is created
Currently, the following events are supported at the moment, per the documentation:
BlobCreated
BlobDeleted
BlobRenamed
DirectoryCreated(Data lake Gen2)
DirectoryRenamed(Data lake Gen2)
DirectoryDeleted(Data lake Gen2)
This means that it is not possible to create such event, but you can try to change the approach(if feasible for your use-case) from 'push' to 'pull'.
I suggest to write a time-triggered function that checks whether container with the given schemes were created. You can leverage the Blob Storage v12 SDK for this task, and get list of the containers.
Save the list to some database(for example CosmosDB), and every time the function gets triggered, you can compare the current state, with the last saved state from the db.
If there is a difference, you can push the message to the EventHub, that triggers another function that actually reacts on this 'new event-type'.
you should use the Azure Event Grid subscribing to the Resource group of your storage account and use for example, the advanced filtering for
"operationName":"Microsoft.Storage/storageAccounts/blobServices/containers/write",
"subject":"/subscriptions/<yourId>/resourcegroups/<yourRG>/providers/Microsoft.Storage/storageAccounts/<youraccount>/blobServices/default/containers/xxxx-input",
"eventType":"Microsoft.Resources.ResourceWriteSuccess",
I've set up a Azure Data Factory pipeline containing a copy activity. For testing purposes both source and sink are Azure Blob Storages.
I wan't to execute the pipeline as soon as a new file is created on the source Azure Blob Storage.
I've created a trigger of type BlovEventsTrigger. Blob path begins with has been set to //
I use Cloud Storage Explorer to upload files but it doesn't trigger my pipeline. To get an idea of what is wrong, how can I check if the event is fired? Any idea what could be wrong?
Thanks
Reiterating what others have stated:
Must be using a V2 Storage Account
Trigger name must only contain letters, numbers and the '-' character (this restriction will soon be removed)
Must have registered subscription with Event Grid resource provider (this will be done for you via the UX soon)
Trigger makes the following properties available #triggerBody().folderPath and #triggerBody().fileName. To use these in your pipeline your must map them to pipeline paramaters and use them as such: #pipeline().parameters.paramaetername.
Finally, based on your configuration setting blob path begins with to // will not match any blob event. The UX will actually show you an error message saying that that value is not valid. Please refer to the Event Based Trigger documentation for examples of valid configuration.
Please reference this. First, it needs to be a v2 storage. Second, you need register it with event grid.
https://social.msdn.microsoft.com/Forums/azure/en-US/db332ac9-2753-4a14-be5f-d23d60ff2164/azure-data-factorys-event-trigger-for-pipeline-not-working-for-blob-creation-deletion-most-of-the?forum=AzureDataFactory
There seems to be a bug with Blob storage trigger, if you have more than one trigger is allocated to the same blob container, none of the triggers will fire.
For some reasons (another bug, but this time in Data factories?), if you edit several times your trigger in the data factory windows, the data factory seems to loose track of the triggers it creates, and your single trigger may end up creating multiple duplicate triggers on the blob storage. This condition activates the first bug discussed above: the blob storage trigger doesn't trigger anymore.
To fix this, delete the duplicate triggers. For that, navigate to your blob storage resource in the Azure portal. Go to the Events blade. From there you'll see all the triggers that the data factories added to your blob storage. Delete the duplicates.
And now, on 20.06.2021, same for me: event trigger is not working, though when editing it's definition in DF, it shows all my files in folder, that matches. But when i add new file to that folder, nothing happens!
If you're creating your trigger via arm template, make sure you're aware of this bug. The "runtimeState" (aka "Activated") property of the trigger can only be set as "Stopped" via arm template. The trigger will need to be activated via powershell or the ADF portal.
An event grid resource provider needs to have been registered, within the specific azure subscription.
Also if you use Synapse Studio pipelines instead of Data Factory (like me) make sure the Data Factory resource provider is also registered.
Finally, the user should have both 'owner' and 'storage blob data contributor' on the storage account.