I have an Azure infrastructure:
2 HTTP Functions -> Event Hub -> 2 Functions -> Table Storage
(so two http functions sending messages to event hub, and two functions triggered by messages in Event Hub, one of them saving message in table storage)
The infrastructure is daily automatically created by Azure ARM templates, with the use of Azure CLI. I haven't changed the logic in recent two months but since beginning of April I have noticed the new, weird behaviour.
At the end of setting up, E2E tests are executed automatically. They are sending some message and after some time they check, if message are in table storage.
And here is the problem: since beginning of April these tests almost always fail! And I did not change anything in logic of function or template.json's for infrastructure.
It looks that Functions that should be triggered by Event Hub are not executed at all! I have already found a workaround for it - if I go to Azure portal and run these functions manually ("Run" button above code editor), then the functions finally starts to work!
Does anybody else encounter this problem?
Is there some way to automatically, directly run non-HTTP triggered function by e.g. Azure CLI or REST interface?
It seems that problem is already quite well known:
https://github.com/Azure/Azure-Functions/issues/210
I'm using currently workaround from this issue, i.e. calling Azure CLI's method to synchronize function triggers after creating infrastructure and zip pushing of functions:
az resource invoke-action --resource-group <resourceGrouName> --action syncfunctiontriggers --name <functionAppName> --resource-type Microsoft.Web/sites
Related
I get the following error in a pipeline that's first activity is to do a lookup on a storage container to get the contents of a file. When I test the connectionns, linked server, datasets or debug the pipeline I do not receive any errors. However when the pipeline is triggered by the storage event, it throws this error:
ErrorCode=AzureBlobCredentialMissing,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Please provide either connectionString or sasUri or serviceEndpoint to connect to Blob.,Source=Microsoft.DataTransfer.ClientLibrary,'
As per your scenario, where the debug is successful but the trigger runs failing. This make me assume that your dev changes have not been published which is why the trigger run fails. In simple terms the most recent published version of your linked service is different than that of your development version which haven't been published.
In case if you are using Source control then I would recommed following this tutorial for best practices - Automated publishing for continuous integration and delivery
If you are using CI-CD, then the issue might indeed cause by the DevOps pipeline not overriding the linked service parameters. Try redeploying the resource bye following below step and it should work as expected. (Linked service parameters had to be overwritten on the Azure resource template)
For example, if you have a linked service such as below:
Then you will still have to add below values into the overrideParameters section of the AzureResourceManagerTemplateDeployment task.
Is there a way for the Azure event grid to trigger when an AZ copy command completes?
We have clients which use az copy to transfer hundreds of files and sub folders into our Azure storage. The number of files is variable. And the az copy command is of a single root folder on their local containing those files and sub folders.
We want to raise an event grid notification when the az copy is complete and successful.
An alternative would be to have a second az copy command in a batch file that transfers a single flag file once the initial command is fully executed successfully. We would then monitor for this single file as the flag to proceed with further processing.
Perhaps if az copy cannot raise the event, then it can add a verification file signaling the end of the transfer?
You can have event grid notifications on an individual blob (or directory, when using ADLS). azcopy is essentially creating individual blobs, so you'd get individual notifications. Azure storage doesn't provide a transactional batch of uploads, so you can't get a single notifiction.
If you wanted a single notification, you'd have to manage this yourself. You mentioned a "flag" file, but you can also create custom topics, use an Azure Function, service bus message, etc. How you ultimately implement this is up to you (and your clients that are uploading content), but tl;dr no, you can't get a single completion event for a batch of uploads.
I have tried to reproduce in my environment get notification successfully
For event grid notification the endpoint which will receive notification in function app or logic apps...I created endpoint in function app
In your function app -> Function -> create ->select azure event grid trigger ->create
once created, in storage account create subscription as below
please make changes as below once you select endpoint by default it shows function in right side as below and confirm selection and create subscription
once azcopy is complete and files uploaded in container you will get a notification like below
I have an Event Grid Trigger Azure function deployed to a function app which is in an ASE. I have created an event grid subscription with this function as an end point. The purpose of this subscription is to call this function once a blob file is created.
When I placed the file in the location, the event is supposed to call the function but I see the event is undelivered.
How can I resolve the issue?
I found this in Azure documents :
" When your function app runs in either an App Service plan or an App Service Environment, you can use non-HTTP trigger functions. For your functions to get triggered correctly, you must be connected to a virtual network with access to the resource defined in the trigger connection. "
Maybe you can refer to this article.enter link description here
I have created an eventgrid subscription that writes the event to a queue whenever a file is created in blob. I have modified exiting EvenGridTrigger function to QueueTrigger function. Both storage account and function app are in same vnet.
I've set up a Azure Data Factory pipeline containing a copy activity. For testing purposes both source and sink are Azure Blob Storages.
I wan't to execute the pipeline as soon as a new file is created on the source Azure Blob Storage.
I've created a trigger of type BlovEventsTrigger. Blob path begins with has been set to //
I use Cloud Storage Explorer to upload files but it doesn't trigger my pipeline. To get an idea of what is wrong, how can I check if the event is fired? Any idea what could be wrong?
Thanks
Reiterating what others have stated:
Must be using a V2 Storage Account
Trigger name must only contain letters, numbers and the '-' character (this restriction will soon be removed)
Must have registered subscription with Event Grid resource provider (this will be done for you via the UX soon)
Trigger makes the following properties available #triggerBody().folderPath and #triggerBody().fileName. To use these in your pipeline your must map them to pipeline paramaters and use them as such: #pipeline().parameters.paramaetername.
Finally, based on your configuration setting blob path begins with to // will not match any blob event. The UX will actually show you an error message saying that that value is not valid. Please refer to the Event Based Trigger documentation for examples of valid configuration.
Please reference this. First, it needs to be a v2 storage. Second, you need register it with event grid.
https://social.msdn.microsoft.com/Forums/azure/en-US/db332ac9-2753-4a14-be5f-d23d60ff2164/azure-data-factorys-event-trigger-for-pipeline-not-working-for-blob-creation-deletion-most-of-the?forum=AzureDataFactory
There seems to be a bug with Blob storage trigger, if you have more than one trigger is allocated to the same blob container, none of the triggers will fire.
For some reasons (another bug, but this time in Data factories?), if you edit several times your trigger in the data factory windows, the data factory seems to loose track of the triggers it creates, and your single trigger may end up creating multiple duplicate triggers on the blob storage. This condition activates the first bug discussed above: the blob storage trigger doesn't trigger anymore.
To fix this, delete the duplicate triggers. For that, navigate to your blob storage resource in the Azure portal. Go to the Events blade. From there you'll see all the triggers that the data factories added to your blob storage. Delete the duplicates.
And now, on 20.06.2021, same for me: event trigger is not working, though when editing it's definition in DF, it shows all my files in folder, that matches. But when i add new file to that folder, nothing happens!
If you're creating your trigger via arm template, make sure you're aware of this bug. The "runtimeState" (aka "Activated") property of the trigger can only be set as "Stopped" via arm template. The trigger will need to be activated via powershell or the ADF portal.
An event grid resource provider needs to have been registered, within the specific azure subscription.
Also if you use Synapse Studio pipelines instead of Data Factory (like me) make sure the Data Factory resource provider is also registered.
Finally, the user should have both 'owner' and 'storage blob data contributor' on the storage account.
In this issue:
Azure Functions - Event Hub not triggering Functions
I described a problem with syncing event hub triggers and I managed to find a solution by simply invoking 'syncfunctiontriggers' action with Azure CLI:
az resource invoke-action --resource-group <resourceGrouName> --action syncfunctiontriggers --name <functionAppName> --resource-type Microsoft.Web/sites
Unfortunately this stopped working since 5th June and triggers are not fired on with messages getting into event hub - even if I call this command above. I need to go to portal and open function apps to sync them again but for me it is not a feasible solution.
I need to have an automated way of creating whole resource group, containing event hub, function apps, storages. At best with the use of Azure CLI.
Has anybody found a workaround for it or is this problem known already to Azure's team?
In the meantime I have found a workaround that is not entering the azure's portal. Simply make a call to trigger's function app page, e.g.:
curl -s https://<function-app-name>.azurewebsites.net > /dev/null
And after that if I run E2E tests, event hub triggers start to work. However, as with previous workarounds that I'd used, I don't how long it might be valid.