I have a custom handler written in Go running as an Azure Function. It has an endpoint with two methods:
POST /entities
PUT /entities
It was easy to make my application run as an Azure function: I added "enableForwardingHttpRequest": true to host.json, and it just works.
What I need to achieve: life happened and now I need to enqueue a message when my entities change, so it will trigger another function that uses a queueTrigger to perform some async stuff.
What I tried: The only way I found so far was to disable enableForwardingHttpRequest, and change all my endpoints to accept the Azure Function's raw JSON input and output, and then output a message in one of the fields (as documented here.
It sounds like a huge change to perform something simple... Is there a way I can enqueue a message without having to change all the way my application handles requests?
As per this Github document as of now custom handlers related to go lang in Azure functions having a bug and which need to be fixed.
I'm guessing this has been asked before, but I haven't been able to find the answer... Is there a way to look at a trigger to see which functions are being started by it?
We are building functions that will be run on different timed triggers. We'd like to be able to see which ones set to the hourly trigger or the daily trigger (as an example).
We were finally able to find an answer to this question... Inside your DataFactory...
Go to Manage
Select Triggers
View the code for the trigger
The associated functions are shown under the pipelines as pipelineReferences
Timer Trigger function is a non Http-Triggered Function & lets you run a function on a schedule based on the CORN Expression.
You can look at the CRON expression which was declared in the function.json file of a particular function.
You can also refer the below documentation to get more understanding about the corn expressions.
Alternatively, you can look at the logs of the function app based on the logs you validate which function is running at hourly basis & daily basis.
You can refer the below SO thread for more information about how to fetch the logs of a particular function under functionapp.
I'm trying to use Azure Durable Functions to implement a minimal server that accepts data via a POST and returns the same data via a GET - but I don't seem able to generate the GET response.
Is it simply not possible to return the response via a simple GET response? What I do NOT want to happen is:
GET issued
GET response returns with second URL
Client has to use second URL to get result.
I just want:
GET issues
GET response with requested data.
Possible?
I'm not really sure whether durable functions are meant to be used as a 'server'. Durable functions are part of 'serverless' and therefore I'm not sure whether it's possible (in a clean way).
To my knowledge durable functions are used to orchestrate long lasting processes. So for example handling the orchestration of a batch job. Using durable functions it's possible to create an 'Async HTTP API', to check upon the status of the processing a batch of items (durable function documentation). I've wrote a blogpost about durable functions, feel free to read it (https://www.luminis.eu/blog/azure-functions-for-your-long-lasting-logic/).
But as for your use case; I think you can create two separate Azure functions. One for posting your data, you can use an Azure Blob Storage output binding. Your second function can have a GET http trigger and depending on your data you can use an blob input binding. No need to use durable functions for it :)!
Not really a direct answer to your question, but hopefully a solution to your problem!
I've only started playing with Azure Logic Apps and already bumped into some problem.
The app that I'm trying to develop has a fairly simple logic: call the API, get the data and then save it to the Blob storage.
App Logic app
It works perfectly fine for a single request (https://example.com/rest/businessObject/objectName/details?fields=abc,dde). However, I would like to make multiple get requests to various objects getting different fields, i.e. change the objectName in the URI and fields values in the get request. Is it possible to parametrize the call using something like JSON object, iterate over it and make different requests in a single app instead of creating multiple apps for each request? And if it's possible then how?
Update
I've used JSON parsing (big thanks to the author of response in this thread). Now my app looks like this:
Seems to work fine:
The problem that I have now however is different - I can't save the body of the response as blob as I can't access it in the create blob. The only variable available for Blob content is 'Current item' which, of course, is a chunk of JSON used in the for loop. Any ideas if it's possible to save it to the storage and how to get it done?
You can do JSON parsing itself, use this example.
You can also do nesting, and also use variables and call different Logic App from another and that can be nested or loops or if-else.
If you have a fixed number of requests and not infinite then make those Logic apps, and using variables make use of Nested Logic App calls. If you have an in-definite (un-fixed) amount of API calls then a better idea would be to use Azure Durable Functions or even may be Azure Logic App as an orchestrator which then uses Azure Functions in nested calls.
My needs are following:
- Need to fetch data from a 3rd party API into SQL azure.
The API's will be queried everyday for incremental data and may require pagination as by default any API response will give only Top N records.
The API also needs an auth token to work, which is the first call before we start downloading data from endpoints.
Due to last two reasons, I've opted for Function App which will be triggered daily rather than data factory which can query web APIs.
Is there a better way to do this?
Also I am thinking of pushing all JSON into Blob store and then parsing data from the JSON into SQL Azure. Any recommendations?
How long does it take to call all of the pages? If it is under ten minutes, then my recommendation would be to build an Azure Function that queries the API and inserts the json data directly into a SQL database.
Azure Function
Azure functions are very cost effective. The first million execution are free. If it takes longer than ten, then have a look at durable functions. For handling pagination, we have plenty of examples. Your exact solution will depend on the API you are calling and the language you are using. Here is an example in C# using HttpClient. Here is one for Python using Requests. For both, the pattern is similar. Get the total number of pages from the API, set a variable to that value, and loop over the pages; Getting and saving your data in each iteration. If the API won't provide the max number of pages, then loop until you get an error. Protip: Make sure specify an upper bound for those loops. Also, if your API is flakey or has intermittent failures, consider using a graceful retry pattern such as exponential backoff.
Azure SQL Json Indexed Calculated Columns
You mentioned storing your data as json files into a storage container. Are you sure you need that? If so, then you could create an external table link between the storage container and the database. That has the advantage of not having the data take up any space in the database. However, if the json will fit in the database, I would highly recommend dropping that json right into the SQL database and leveraging indexed calculated columns to make querying the json extremely quick.
Using this pairing should provide incredible performance per penny value! Let us know what you end up using.
Maybe you can create a time task by SQL server Agent.
SQL server Agent--new job--Steps--new step:
In the Command, put in your Import JSON documents from Azure Blob Storage sql statemanets for example.
Schedules--new schedule:
Set Execution time.
But I think Azure function is better for you to do this.Azure Functions is a solution for easily running small pieces of code, or "functions," in the cloud. You can write just the code you need for the problem at hand, without worrying about a whole application or the infrastructure to run it. Functions can make development even more productive, and you can use your development language of choice, such as C#, F#, Node.js, Java, or PHP.
It is more intuitive and efficient.
Hope this helps.
If you could set the default top N values in your api, then you could use web activity in azure data factory to call your rest api to get the response data.Then configure the response data as input of copy activity(#activity('ActivityName').output) and the sql database as output. Please see this thread :Use output from Web Activity call as variable.
The web activity support authentication properties for your access token.
Also I am thinking of pushing all JSON into Blob store and then
parsing data from the JSON into SQL Azure. Any recommendations?
Well,if you could dump the data into blob storage,then azure stream analytics is the perfect choice for you.
You could run the daily job to select or parse the json data with asa sql ,then dump the data into sql database.Please see this official sample.
One thing to consider for scale would be to parallelize both the query and the processing. If there is no ordering requirement, or if processing all records would take longer than the 10 minute function timeout. Or if you want to do some tweaking/transformation of the data in-flight, or if you have different destinations for different types of data. Or if you want to be insulated from a failure - e.g., your function fails halfway through processing and you don't want to re-query the API. Or you get data a different way and want to start processing at a specific step in the process (rather than running from the entry point). All sorts of reasons.
I'll caveat here to say that the best degree of parallelism vs complexity is largely up to your comfort level and requirements. The example below is somewhat of an 'extreme' example of decomposing the process into discrete steps and using a function for each one; in some cases it may not make sense to split specific steps and combine them into a single one. Durable Functions also help make orchestration of this potentially easier.
A timer-driven function that queries the API to understand the depth of pages required, or queues up additional pages to a second function that actually makes the paged API call
That function then queries the API, and writes to a scratch area (like Blob) or drops each row into a queue to be written/processed (e.g., something like a storage queue, since they're cheap and fast, or a Service Bus queue if multiple parties are interested (e.g., pub/sub)
If writing to scratch blob, a blob-triggered function reads the blob and queues up individual writes to a queue (e.g., a storage queue, since a storage queue would be cheap and fast for something like this)
Another queue-triggered function actually handles writing the individual rows to the next system in line, SQL or whatever.
You'll get some parallelization out of that, plus the ability to start from any step in the process, with a correctly-formatted message. If your processors encounter bad data, things like poison queues/dead letter queues would help with exception cases, so instead of your entire process dying, you can manually remediate the bad data.