Azure Python function Service Bus outbound attribute - azure

I have a azure python function : It gets triggered by HTTP , responds with HTTP response and puts the message in the Azure Service Bus Queue.
Function.json: for outbound Azure Service bus
{
"type": "serviceBus",
"connection": "ServiceBus",
"name": "outputMessage",
"queueName": "testqueue",
"accessRights": "send",
"direction": "out"
}
I have function as
def main(req: func.HttpRequest, outputMessage: func.Out[func.ServiceBusMessage]) -> str:
I get below error:
Result: Failure
Exception: FunctionLoadError: cannot load the HttpTrg function: type of outputMessage binding in function.json "serviceBus" does not match its Python annotation "ServiceBusMessage"
Question:
1. What should be the python annotation for Azure Service Bus outbound ?
def main( , outputMessage: func.Out[func.ServiceBusMessage])
Can I keep -> str for Azure Service Bus ?
func.Out[func.ServiceBusMessage]) -> str
Can i use the set method to send message like :
outputMessage.set(text)
"To produce multiple outputs, use the set() method provided by the azure.functions.Out interface to assign a value to the binding" -> will this work?
Thanks
Sandeep

Another example of being able to do output binding to service bus with python:
// function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "function",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "$return"
},
{
"type": "serviceBus",
"direction": "out",
"connection": "AzureWebJobsServiceBusConnectionString",
"name": "msg",
"queueName": "outqueue"
}
]
}
.
# __init__.py
import azure.functions as func
def main(req: func.HttpRequest, msg: func.Out[str]) -> func.HttpResponse:
msg.set(req.get_body())
return 'OK'

When you say Azure Service Queue you probably mean Azure Storage Queue.
The binding serviceBus is specifically for Service Bus, not Storage Queues.
You need to use queueName.
Here's an example of output binding. Firstly, we have the function.json file:
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "function",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "$return"
},
{
"type": "queue",
"direction": "out",
"name": "msg",
"queueName": "outqueue",
"connection": "AzureWebJobsStorage"
}
]
}
Now we can use it like so:
import logging
import azure.functions as func
def main(req: func.HttpRequest, msg: func.Out[func.QueueMessage]) -> str:
name = req.params.get('name')
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
msg.set(name)
return func.HttpResponse(f"Hello {name}!")
else:
return func.HttpResponse(
"Please pass a name on the query string or in the request body",
status_code=400
)
Output Binding with Python to Service Bus
Here's your function.json code:
{
"name": "outputSbQueue",
"type": "serviceBus",
"queueName": "testqueue",
"connection": "MyServiceBusConnection",
"direction": "out"
}
Also, the following references may be helpful for you:
Azure Function - Python - ServiceBus Output Binding - Setting Custom Properties
https://github.com/yokawasa/azure-functions-python-samples/blob/master/docs/quickstart-v2-python-functions.md
https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-python
https://azure.microsoft.com/en-us/blog/taking-a-closer-look-at-python-support-for-azure-functions/
https://unofficialism.info/posts/quick-start-with-azure-function-v2-python-preview/
Sending messages to service bus queue without an explicit binding
from azure.servicebus import QueueClient, Message
# Create the QueueClient
queue_client = QueueClient.from_connection_string(
"<CONNECTION STRING>", "<QUEUE NAME>")
# Send a test message to the queue
msg = Message(b'Test Message')
queue_client.send(msg)

Using the Azure ServiceBusMessage class as an outbound parameter is not supported:
Attributes are not supported by Python.
Use the Azure Service Bus SDK rather than the built-in output binding.
Azure Service Bus output binding for Azure Functions
ServiceBusMessage can only be used for Trigger binding:
import azure.functions as func
def main(msg: func.ServiceBusMessage):
The queue message is available to the function via a parameter typed as func.ServiceBusMessage. The Service Bus message is passed into the function as either a string or JSON object.
Azure Service Bus trigger for Azure Functions

Related

Process csv file stored in blob storage using Python Azure functions

I have a csv file with 'n' number of records stored in blob storage. I want to read the new records being added to the csv file, process it and stored it back to another container in the blob storage. I want to achieve this flow using Python Azure Functions. I am unable to write the code for inbound and outbound in the Python Azure Functions.
Please help. Thanks
Below is the code that worked for me. I'm using HTTP Trigger to achieve your requirement.
function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"type": "blob",
"direction":"in",
"name": "inputblob",
"path": "input/input.csv",
"connection": "storageacc_STORAGE"
},
{
"type": "blob",
"direction":"out",
"name": "outputblob",
"path": "output/output.csv",
"connection": "storageacc_STORAGE"
}
]
}
init.py
import csv
import logging
import azure.functions as func
def main(req: func.HttpRequest, inputblob: func.InputStream,outputblob: func.Out[bytes]) -> func.InputStream:
# Reading from the input binding
input_file = inputblob.read()
# Processing the csv file
# Writing to output binding
outputblob.set(input_file)
local.settings.json
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "",
"FUNCTIONS_WORKER_RUNTIME": "python",
"storageacc_STORAGE": "<Your_Connection_String>"
}
}
RESULTS:

Azure Function not picking up Eventhub events

I started playing around with Azure Functions and am running in the issue that my Function is not being triggered by events entering my eventhub.
This is the code for my Function:
host.json:
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[2.*, 3.0.0)"
}
}
function.json:
"scriptFile": "__init__.py",
"bindings": [
{
"type": "eventHubTrigger",
"name": "events",
"direction": "in",
"eventHubName": "eventhub",
"connection": "eventhub_connection",
"cardinality": "many",
"consumerGroup": "$Default",
"dataType": "stream"
}
]
}
init.py:
import logging
import azure.functions as func
def main(events: List[func.EventHubEvent]):
for event in events:
logging.info('Python EventHub trigger processed an event: %s',
event.get_body().decode('utf-8'))
logging.info(f'Function triggered to process a message: {event.get_body().decode()}')
logging.info(f' EnqueuedTimeUtc = {event.enqueued_time}')
logging.info(f' SequenceNumber = {event.sequence_number}')
logging.info(f' Offset = {event.offset}')
# def main(event: func.EventHubEvent):
# logging.info(f'Function triggered to process a message: {event.get_body().decode()}')
# logging.info(f' EnqueuedTimeUtc = {event.enqueued_time}')
# logging.info(f' SequenceNumber = {event.sequence_number}')
# logging.info(f' Offset = {event.offset}')
# # Metadata
# for key in event.metadata:
# logging.info(f'Metadata: {key} = {event.metadata[key]}')
"IsEncrypted": false,
"Values": {
"FUNCTIONS_WORKER_RUNTIME": "python",
"AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=storageaccount;AccountKey=storageacciuntaccesskey=;EndpointSuffix=core.windows.net",
"eventhub_connection": "Endpoint=sb://eventhub01.servicebus.windows.net/;SharedAccessKeyName=function;SharedAccessKey=0omitted;EntityPath=eventhub"
}
}
I started out with the basic eventhub python code provided by the Azure Function Core tools. And have been testing different pieces of code found in online examples from people's blogs and the Microsoft docs.
When switching to cardinality: one -> I switch to the code which is currently commented out. I don't know if that is supposed to go like that, it just feels right to me.
In any case, regardless of the cardinality setting, or the datatype being changed between binary, stream or string. My Function simply does not trigger.
I can query my eventhub and see/read the events. So I know my policy, and the sharedkey and such, work fine. I am also only using the $Default consumer group.
I also tried setting up a HTTP triggered function, and this function gets triggered from Azure Monitor. I can see in the logs each request entering the function.
Am I doing something wrong in the code for my eventhub function?
Am I missing some other configuration setting perhaps? I already checked the Access Rules on the function, but that realy doesn't matter does it? The function is pulling the event from the eventhub. It's not being sent data by an initiator.
Edit: Added the local.settings.json file configuration and updated the function.json
Edit 2: solution to my specific issue is in the comments of the answer.
Update:
__init__.py of the function:
from typing import List
import logging
import azure.functions as func
def main(events: List[func.EventHubEvent]):
for event in events:
logging.info('Python EventHub trigger processed an event: %s',
event.get_body().decode('utf-8'))
Send message to event hub:
import asyncio
from azure.eventhub.aio import EventHubProducerClient
from azure.eventhub import EventData
async def run():
# Create a producer client to send messages to the event hub.
# Specify a connection string to your event hubs namespace and
# the event hub name.
producer = EventHubProducerClient.from_connection_string(conn_str="Endpoint=sb://testbowman.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxxxxx;EntityPath=test", eventhub_name="test")
async with producer:
# Create a batch.
event_data_batch = await producer.create_batch()
# Add events to the batch.
event_data_batch.add(EventData('First event '))
event_data_batch.add(EventData('Second event'))
event_data_batch.add(EventData('Third event'))
# Send the batch of events to the event hub.
await producer.send_batch(event_data_batch)
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
And please make sure you give the right event hub name:
It seems your function.json has a problem, the connection string should not directly put in the binding item.
It should be like below:
function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"type": "eventHubTrigger",
"name": "events",
"direction": "in",
"eventHubName": "test",
"connection": "testbowman_RootManageSharedAccessKey_EVENTHUB",
"cardinality": "many",
"consumerGroup": "$Default",
"dataType": "binary"
}
]
}
local.settings.json
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=0730bowmanwindow;AccountKey=xxxxxx;EndpointSuffix=core.windows.net",
"FUNCTIONS_WORKER_RUNTIME": "python",
"testbowman_RootManageSharedAccessKey_EVENTHUB": "Endpoint=sb://testbowman.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxxxxx;EntityPath=test"
}
}
check for configuration of function app and EventHub. pre-warned instance of function app should be lesser/equal to partition count of EventHub. Worked for me ad was able to receive events properly after this configuration.

Http Trigger-Python-Log Specific Message Constantly and not URL body or payload

Building on an earlier question. The following code is an httptrigger that listed to a gis layer edits and updates. It logs into the queue the url payload. I do not want the payload loaded but a specific repetitive message so that it is overwritten everytime for I do not want to dequeue every now and then. How can I go about this?
import logging
import azure.functions as func
def main(req: func.HttpRequest,msg: func.Out[str]) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
input_msg = req.params.get('message')
logging.info(input_msg)
msg.set(req.get_body())
return func.HttpResponse(
"This is a test.",
status_code=200
)
**function.json**
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "$return"
},
{
"type": "queue",
"direction": "out",
"name": "msg",
"queueName": "outqueue1",
"connection": "AzureStorageQueuesConnectionString"
}
]
}
I do not want the payload loaded but a specific repetitive message so
that it is overwritten everytime for I do not want to dequeue every
now and then.
No, when you put in the same message, It will not overwritten. It just queued in the queue storage.
If you want to process the message in queue, just use queueclient or use queuetrigger of azure function.(queuetrigger of function is based on queueclient, they are basically same.)
This is the API reference of queue:
https://learn.microsoft.com/en-us/python/api/azure-storage-queue/azure.storage.queue?view=azure-python
You can use it to process message in the queue with python code.
And this is the queuetrigger of azure function:(This is already integrated and can be used directly)
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-queue-trigger?tabs=python

How send data to the event while using timer trigger azure function v2 in python

I'm trying to send data every 10 seconds to an Event Hub using the Azure Functions v2.
Here my code for function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"name": "mytimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "*/10 * * * * *"
},
{
"type": "eventHub",
"name": "$return",
"eventHubName": "sqlserverstreaming",
"connection": "amqps://desrealtimestreaming.servicebus.windows.net",
"direction": "out"
}
]
}
Here my code for init.py
import datetime
import logging
import azure.functions as func
def main(mytimer: func.TimerRequest) -> None:
utc_timestamp = datetime.datetime.utcnow().replace(
tzinfo=datetime.timezone.utc).isoformat()
print('saran')
return 'Message created at: {}'.format(utc_timestamp)
# if mytimer.past_due:
# logging.info('The timer is past due!')
# logging.info('Python timer trigger function ran at %s', utc_timestamp)
And I'm getting the following error.
Executed 'Functions.TimerTriggerFUnction' (Failed, Id=fa924331-418f-427f-b672-f525c3ee6b61)
[07-08-2019 09:03:40] System.Private.CoreLib: Exception while executing function: Functions.TimerTriggerFUnction. System.Private.CoreLib: Result: Failure
Exception: FunctionLoadError: cannot load the TimerTriggerFUnction function: Python return annotation "NoneType" does not match binding type "eventHub"
Stack: File "C:\Users\SivaSakthiVelan\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\deps\azure\functions_worker\dispatcher.py", line 240, in _handle__function_load_request
function_id, func, func_request.metadata)
File "C:\Users\SivaSakthiVelan\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\deps\azure\functions_worker\functions.py", line 241, in add_function
f'Python return annotation "{return_pytype.__name__}" '
Change your return type from None to str
def main(mytimer: func.TimerRequest) -> str:
See here for complete example:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs#output---python-example
Also, your connection property should have the name of the app setting that contains the connection string, not the connection string itself (see also in the example), so like this:
"type": "eventHub",
"name": "$return",
"eventHubName": "sqlserverstreaming",
"connection": "MyConnStringAppSetting",
"direction": "out"
And then create an app setting named MyConnStringAppSetting and put your full connection string in there.

Azure Function binding: get both EventData and strongly typed message

I have the following function definition.
Message type:
type MailboxItem = {
CustomerID: int
AssetID: int
}
Code:
let Run(item: MailboxItem, userNames: string, log: TraceWriter) =
log.Verbose("F# function executing for " + item.AssetID.ToString())
And function.json:
{
"bindings": [
{
"type": "eventHubTrigger",
"name": "item",
"direction": "in",
"path": "eventhubpath",
"connection": <connection>,
"consumerGroup": "$Default"
},
{
"type": "blob",
"name": "userNames",
"path": "blobpath/{CustomerID}-{AssetID}",
"connection": <connection>,
"direction": "in"
}
],
"disabled": false
}
As you can see, I'm using properties of the incoming message to bind an input blob from Blob Storage.
Now, I need to extend my function to access some metadata of the incoming message via EventData class (e.g. sequence number). Is it possible to add EventData parameter but also keep the binding to properties of the message body?
No not currently, unfortunately, though this is a common ask and something we're tracking in our repo here and will hopefully get to soon. Until we do, it is an either/or - you can bind to EventData or your custom POCO.

Resources