Google Pub/Sub - No event data found from local function after published message to topic - python-3.x

I'm using the Functions Framework with Python alongside Google Cloud Pub/Sub Emulator. I'm having issues with an event triggered from a published message to a topic, where there's no event data found for the function. See more details below.
Start Pub/Sub Emulator under http://localhost:8085 and project_id is local-test.
Spin up function with signature-type: http under http://localhost:8006.
Given a background cloud function with signature-type: event:
Topic is created as test-topic
Function is spinned up under http://localhost:8007.
Create push subscription test-subscription for test-topic for endpoint: http://localhost:8007
When I publish a message to test-topic from http://localhost:8006 via POST request in Postman, I get back a 200 response to confirm the message was published successfully. The function representing http://localhost:8007 gets executed as an event as shown in the logs from the functions-framework. However, there's no actual data for event when debugging the triggered function.
Has anyone encountered this? Any ideas/suggestions on this?Perhaps, this is true? #23 Functions Framework does not work with the Pub/Sub emulator
Modules Installed
functions-framework==2.1.1
google-cloud-pubsub==2.2.0
python version
3.8.8

I'll close this post, since the issue is an actual bug that was reported last year.
Update: As a workaround until this bug is fixed, I copied the code below locally to functions_framework/__init__.py within view_func nested function, inside _event_view_func_wrapper function.
if 'message' in event_data:
if 'data' not in event_data:
message = event_data['message']
event_data['data'] = {
'data': message.get('data'),
'attributes': message.get('attributes')
}

Related

Stopping the listener 'Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobListener' for function <XXX>

I have function app where I have one HttpTrigger and 3 BlobTrigger functions. After I deployed it, http trigger is working fine but for others functions which are blob triggers, it gives following errors
"Stopping the listener 'Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobListener' for function " for one function
Stopping the listener 'Microsoft.Azure.WebJobs.Host.Listeners.CompositeListener' for function
" for another two
I verified with other environments and config values are same/similar so not sure why we are getting this issue in one environment only. I am using consumption mode.
Update: When file is placed in a blob function is not getting triggered.
Stopping the listener 'Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobListener' for function
I was observed the same message when working on the Azure Functions Queue Trigger:
This message doesn't mean the error in function. Due to timeout of Function activity, this message will appear in the App Insights > Traces.
I have stopped sending the messages in the Queue for some time and has been observed the traces like Web Job Host Stopped and if you run the function again or any continuous activity is present in the Function, then this message will not appear in the traces.
If you are using elastic Premium and has VNET integrated, the non-http trigers needs Runtime scale monitoring enabled.
You can find Function App-->Configuration--> Function runtime settings and turn on Runtime scale monitoring.
If function app and storage account which holds the metadata of the function Private linked, you will need to add the app settings WEBSITE_CONTENTOVERVNET = 1.
Also, make sure you have private linked for blob, file, table and queue on storage account.
I created ticket with MS to fix this issue. After analysis I did some code changes as
Function was async but returning void so changed to return Task.
For the trigger I was using connection string from app settings. But then I changed it to azureWebJobStorage(even though bobth were same) in function trigger attribute param
It started working. So posting here in case it is helpful for others

AWS MSK Trigger - Lambda (consumer) running infinitely

I am working on a notification service which is built on AWS infra and uses MSK, lambda and SES.
The lambda is written in Nodejs for which the trigger is an MSK topic. Now the weird thing about this lambda is its getting invoked continuously even after the messages are processed. Inside lambda is the code to fetch recipients and send emails via SES.
I have ensured that there is no loop present inside the code. so my guess is for some reason the messages are not getting marked as consumed.
One reason this could happen is if the code executing throws an Error at some point. But I have no error in the logs.
Can execution time (lambda getting timed out be an issue? I don't see anything like that in the logs though), the volume of messages be responsible for this behavior ?
Lambda is setup using serverless framework:
notificationsKafkaConsumer:
handler: src/consumers/notifications.consumer
events:
- msk:
arn: ${ssm:/kafka/cluster_arn~true}
topic: "notifications"
startingPosition: LATEST
It turned out that the lambdas getting timeout out was the issue. The message got lost in the huge volume of logs and interestingly "timed out" is not an error so filtering logs with "ERROR" didn't work.

Azure durable function log stream showing deleted function

I have deployed a change to a durable function product into Azure DevOps and part of the work was to rename an activity function.
However the warning below is appearing in the log stream containing the old name for the function.
[Warning] Activity function 'PublishNotification' does not exist..
InstanceId: . Function: PublishNotification.
This function publishes a message to a Service bus topic, and there is definitely now no reference to that function name anywhere in the code.
Any ideas where this might be coming from?
From your description, it seems that your function name has been changed, the orchestrator function listener and other function listeners are been deregistered:
https://github.com/Azure/azure-functions-durable-extension/blob/f6f1dce716b68d8baaa99ed64a9db1306577c58d/src/WebJobs.Extensions.DurableTask/Listener/DurableTaskListener.cs#L44-L54
But the orchestrator function is still running, and it cannot find your previous function name:
https://github.com/Azure/azure-functions-durable-extension/blob/f6f1dce716b68d8baaa99ed64a9db1306577c58d/src/WebJobs.Extensions.DurableTask/DurableOrchestrationContext.cs#L385
If you restart the function app and let everything start again, can you still reproduce this error?
I went into the Azure web job storage queues and had a look around. Turns our there was an old message (about 4 days old) stuck in one of the queues and when that was cleared out the warning messages disappeared too.

BadRequestObjectResult does not return detailed error message to the client

I have a Azure function with an HTTP trigger. It produces a 400 response when an input value is not specified:
if (string.IsNullOrEmpty(artistName))
{
return new BadRequestObjectResult("Artist name not specified.");
}
When the function is running locally (func host start) and the API is called incorrectly (I am using curl -X POST ...), the above error message is shown.
When deployed to Azure and called in a similar way - instead of the detailed error message - only Bad Request is returned.
Why?
How can I change this behaviour?
It's a known issue for function runtime 2.0.11888 that HttpTrigger does not return response content properly.
If you want to solve this, go to Application settings, pin your FUNCTIONS_EXTENSION_VERSION to previous runtime 2.0.11857-alpha until new version is released.
See Azure Functions Runtime 2.0.11888 Preview.
Update
This issue has already been fixed from 2.0.11933.

google-cloud-pubsubTypeError: state.topic.publish is not a function

This is image from node red in terminal
I am working in the sphere of IoT, and want to push message to Pub/Sub in Google, but every time when I run my node-Red, I've got the following error:
25 Dec 18:40:49 - [error] [google-cloud-pubsub out:b2451409.071148] TypeError: state.topic.publish is not a function*
As a source code, I used pub/sub contribution in github, link:
https://github.com/GoogleCloudPlatform/node-red-contrib-google-cloud/blob/master/pubsub.js
It seems that code works fine with credentials and it does create new-topic in Google,in the case, when topic is not present in the cloud, however the message is not published to the topic. In the case of repeating messages in particular interval, the problem above is arising.
Does anyone know how to solve this problem?
I think you've been using an older version of the pubsub API:
const topic = pubsub.topic('YOUR-TOPIC-NAME')
topic.publish(yourData, callback)
The new API as documented here (https://cloud.google.com/pubsub/docs/publisher) looks like this:
const topic = pubsub.topic('YOUR-TOPIC-NAME')
const publisher = topic.publisher()
publisher.publish(dataBuffer, dataJSON, callback)
Hope this fixes your problem.

Resources