Is there a way to send databricks job failure notification to AWS SNS? I could not find anything on this in the databricks documentation. It says it can but does not describe how to do it.
Related
I am trying to figure out the equivalent of AWS lambda EventBridge (CloudWatch Events) where the lambda is triggered when a Glue job state changed. But In GCP I am not found a way how to do it.
If anyone knows please describe or share some resources.
I am using Pub/Sub but it doesn't help out.
I have a background service which produces files in Google Cloud Storage. Once it is done it generates a file in the output folder.
In my flow I need to get the list of these files and start DataProc Spark job with the list of files. The processing is not real-time and takes tens of minutes.
GCS has a notifications system. It can stream the notification to Pub/Sub service.
In GCS there will be a file .../feature/***/***.done created to identify the service job completion.
Can I subscribe to new files in GCS by wildcard?
Once the file is created the notification gets to Pub/Sub service.
I believe I can write Cloud Function that would read this notification, by some magic will get the location of the modified file and will be able to list all files from that folder. Then publish another message to Pub/Sub with all the required information
Is that possible to start DataProc job by Pub/Sub notification?
Ideally, it would be great to use Jobs instead of Streaming to reduce costs. This may mean that PubSub initiates Job instead of streaming Job pulls the new message from PubSub
Question 1: Can I subscribe to new files in GCS by wildcard?
You can set up GCS notifications to filter by path prefix. See the -p option here. Cloud Pub/Sub also has a filtering-by-attribute API in the Beta. You can use it to filter by the attributes set by GCS. The filtering language supports exact matches and prefix checks on the attributes set by GCS.
The message published to the Cloud Pub/Sub topic will have attributes that give you the bucket and name of the object, so you should be able to easily read other files in that bucket/subpath.
Question 2: Is it possible to start a DataProc job by Pub/Sub notification?
Yes, you can set up a Cloud Function to subscribe from your Cloud Pub/Sub topic. The function can then start up a DataProc cluster using the DataProc client library, or do any other action.
My requirement is to send data received in Lambda from DynamoDB to Azure Queue in node.js.
Steps taken - AWS Side
1. Created DDB Table
2. Added Stream and Trigger
3. Wrote Lambda
Steps taken - Azure Side
1. Created an Azure Queue (Service Bus)
So far so good. I can see DDB Events making its way to Lambda.
My question is now I want to send these events to Azure Queue, I could not find any online google result for this. Is it possible to put elements in Azure Queue from AWS Lambda?
You can use Azure Service Bus REST API in order to send your messages:
POST http{s}://{serviceNamespace}.servicebus.windows.net/{queuePath|topicPath}/messages
https://learn.microsoft.com/en-us/rest/api/servicebus/send-message-to-queue
The requirement is that we have an Azure Service Bus topic which we want to set up as Push Subscriber in Google PubSub Topic. This way any messages published to Google PubSub Topic will be pushed to Azure SB Topic without any intermediate layer involved.
On Paper this should work because messages can be published to Azure SB Topic using API framework and Google PubSub can also configure API as Push Subscribers.
I have gone through following articles but couldn't make this linking workout.
Azure SB as API: https://learn.microsoft.com/en-us/rest/api/servicebus/send-message-to-queue
Google PubSub Push Subscriptions: https://cloud.google.com/pubsub/docs/push
Has anyone done this kind of linking before?
Thanks in Advance
It would be nice to create a pub/sub push subscription, which pushes to Azure Event Hub. I ran into the same problem when configuring this setup.
Pub/sub push subscriptions currently do not support custom Authorization headers, which are needed as per the Event Hub documentation.
POST https://your-namespace.servicebus.windows.net/your-event-hub/messages?timeout=60&api-version=2014-01 HTTP/1.1
Authorization: SharedAccessSignature sr=your-namespace.servicebus.windows.net&sig=your-sas-key&se=1403736877&skn=RootManageSharedAccessKey
Content-Type: application/atom+xml;type=entry;charset=utf-8
Host: your-namespace.servicebus.windows.net
{ "DeviceId":"dev-01", "Temperature":"37.0" }
So the only two options I see here:
Push setup: Create a cloud function or dataflow job in Google Cloud. Push pub/sub events to this endpoint and then pass on the event to Azure Event hub with the appropriate headers.
Pull setup: Poll a pub/sub pull subscription from the Azure side with an Azure function or WebJob.
Both options require extra compute resources, so definitely not the preferred way of doing this. I would always try a push setup first, since then you don't have t o continuously have a polling job running in the background.
I have hopes that pub/sub push subscriptions will support custom headers anywhere in the future. Lately some other useful features have been added like: Dead Lettering, Message Ordering and Partitioning (Lite Topics). Hopefully they will add custom headers as well.
I have a workaround on this problem.
1- you can create cloud function on google cloud that can push data for you on azure service bus.
2- you can develop web job on azure that will run continuously and check google pub/sub topic with the help of provided connection string and google pub/sub supporting libraries.
From above mentioned solution you can get data from google cloud and push to your azure service bus.
Is it possible to pull AWS DMS replication task percentage progress and use that to create an AWS Lambda function for sending notifications to Amazon SNS for let's say, every 10% completion? I can't find any metrics/event categories anywhere in relation to this while I was browsing. Thanks.
You don't need a lambda here. You can monitor the progress using CloudWatch then set up an SNS notification.
monitoring: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Monitoring.html
sns for cloudwatch: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_SetupSNS.html