I get the above message in my Amazon RDS instance alerts section. How do I get notified by email when such violations are reported by RDS.
The most straight forward option for monitoring Amazon RDS (and any other AWS service for that matter) is Amazon CloudWatch, which provides a reliable, scalable, and flexible monitoring solution that you can start using within minutes and specifically includes Alarms:
[...] Alarms can automatically initiate actions on your behalf, based
on parameters you specify. An alarm watches a single metric over a
time period you specify, and performs one or more actions based on the
value of the metric relative to a given threshold over a number of
time periods. The action is a notification sent to an Amazon SNS topic
or Auto Scaling policy. [...] [emphasis mine]
Amazon SNS supports notifications over multiple transport protocols in turn, amongst those Email and Email-JSON, see the respective FAQ What are the different delivery formats/transports for receiving notifications?:
[...] Customers can select one the following transports as part
of the subscription requests:
[...]
”Email”, “Email-JSON” – Messages are
sent to registered addresses as email. Email-JSON sends notifications
as a JSON object, while Email sends text-based email.
The metric in question is the FreeStorageSpace RDS metric (see Amazon RDS Dimensions and Metrics for details on the available ones) as discussed in Scaling DB Instance Storage:
Important
We highly recommend that you constantly monitor the
FreeStorageSpace RDS metric published in CloudWatch to ensure that
your DB Instance has enough free storage space. For more information
on monitoring RDS DB Instances, see Viewing DB Instance Metrics.
Accordingly, you'll need to create an alarm mirroring or approximating the threshold reported to you by AWS in the console, publish it to an SNS topic and subscribe to this topic via an email address of your choice.
Related
I am trying to use AWS services to implement a real-time email-sending feature for one of my projects. It is like someone uses my app to schedule a reminder from my project and then the email will be sent to them nearby or at the actual time that he scheduled.
I know the AWS services such as AWS CloudWatch rules (CRONs) and DynamoDB stream (TTL based). But that is not perfect for such a feature. Can anyone please suggest a better way to implement such a feature?
Any type of guidance is acceptable.
-- Thanks in advance.
Imagine your service at huge scale. At such scale, there are likely to be multiple messages going off every minute. Therefore, you could create:
A database that stores the reminder times (this could be DynamoDB or an Amazon RDS database)
An AWS Lambda function that is configured to trigger every minute
When the Lambda function is triggered, it should check the database to retrieve all reminders that should be sent for this particular minute. It can then use Amazon Simple Email Service (SES) to send emails.
If the number of emails to be sent is really big, then rather than having the Lambda function call SES in series, it could put a message into an Amazon SQS queue for each email to be sent. The SQS queue could then trigger another Lambda function that sends the email via SES. This allows the emails to be sent in parallel.
I would like to setup following alerts for domain topics when
Delivery Failed Events (at domain) exceed x in y amount of time
Delivery Failed Events (at domain topic 1) exceed x in y amount of time
Delivery Failed Events (at domain topic 2) exceed x in y amount of time
The reason why I want the domain topic granularity is that topic 1 customer may be fine but topic 2 customer may be having issues. So customer (for topic 2) is down currently and is in extended outage period (that may last more than a day). So I want to be able to disable the alert for topic 2 only and would like to enable it once customer (for topic 2) is up and running again. Meanwhile, I want to have all other topic level alerts enabled.
I did not see a way to configure the above in the portal. Is it possible (or not) to configure above at this time in any other way? If so, can please provide the direction on how to achieve it?
The AEG provides durable delivery for each event message at least once to each subscriber based on its subscription. More details can be found in the docs.
In the case, when the AEG can not successfully deliver a message after retrying, the dead-lettering feature (configured for each subscriber) can be used for notification and/or analyzing process via a storage eventing, where a dead-letter message is stored.
On the publisher side, the publisher received a standard Http response from the event domain endpoint immediately after its posting, see more details in the docs.
The current version of the AEG is not integrated to the Diagnostic settings (for instance, like it is done for Event Hubs) which will be enabled to push the metrics and/or logs to the stream pipeline for their analyzing process.
However, as a workaround for that, the Azure Monitoring REST API can help you.
Using Lists the metrics values for event domain, we can obtained the metrics for topics such as Publish Succeeded, Publish Failed and Unmatched.
the following is an example of the REST Get:
https://management.azure.com/subscriptions/{myId}/resourceGroups/{myRG}/providers/Microsoft.EventGrid/domains/{myDomain}/providers/Microsoft.Insights/metrics?api-version=2018-01-01&interval=PT1M&aggregation=none&metricnames=PublishSuccessCount,PublishFailCount,PublishSuccessLatencyInMs,DroppedEventCount
Based on the polling technique, you can push the event domain metrics values to the stream pipeline for their analyzing, monitoring, alerting, etc. using an Azure Stream Analytics job. Your management requirements (for instance, publisher_topic1 is disabled, etc.) can be referenced to the input stream job.
Note, that the event domain metrics didn't give a topic granularity and also there is no an activity event log at that level. I do recommend to use the AEG feedback page.
Let's say I've got an azure service bus in a microservice scenario.
One microservice pushes master data changes to the other services with a subscription.
Now let's say a new service is introduced and subscribes to the master data service. How can I make sure that the new service receives all neccessary data?
Do I have to resend all master data on the master data service or does the azure service bus (or alternatives) provide some features for that?
As far as I know there is no way to achieve what you want within the capabilities of Azure Service Bus. Also, I don't think this what Service Bus is there for.
Of course there is a configurable "time to live" value for messages within queues and topics, which could probably be set to some really high value, but this would still not make your master data be infinitely available for future services. And - but this is just my opinion and I'm far from being an expert - I wouldn't want to load up my service bus with potentially thousands or even millions of messages (depending on what you're doing) without them being processed quickly.
For your specific concern I'd rather implement something like a "master data import service" without any service bus integration. Details of this, however, depend on your environment and specific requirements.
Couple of points:
1) This is not possible with Azure Service bus. Even If you set TTL at Topic level, the messages will only be delivered to available subscriptions at that point of time. you cant read messages directly from Topic.
2) you can consider Eventhub option where you can create new consumer group with offset from when you want to start reading messages but Eventhub has maximum retention period as 7 days. If you need message retention beyond 7 days, enabling Event Hubs Capture on your event hub pulls the data from your event hub to the Storage account. But in this case you would require additional logic to read from this storage account to replay the messages.
The Problem:
So we are building a newsletter system for our app that must have a capacity to send 20k-40k emails up to several times a day.
Tools of Preference:
Amazon SES - for pricing and scalability
Azure Functions - for serverless compute to send emails
Limitations of Amazon SES:
Amazon SES Throttling having Max Send Rate - Amazon SES throttles sending via their services by imposing a max send rate. Right now, being out of the sandbox SES environment, our capacity is 14 emails/sec with 50K daily emails cap. But this limit can be increased via a support ticket.
Limitations of Azure Functions:
On a Consumption Plan, there's no way to limit the scale as to how many instances of your Azure Function execute. Currently the scaling is handled internally by Azure, and thus the function can execute between just a few to hundreds of instances.
From reading other post on Azure Functions, there seems to be "warm-up" period for Azure Functions, meaning the function may not execute as soon as it is triggered via one of the documented triggers.
Limitations of Azure Functions with SES:
The obvious problem would be Amazon SES throttling sending emails from Azure functions because the scaled execution of Azure Function that sends out an email will be much higher than allowed send rate for SES.
Due to "warm-up" period of Azure Function messages may end up being piled up in a queue before Azure Function actually starts processing them at a scale and sending out the email, thus there's a very high probability of hitting that send/rate limit.
Question:
How can we utilize sending emails via Azure Functions while still being under X emails/second limit of SES? Is there a way to limit how many times an Azure Function can execute per time frame? So like let's says we don't want more than 30 instances of Azure Function running per/second?
Other thoughts:
Amazon SES might not like continuous throttling of SES for a customer if the customer's implementation is constantly hitting up that throttling limit. Amazon SES folks, can you please comment?
Azure Functions - as per documentation, the scaling of Azure Functions on a Consumption Plan is handled internally. But isn't there a way to put a manual "cap" on scaling? This seems like such a common requirement from a customer's point of view. The problem is not that Azure Functions can't handle the load, the problem is that other components of the system that interface with Azure Functions can't handle the load at the massive scale at which Azure Functions can handle it.
Thank you for your help.
If I understand your problem correctly, the easiest method is a custom queue throttling solution.
Basically your AF just retrieve the calls for all the mailing requests, and then queue them into a queue system (say ServiceBus / EventHub / IoTHub) then you can have another Azure function that runs by x minute interval, which will pull a maximum of y messages and push it to SES. Your control point becomes that clock function, and since the queue system will help you to ensure you know your message delivery status (has sent to SES yet) and you can pop the queue once done, it would allow you to ensure the job is eventually processed.
You should be able to set the maxConcurrentCalls to 1 within the host.json file for the function; this will ensure that only 1 function execution is occurring at any given time and should throttle your processing rate to something more agreeable from AWS perspective in terms of sends per second:
host.json
{
// The unique ID for this job host. Can be a lower case GUID
// with dashes removed. When running in Azure Functions, the id can be omitted, and one gets generated automatically.
"id": "9f4ea53c5136457d883d685e57164f08",
// Configuration settings for 'serviceBus' triggers. (Optional)
"serviceBus": {
// The maximum number of concurrent calls to the callback the message
// pump should initiate. The default is 16.
"maxConcurrentCalls": 1,
...
I know amazon provides awesome metrics for monitoring rds box, but my question is, if I only want to monitor whether it's reachable or not, like zabbix ping, what metric shall i use when creating an alarm?
On the RDS console you can create event subscriptions, select events (like availability and failure) and assign notification groups.
I didn't find an option on cloudwatch to do this.