I've searched the web and browsed the RDS docs, yet I can't seem to find an open connection limit.
For what it's worth, I'm planning on using the new Postgres flavor of RDS, but I suppose an answer from the mySQL side of the house would be acceptable as well.
Thanks!
If you setup a PostgreSQL RDS instance you can query for max_connections:
select * from pg_settings where name='max_connections';
max_connections is set to 5696.
Simply with show max_connections;
You can modify this value on the Parameter Groups on the RDS console. The value of max_connections in the default postgres9.3 Parameter Group is {DBInstanceClassMemory/12582880}. In version 9.4 the default value is {DBInstanceClassMemory/31457280}.
You can change the max_connections. Make a clone of the default parameters. Then change the value and save and set the instance group to the new options group and finally reboot your instance to make the setting take effect.
Afaik the 'max_connections' returned above is actually the current value of max_connections as set in the ParameterGroup. I wanted to know how to dynamically figure out the max possible connections of a given Postgres instance in AWS, but for some reason could not even find an info sheet from AWS. However, I noticed that AWS console RDS section shows you a progress bar of your current connections vs total possible:
Examining this in Google Chrome Tools, I have established that the blue part is 4 pixels wide, while the total width of the progress bar is 50 pixels, which means that my current 121 connections constitute roughly 8% of total, which gives an approximation of max possible connections for my db.r3.large instance at 1512.
May be it's too late for answering, but you could find the Amazon Aurora RDS Postgresql connection limit here:
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Managing.html#AuroraPostgreSQL.Managing.MaxConnections
Related
I am intermittently getting the following error back from the Splunk API (about 40% of the time search works as expected):
HTTP 503 Service Unavailable -- Search not executed: This search could
not be dispatched because the role-based disk usage quota of search
artifacts for user "[REDACTED]" has been reached (usage=1067MB,
quota=1000MB). Use the [[/app/search/job_manager|Job Manager]] to
delete some of your search artifacts, or ask your Splunk administrator
to increase the disk quota of search artifacts for your role in
authorize.conf., usage=1067MB, quota=1000MB, user=[REDACTED],
concurrency_category="historical",
concurrency_context="user_instance-wide"
The default ttl for a search in the splunk api is 10 min (at least for my company). I am told I need to lower the TTL for my searches (which are massive) and I will stop running out of space. I do not have admin access, so no ability to increase my space or clear space on the fly (as far I know). I can find code on how to lower TTL using saved searches, but I use oneshot searches. It is not reasonable for me to switch.
How do I lower ttl for oneshot searches?
Here is what I have now that does not seem to lower TTL:
#setup splunk connection
service = client.connect(
host=HOST,
port=PORT,
username=suser,
password=spass,
autologin=True,
)
#setup arguments
kwargs_oneshot = {"count" : "0",
"earliest_time": begin,
"latest_time": end,
"set_ttl":60
}
#setup search job
oneshotsearch_results = service.jobs.oneshot(query, **kwargs_oneshot)
# Get the results and display them using the ResultsReader
reader = results.ResultsReader(oneshotsearch_results)
Rather than set_ttl, I believe you need ttl or timeout. See https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTsearch#search.2Fjobs
Also, consider making your searches less massive or running them less often.
I have an AWS RDS Mysql 8.0.16 instance: db.t3.micro, 200GB of gp2 storage. A script (Java 11, JDBC, single connection) populates the database with some dummy data. Pseudocode:
insert into TemplateClasses(...) values(...);
insert into TemplateStudents(...) values(...);
for (int _schoolId = 0; _schoolId < 20; ++_schoolId) {
insert into Schools(id) values(_schoolId);
insert into Classes(schoolId, ...) select _schoolId, ... from TemplateClasses;
insert into Students(schoolId, ...) select _schoolId, ... from TemplateStudents;
}
I have 100% RDS burst balance and I start this script. The first few iterations usually consume 10-30% of the burst balance, but after it things get slow, and I see that burst balance is not being used (the script keeps running and there are no changes to the workload).
Here's a very typical picture:
What's the most probable cause of this behavior? How do I force RDS to spend 100% of the burst balance?
Added on 1/9/2021: I've changed the RDS instance type to db.t3.small and wasn't able to reproduce the issue. While this practically solves my problem, I'm still curious about the reason db.t3.micro behaves differently.
My postgres instance CPU utilisation has shot up recently. I'm trying to identify the root cause. I will add the details below.
My postgres database instance running on GCP has the following configuration:
v PostgreSQL 9.6
vCPUs-1
Memory-3.75 GB
SSD storage-15 GB
I'm running 5 databases in the above DB server which are connected with a nodejs app.
I use sequelize as my ORM and recently upgraded my sequeliz from 4.6.x to 5.8.6".
Before this upgrade the CPU utilization would usually remain less than 20 percent. But after the upgrade, I see a lot of fluctuation in CPU utilization graph. And it hits 100 percent too often as well. Also, when it hits 100%, my services start wont work as expected ( because they cant interact with the DB).
I tried running this query .
SELECT "usesysid", "backend_start", "xact_start","query_start", "state_change", "state", "query" FROM pg_stat_activity ORDER BY "query_start" DESC
And, it returns the following:
But I'm not sure if this info is enough for me to find out which query could be causing this issue.
I also ran this query:
SELECT max(now() - xact_start) FROM pg_stat_activity WHERE state IN ('idle in transaction', 'active');
and it returns max = 1 day 01:42:10.987635. I think this is something alarming, but i dont know how to put this info to use.
Another thing which i think is worth mentioning is, I have started using sequelize's bulk update.
Its syntax is something like this:
Model.bulkCreate(scalesToUpdate, {
updateOnDuplicate: [
'field1',
'field2'
],
})
And, this gets translated into SQL like below:
INSERT INTO "mymodel" ("id","field1","field2","field3","field4","field5","field6","field7") VALUES (') ON CONFLICT ("id") DO UPDATE SET "field3"=EXCLUDED."field3","field4"=EXCLUDED."field4","field6"=EXCLUDED."field6","field7"=EXCLUDED."field7"
And, this query gets fired 5 times per second. Could this be the culprit?
Any insight into this is highly appreciable.
You could try the next things:
Increase ht machine type to have one core more having vCPUs= 2
It might be that sequelize 5.8.6 requires more resources than the old version, you could try to install one of the tools and run it, run the queries that you typed, to review which query has more resource usage.
If you have that query running 5 times per second, that could be using more resources. Test using one of the tools in order to be able to have a better approach.
I have an RDS instance running MySQL 5.5.46 which has a table with a primary key of int that it is currently at 1.9 billion records and approaching the 2.1 billion limit and ~425GB in size. I'm attempting to use pt-osc to alter the column to a bigint.
I was able to successfully test the change on a test server (m3.2xlarge) and, while it took about 7 days to complete, it did finish successfully. This test server was under no additional load. (Side note: 7 days seemed like a LONG time).
For the production environment, there is no replication/slave present (but there is Multi-AZ) and, to help with resource contention and speed things up, I'm using an r3.8xlarge instance type.
After two attempts, the production migration would get to about 50% and a 1 day left and then the RDS would seemingly stop accepting connections forcing the pt-osc both times to roll back or fail outright, because the RDS needed to be rebooted.
I don't see anything obvious in the RDS console or logs to help indicate why this happened, and I feel like the instance type should be able to handle a lot of connections/load.
Looking at the CloudWatch metrics during my now third attempt, the database server itself doesn't seem to be under much load: 5% CPU, 59 DB Connections, 45GB Freeable Memory, Write IOPS ~2200-2500.
Wondering if anyone has ran into this situation and, if so, what helped with the connection issue?
If anyone has suggestions on how to speed up the process in general I'd love to hear. I was considering trying a larger chunk-size and off hours, but wasn't sure how that would end up affecting the application.
We host a website on Microsoft Azure and we have Traffic Manager distributing traffic on two AppServices each with 15 instances and a maxpool size of 80 in the connection strings.
We also use P11 database which has a max connection count of 2400
From our understanding, the maximum possible connections would be:
Instance count * maxpool
30 * 80 = 2400
But we get errors regarding exceeding the maximum allowed number and to our surprise, running a query on the database to show us active connections yields 2600.
We don't have any webjobs running.
Can someone please explain what is happening?
Using the following query you can identity which program name have the most sessions, which login name creates the most sessions, which host creates the most sessions.
SELECT
c.session_id, c.net_transport, c.encrypt_option,
c.auth_scheme, s.host_name, s.program_name,
s.client_interface_name, s.login_name, s.nt_domain,
s.nt_user_name, s.original_login_name, c.connect_time,
s.login_time
FROM sys.dm_exec_connections AS c
JOIN sys.dm_exec_sessions AS s
ON c.session_id = s.session_id
The following statement shows you the maximum number of connections for the current tier.
SELECT ##MAX_CONNECTIONS AS 'Max Connections';
Hope this helps.
Regards,
Alberto Morillo