I'm trying to understand what it is and how it's calculated a max_connection for RDS Aurora DB.
At the moment we are trying to use db.t2.small, which based on the AWS documentation allows max_connections = 45
Is 45 p/month?
Can someone please explain what defines 1 connection? Is it a query to the database?
If it's a query, what if a query returns 100 values? Are those counted as connections?
Thanks
Joe
I am not a database expert, but I can try to answer your questions from what my understanding is.
1)Is 45 p/month?
No, the max_connections is not per month but the number of concurrent
connections allowed simultaneously.
2) Can someone please explain what defines 1 connection? Is it a query to the database?
Connection could either be a query or you trying to manually enter into the
database.
Since for a query to read/write into the database, it needs to open a
connection to the DB, hence it is also counted as one connection, as something
(read here as a piece of code)is connected to your database. It is the same thing
as you opening a client, typing username/password etc and entering into the DB.
3) If it's a query, what if a query returns 100 values? Are those counted as connections?
The value returned by the query doesn't matter. It can return as many values as it
like(you need to handle that in your code though). What happens in the code is that
you open a connection and then run a query after that you close that connection.
What happens between when the connection open and closes doesn't affect the number
of connection.
However you need to take care of other parameters while running a long query into a
database. As it could spike your CPU/memory utilization etc. but the number of
connection per query would still be 1.
Related
as the title already says, I'm trying to run a raw SELECT query that results in 3.6 records.
When I use the MariaDB Connector (https://github.com/mariadb-corporation/mariadb-connector-nodejs) it works just fine and is done in ~2 minutes.
But Sequelize takes much longer and in the end, Node crashes after 20 to 30 minutes.
Do you guys have any idea how to fix this?
Thank you very much and have a great day!
Take care,
Patrick
When you perform your request, sequelize will perform a SELECT on the underlying database.
Then two thing will happend consecutively :
MariaDB will load all the data matching your criteria
MariaDB will send all the data to sequelize, that will :
Overload your app memory (all the data will be stored into node.js memory)
Crash sequelize because it is not made to handle that much data
When you perform request on huge dataset, use cursors. With cursors, MariaDB will load all the data but then, sequelize will get the data by group (For example, sequelize load 100 data, you treat it, then it load 100 data again, which means that at top you will have loaded 100 data on your node.js memory).
https://github.com/Kaltsoon/sequelize-cursor-pagination
https://mariadb.com/kb/en/cursor-overview/
Redesign your app to do more of the work in the database. Downloading 3.6M rows is poor design for any application.
If you would like to explain what you are doing with that many rows, maybe we can help create a workaround. (And it may even work faster than 2 minutes!)
Currently I store data on my MongoDB like so:
1 database per customer
each customer has about 40 collections
a total of 50 customers (there are 40*50=2000 collections)
database size is 3 GB for all 50 customers
My MongoDB service crashes with the error
"Too many open files"
and MongoDB "Too Many Open Files"? Raise the limit shows how to solve the problem but I am still getting crashes.
The error "Too many open files" means that MongoDB process has too many files in use and the OS is complaining. This is how I know MongoDB is using a lot of files:
Obtain the processes id of MongoDB with service mongodb status. The process id will show in the info
Then to get the files used by MongoDB I use this command lsof -a -p <ProcessId>
When I run that command I see 1010 files being used by that processes!
The more customer databases I create the larger that number becomes! So I guess my solution is combining all the databases into one. If I do that I will have to add the column AccountId to all my collections. If I make that change what index should I assign to AccountId so that my searches can be efficient? For example I will like to get all PurchaseOrders where IdAccount=34 fast. Performing this change is something you guys recommend? Should I place all 50 Databases into one?
PS: On a different Linux computer I created a MongoDB database with only 1 database and 40 collections. I filled the 40 collections with 6 GB of data (double of what I have now). MongoDB used 200 files even though this database is twice as big!
The same day after posting this question I combined all the databases into one. Moreover, I added the follwoing indexes:
db.CollectionA.createIndex({Id_Account:1})
db.CollectionB.createIndex({Id_Account:1})
// etc...
To prove that my queries continued to be as efficient as before I do:
db.getCollection('CollectionA').find({"Id_Account":28}).explain("executionStats")
That query gives the execution stats. It tells you how many documents it searched for and how many matched. Using NO index will result in scanning the whole collection every time I did find({"Id_Account":28})
Mongo has not crashed so far and it never locks more than 300 files. From know on I will always use the same database instead of having multiple databases.
I have a 2010 MS Access database that is being updated from an excel VB form to hold vehicle details. That's the easy bit and all the VB works fine and the database is updating.
The difficult bit comes when I created a telephony IVR call flow in your site explorer (Mitel). This reads the database every 30 seconds to see if there has been any new records entered, if there has then it marks the database record as 'call in progress' and sets the time to the next call as 5 minutes time. Again, this all works fine UNTIL I add more than 1 port to the call flow.
I am now in the situation that the first port will call out and update the database, but another port will access the same record before the first port updates the database and so calls the same destination about the same job.
I've tried changing the refresh rates within access to 1 sec, but no success.
I realise this is all a bit complicated, but it's basically a problem with access not updating quickly enough (i.e. it needs to update within a second). Any help would be gratefully accepted, as this is now totally doing my head in.
The database is saved as a mdb and not accdb, if this makes any difference.
Thanks very much.
Answer turned out to be, open and close the database in the vb code after inserting data into the database table via an ODBC link. This automatically updates the database table with the record just added. The outbound telephony ports didn't then try to call twice (or more) for the same record, as the record was updated so quickly. I then set the refresh rate of the database to 1 sec for both ODBC updates and normal updates, this made sure the database was then updated within 1 sec, thus solving the problem for 95% of calls. I also set the default record locking to editor locking, so that even if 2 ports were being used for the same record, only 1 updated the database.
I am bypassing the ORM and using the Model.query function to query and return a "large" result set from PostgreSQL. The query returns around 2 million rows. When running the query directly from postgres it returns in around 20s. The query fails silently when executed from sails. Is there a limit on the number of rows that can be returned?
Is there a limit on the number of rows that can be returned?
No there is no limit.
The query fails silently when executed from sails
What does "fails silently" mean? How do you know that it's failed? It may still be processing; or the adapter might have a connection timeout that you're breaching.
Two million rows serialized out of the database, translated to JSON, and piped down to the client is much different than just running SQL directly on the database. It could take 20x longer, depending on your system resource situation. I strongly recommend using sails.js's paging features to pull the data out in chunks. Pulling two million rows in one operation from a web server doesn't make a lot of sense.
I am currently using Cassandra 1.6.6, however I having a big problem. I am trying to fetch more than 100000 rows using the LIMIT clause, however I always get the error below and then the database just shutdown.
TSocket read 0 bytes.
Secondly, does any know how to update all the rows in a Cassandra database.
Thanks awaiting your reply. I just can't find anything online, very distress.
TSocket read 0 bytes means you lost the connection to Cassandra, possibly due to the timeout that stops running a malformed query that would cause system instability. I don't think you can run one query that updates all rows because you need to specify the unique key to update a row.