I'm using socket.io to handle realtime connections on my site. When a client connects, though, I save out the connection to a mysql database along with other options like what url they are on. I'm using the socket.id as the primary key for that mySQL table as I assumed that would be unique. The purpose of the mySQL database is to make it easier to perform sorting later on. If there's a better (in-node) way to do that I'm all ears.
When the socket disconnect I remove the row from the mySQL table.
However, I seem to be getting some asynchronous issues here that I can't figure out a workaround. Since the Insert into mysql is asynchronous, it seems possible that a user may disconnect BEFORE mysql has inserted into the database. At which point the internals of socket.io and the mySQL database will be out of synch.
How would you go about keeping the two in synch? Is it worth BLOCKING on the mySQL Insert? Can node even do that? Or, is there a different approach?
Related
Similar to tools like Retool, I'm looking to create a tool where developers can connect their own databases and query.
At the moment, to connect and query the respective database, I do the following:
Establish a connection to the DB given the credentials save
Query
Destroy the connection
This is unfortunately slow and takes about 1.5-2s to finish.
Another alternative is to save the connection in memory and reuse it when needed.
Thoughts on how to solve this?
I have followed this tutorial for sequelize transactions:
https://sequelize.org/master/manual/transactions.html
It does work, but I have noticed that for every command in that code sequelize sends a query to the database server and waits for a response.
The reasons this is a problem is because my database server is not really close to my node server, so every round trip can be expensive.
This seems extremely wastefull. Since all sequelize actually does is turn js code into MySQL syntax, wouldn't it be more efficient to just construct the query and then send it all at once to the database? Or at least, avoid sending a query when sequelize.transaction() is called and send "Start Transaction" on the subsequent command.
So basically is there a way to create a transaction with multiple lines of query code and then send that all at once to the database?
Unfortunately, what you wanna do seems like anti-pattern of ORM, which includes sequelize.js. ORM is functional way to manage database connection not the query itself.
You might wanna use query builer instead. check this out. http://knexjs.org/
Difference between query builder and ORM
ORM provides you the connection with database and you can run the query using APIs. Query is black-boxed to the user in most of cases.
Query builder API doesn't run the query until the user put the generated query into database connection.
Also, transaction is to be used rollback and commit for keeping same context(google atomicity) and can't put the queries together for relieving the network workload.
I would like to make sql/sp calls to multiple databases on the same server. I am using mssql and don't see the "changeUser" option that mysql provides. Do I have to close and reconnect every time I wish to change databases? Seems like this would be a commonly desired behaviour.
Simply having an sql connection for each database is not doable, as the error "Global connection already exists." will occur.
We have architecture problem on our project. This project requires sharding, as soon as we need almost unlimited scalability for the part of services.
Сurrently we use Node.js + MongoDb (Mongoose) and MySQL (TypeORM). Data is separated by databases through the simple 'DB Locator'. So node process needs connections to a lot of DBs (up to 1000).
Requests example:
HTTP request from client with Shop ID;
Get DB IP address/credentials in 'DB Locator' service by Shop ID;
Create connection to specific database with shop data;
Perform db queries.
We tried to implement it in two ways:
Create connection for each request, close it on response.
Problems:
we can't use connection after response (it's the main problem, because sometimes we need some asynchronous actions);
it works slower;
Keep all connections opened.
Problems:
reach simultaneous connections limit or some another limits;
memory leaks.
Which way is better? How to avoid described problems? Maybe there is a better solution?
Solution #1 perfectly worked for us on php as it runs single process on request and easily drops connections on process end. As we know, Express is pure JS code running in v8 and is not process based.
It would be great to close non-used connections automatically but can't find options to do that.
The short answer: stop using of MongoDB with Mongoose 😏
Longer answer:
MongoDB is document-oriented DBMS. The main usage case is when you have some not pretty structured data that you have to store, but you don't need to use too much. There is lazy indexing, dynamic typing and many more things that not allow you to use it as RDBMS, but it is great as a storage of logs or any serialized data.
The worth part here is Mongoose. This is the library that makes you feel like your trashbox is wonderful world with relations, virtual fields and many things that should not to be in DODBMS. Also, there is a lot of legacy code from previous versions that also make some troubles with connections management.
You already use TypeORM that may works instead Mongoose. With some restrictions, for sure.
It works exactly same way as MySQL connection management.
Here is some more data: https://github.com/typeorm/typeorm/blob/master/docs/mongodb.md#defining-entities-and-columns
In this case you may use you TypeORM Repository as transparent client that will init connections and close it or keep it alive on demand.
in my application I have a default database and other database I have to connect to in function of client's requests , since with mongoose in node as far as I understood: there is a pool of connections application wide, if I change database, it is changed for all the subsequent requests, I think it could cause some problems, what is the best way to switch Database with mongoose?
Mongoose 3.7.1 (unstable) supports switching databases.
Otherwise you'll need to create separate connection instances for each database.