I have followed this tutorial for sequelize transactions:
https://sequelize.org/master/manual/transactions.html
It does work, but I have noticed that for every command in that code sequelize sends a query to the database server and waits for a response.
The reasons this is a problem is because my database server is not really close to my node server, so every round trip can be expensive.
This seems extremely wastefull. Since all sequelize actually does is turn js code into MySQL syntax, wouldn't it be more efficient to just construct the query and then send it all at once to the database? Or at least, avoid sending a query when sequelize.transaction() is called and send "Start Transaction" on the subsequent command.
So basically is there a way to create a transaction with multiple lines of query code and then send that all at once to the database?
Unfortunately, what you wanna do seems like anti-pattern of ORM, which includes sequelize.js. ORM is functional way to manage database connection not the query itself.
You might wanna use query builer instead. check this out. http://knexjs.org/
Difference between query builder and ORM
ORM provides you the connection with database and you can run the query using APIs. Query is black-boxed to the user in most of cases.
Query builder API doesn't run the query until the user put the generated query into database connection.
Also, transaction is to be used rollback and commit for keeping same context(google atomicity) and can't put the queries together for relieving the network workload.
Related
I am testing postgresql with a nodejs backend server, using Pg npm module to query the database. The issue I am having is that when I run a particular query directly on the postgres database table using query tool on pgAdmin4, the data is fetched within 5 seconds. But the same query when requested from the backend through my nodejs server, the process is split between parallel workers and a client backend using IPC: messagequeuesend, this runs for almost 17minutes before return the data. I can't understand why the same query is fast using query tool, it just processes it fast but the one coming from my server has to delay. Is there a way to increase the priority for queries coming from backend to run like it was queried inside pgAdmin. I noticed when I check pg_stat_activity, there is an application value for the query when using query tool, but when the same query comes from the nodejs server the application value is null. I do not seem to understand why its like this, i have been searching every community for answers to this for the past 5 days, and there is no question or answer for this. Please any help will be appreciated. Thanks in advance
I tried running a query from the backend, but its split using IPC processes and result comes in after 17 minutes, the same query takes only 5 seconds to return a result inside pgAdmin query tool
I'm working on an API that sends messages contained in a SQL Server database.
The problem is that I implemented it in a way that every 10 seconds the API performs a query for messages not yet sent.
I would like to optimize this by making the SQL Server warn every time my table receives an insert, so that the application can query the messages to be sent.
For that I'm using node JS and importing Sequelize.
I also think it's important to comment that the inserts of this table are made by another application.
If your infrastructure has a message broker set up (it probably doesn't, but ask your operations crew) you can use an event notification for this.
Otherwise you're stuck polling your server, as you described in your question. That is a workable strategy, but it's worth hard work to make sure the polling query is very efficient.
We have architecture problem on our project. This project requires sharding, as soon as we need almost unlimited scalability for the part of services.
Сurrently we use Node.js + MongoDb (Mongoose) and MySQL (TypeORM). Data is separated by databases through the simple 'DB Locator'. So node process needs connections to a lot of DBs (up to 1000).
Requests example:
HTTP request from client with Shop ID;
Get DB IP address/credentials in 'DB Locator' service by Shop ID;
Create connection to specific database with shop data;
Perform db queries.
We tried to implement it in two ways:
Create connection for each request, close it on response.
Problems:
we can't use connection after response (it's the main problem, because sometimes we need some asynchronous actions);
it works slower;
Keep all connections opened.
Problems:
reach simultaneous connections limit or some another limits;
memory leaks.
Which way is better? How to avoid described problems? Maybe there is a better solution?
Solution #1 perfectly worked for us on php as it runs single process on request and easily drops connections on process end. As we know, Express is pure JS code running in v8 and is not process based.
It would be great to close non-used connections automatically but can't find options to do that.
The short answer: stop using of MongoDB with Mongoose 😏
Longer answer:
MongoDB is document-oriented DBMS. The main usage case is when you have some not pretty structured data that you have to store, but you don't need to use too much. There is lazy indexing, dynamic typing and many more things that not allow you to use it as RDBMS, but it is great as a storage of logs or any serialized data.
The worth part here is Mongoose. This is the library that makes you feel like your trashbox is wonderful world with relations, virtual fields and many things that should not to be in DODBMS. Also, there is a lot of legacy code from previous versions that also make some troubles with connections management.
You already use TypeORM that may works instead Mongoose. With some restrictions, for sure.
It works exactly same way as MySQL connection management.
Here is some more data: https://github.com/typeorm/typeorm/blob/master/docs/mongodb.md#defining-entities-and-columns
In this case you may use you TypeORM Repository as transparent client that will init connections and close it or keep it alive on demand.
I'm developing an loopback application to get data using oracledb npm module from ORACLE and convert it to JSON format to be stored in MONGODB.
MONGODB is accessed using "loopback-connector-mongodb".
The data to be stored would be around for 100 collections as of for 100 tables from ORACLE.
I'm pushing data with http request row by row for the entire collection list from node server from my local application to another server application on remote machine using http-request through remote method call.
When the data write operation the remote machine application stops throwing an error showing "EMFILE error".
I googled and some reference showed that it is because of the maximum number of opened files/sockets. Hence i tried disconnecting the DataSource for each request. Still i'm getting the same error.
Please help me on this!!
Thanks.
If you are making an http request for each row of data, and aren't throttling or otherwise controlling the order of those requests, it is possible you are simply making too many requests at once because of node's async io model.
For example, making those calls in a simple for loop would actually result in all of them being made in parallel.
If this is the case, you might want to consider using something like the async module, which includes some utilities for throttling the parallelism.
Don't forget Oracle Database 12.1.0.2 has JSON support. Maybe you don't need to move the data in the first place?
See JSON in Oracle Database. To quote the manual:
'Oracle Database supports JavaScript Object Notation (JSON) data
natively with relational database features, including transactions,
indexing, declarative querying, and views.'
I'm using socket.io to handle realtime connections on my site. When a client connects, though, I save out the connection to a mysql database along with other options like what url they are on. I'm using the socket.id as the primary key for that mySQL table as I assumed that would be unique. The purpose of the mySQL database is to make it easier to perform sorting later on. If there's a better (in-node) way to do that I'm all ears.
When the socket disconnect I remove the row from the mySQL table.
However, I seem to be getting some asynchronous issues here that I can't figure out a workaround. Since the Insert into mysql is asynchronous, it seems possible that a user may disconnect BEFORE mysql has inserted into the database. At which point the internals of socket.io and the mySQL database will be out of synch.
How would you go about keeping the two in synch? Is it worth BLOCKING on the mySQL Insert? Can node even do that? Or, is there a different approach?