I am developing an Angular Web App that receives its data from a nodejs/express API.
This API runs mongoose that connect to MongoLab (the free account).
When receiving data, I experience a response time > 500ms for small data sets (1.5kb) and > 1s for "large" data sets (hundreds of kb).
This is clearly already too much and I am affraid it will be even worse when my db will grow.
The current process is as follow:
Client goes to mysite.com/discover
Server send the Angular App
Client does an ajax request to mysite.com/api/collections
Server connects to MongoLab, receives data
Server send back data to client
This process is very fast in local development (local node, local MongoDB) (<20ms) but takes so much time when put online. I investigated what was taking so much time and I found two equal contributions:
API response time
MongoLab response time
The MongoDB query takes no time (<1ms).
The Question
What are my options to reduce this response time? Is it possible to store locally the data and use mongoLab as a "copy" (it would remove the MongoDB latency in most cases)? If so, would you suggest disk temporary storage, mongoDB replica, ...?
What I tried
I migrated my mongoLab DB to match the physical localization of my server (VM on digitalocean), it improve by a few 50ms, not much more.
Thanks a lot
Related
I have a MongoDB database, Node.js server and User Interface.
The connection between the three is working and data is being sent between them.
I am creating a real-time web application and I am using a setTimeout() feature to request data from database via the client's webpage every 4 seconds.
BUT the data that is coming back is the same everytime and isnt updating from the database.
It is acting as if there is only one MongoDB session and just sending the same set of data that is collected from when the server starts.
On the Node.js server code, I made sure that a connection is opened and close once the query has completed but it is collecting the same set of data every time and no new data that has come into the database. The only way I can get it to update the data from the database is to turn the server on and off to reconnect to the database.
Would using some thing like socket.io be better for the real time requests or is it possible to do real time data just with a setTimeout() feature on the client side and normal MongoDB collection queries?
Is there a way to constantly refresh the connection to the database?
Issue
I have an Express (Node.JS) + MongoDB app with a server response load time of 4 - 7 seconds on average (slow).
I understand that the average server response time is under 200ms as per google pagespeed tools.
This app is fetching data from a mongoDB asynchronously but the roundtrip times to the database is extremely slow with each call averaging about 500ms - 1s. These calls are simple findAll calls to retrieve data of less than < 100 records.
Context
Mongoose version: 4.13.14
DB server's MongoDB version is 3.4.16
DB server is hosted on MongoDB Atlas M10 in AWS / Oregon (us-west-1)
Web server is hosted with now.sh in SFO1 (us-west-1)
Have performed recommended indexes as advised by MongoDB Atlas's performance advisor
Data fetching perfectly fine in local environment (local server + local db) as data is queried in a matter of few ms
Mongoose logs for the affected page can be found in this gist
Mongo Server configuration
Mongo Atlas M10
2GB Ram
10 GB Storage
100 IOPS
Encrypted
Auto-expand storage
Attempted solutions:
I have checked my DB metrics, they looked fine. There are also no slow queries. These are simple findAll queries. Performance advisor on mongo atlas reports nothing unusual.
The production application and database are both hosted in the same region.
I have already tried optimising the application layer of the query (mongoose) by running .lean()
Question:
Where else should i look to improve the database latency? How can a simple query take so long? Otherwise, why is my server response time taking up to 4s when the expected is about 200ms?
Hey you can try hosting your server and database in the same region. I think the network is creating a overhead in this case. If the server and the database are in the same region, They are on the same network which will reduce the latency significantly. there is a diagram on aws for this
I add some problem like yours with an app that i developed in my master degree. I add to put a node.js api running online to present it in class room.And i realized that every time i wanted to make a call in the api the response was taking allot of time. I realized that one of the problems was the school network because of the firewalls. Also the place where i put the server heroku.com was giving some delay as well. What i did was use Redis ( https://redis.io/ ) to improve the performance, also heroku was giving me some problems because of the requests being http and not https.
Make a test running the app and data on your localhost and see the performance. if you don´t have any issue try to check if nothing is messing with your request like the place where you host your node server.
Let me know if this helps or if you still have issues so i can try to help you out better.
I had the same issue once with my nodejs code using the same development stack(mongodb,nodejs), I got into trouble of late response from api, and after spending a lot of time I found my server the real culprit I then changed from heroku to amazon aws EC2 instance and things started working fast and amazingly fast, so probably
your web server is culprit
to make sure mongodb is not culprit, write an api endpoint where you can just return some json response without making any query to database.
I have a app which lets yoy keep your notes at a single place its realtime bw all the devices you are logged in I am using a nodejs wesocket it was working fine but a recently i found out someone was sending a huge amount of requests to my websocket server. He sent a large amount of data through websockets to my mongodb and the data was sent just for the purpose of taking the app down (useless crap data just had 'aaaaa')
What i want is prevent those clients from using the websockets who are making more than 10requests per minute.
As mentioned in the comments its better to go with services like CloudFlare, but for your specific use case (to implement directly on server) you should look at ways to rate limit the requests.
Here is an example of an library to rate limit web-sockets in node
https://www.npmjs.com/package/ws-rate-limit
I am developing a Twitter app that (on the backend) consumes Tweets does some fairly intense processing and then stores the data in a database for use later by the client. All of my servers are running node.js.
I am going to have a server connected to the Twitter Streaming API using nTwitter for node.js. I want to then have this server pass the Tweets along to worker servers and distribute the load based on the Tweet ID (the last digit in the ID would be used).
Right now, I am using Socket.io (and Socket.io-client) which seems to run pretty well. It seems like the Websocket protocol is ideal for this. I am wondering if there are there any reasons not to use Socket.io in this manner?
With the free account on Heroku and the free account on MongoLab (not with the Heroku plug-in) I get response time of ~1000ms per request (single user, it is just me still, relevant to all requests, not only first one after a long idle time).
I've checked from my own computer + the same free MongoLab account and I get ~168ms per the same type of requests.
While it is still high, I want to ask regarding Heroku. Is it reasonable to have such poor response time, even with the free account.
Will the response time go significantly better when I pay them?
Mongoose, MongoDB (node.js) Native Driver.
Do you have any idea for me what to check?
MongoLab helped me realize my database is defined in Europe, while the server (Heroku) is in the US. They also told me how to clone the existing database to a new one in us-east-1 (from their web console).