I have a nodejs/ExpressJS/Mongodb/Mongoose app hosted on aws elasticbeanstalk.
The problem is elasticbeanstalk health degrades randomly ( no specific times ), that happens because any request that requires database interaction results the following in logs:
*1360931 upstream timed out (110: Connection timed out) while reading response header from upstream
This happens no matter how much data I try to load. it happens with the least amount of data, this can last from a minute up to 20 minutes and it works again on its own, it is completely random.
And I can force it to work immediately by restarting the environment ( I connect to mongodb using connection string on app startup ).
While other requests that don't require database interaction work 100%.
The thing is while database queries aren't working , I can connect to the same database from localhost and database requests work like a charm, they even work really fast.
What is even more strange is I have 4 other identical apps with the same setup, and This situation doesn't occur with any of them, only this app faces this problem !
What is the problem here ?
The above error usually means that your server closed the connection due to shorter timeout , but your application is not aware about it. You may need to check your connection string and modify the timeouts maybe decrease them , example connection string:
MONGO_URI=mongodb://user:password#127.0.0.1:27017/dbname?keepAlive=true&poolSize=30&autoReconnect=true&socketTimeoutMS=360000&connectTimeoutMS=360000
Related
I'm using Mongo Atlas with Prisma. Whenever I launch my project, it takes about 3 minutes for database to connect, and in this period of time, I can't see anything in Mongo Atlas' connection tab. I've tried Prisma's logger, no error prints at all. Interestingly, it connects but it just takes time.
It eventually connects, but this issue with connection time makes using hot reload quite hard. I've tried allowing incoming Node apps in Advanced Firewall Settings, and my firewall is disabled. Actually I couldn't find a lot of people experiencing this, so I couldn't try much to be honest. And to note, when somebody else tries to launch exact same code, their database starts working instantly.
I am using arangojs v6.14.1 and arangodb version 3.6.4.. I also have the nodejs express app which is intended to serve client requests.
I am experiencing an issue while executing concurrent requests. The database connection hangup when I concurrently process client requests:
What the app will do when it receives a request?
Open a database connection -
db = new Database(dbConfig.url);
db.useDatabase(dbConfig.name);
db.useBasicAuth(dbConfig.username, dbConfig.password);
There are multiple middleware functions that need to access to perform various functions and accessibility checks. And for each middleware I tried to
open a new database connection ->
perform the action ->
close the connection ->
return the result to next middleware.
This works fine with single request at a time. But if I tried to call two APIs at same time, I am getting CONNECTIONRESET error. And also throwing socket hangup error.
I tried to commend out the close connection method and it started working fine for a while. But when I increased the number of connections, it again showing the same error as "CONNECTIONRESET".
I have searched the documentation of arangojs regarding the connection manipulation. But I haven't found any information regarding the same.
Any help would be deeply appreciated.
This might be too late for you, but I had a similar issue. After talking with ArangoDB support, it turned out to be an issue with sockets.
Looking at the output from netstat -aplnt I was seeing a lot of CLOSE_WAIT and TIME_WAIT messages, indicating that there were A LOT of connections that were not being closed properly.
The solution was to enable persistent connections (see arangojs config for keep-alive and maxSockets) and then re-using a single connection whenever possible. This was more than just a settings change, requiring some code modifications as well.
I've been working with MongoDB for a while now and I've been liking it a lot. One thing I do not understand however is "Connections". I've searched online and everything just has very vague and basic answers. I'm using MongoDBs cloud service called "Atlas" and it describes the connection count as
The number of currently active connections to this server. A stack is allocated per connection; thus very many connections can result in significant RAM usage.
However I have a few questions.
What is a connection I guess? As I understand it, a connection is made between the server and the database service. Essentially when I use mongoose.connect(...);, a connection is made. So at most, there should only be one connection. However when I was testing my program I noticed my connection count was at 2 and in some moments it spiked up all the way to 7 and went to 5 and fluctuated. Does a "connection" have anything to do with the client? On the dashboard of Atlas it says I have a max connection amount of 500. What does this value represent? Does this mean only 500 users can use my website at once? If that's the case, how can I increase that number? Or how can I make sure that more than 500 connections never get passed? Or is a connection something that gets opened and I have to manually close myself? Because I've been learning from tutorials and I've never seen/heard anything like that.
Thanks!
mongoose.connect doesn't limit itself to 1 connection to the Mongo Server.
By default, mongoose creates a pool of 5 connections to Mongo.
You can change this default if necessary.
mongoose
.connect(mongoURI, {poolSize : 200});
See https://mongoosejs.com/docs/connections.html
More number of connections which you see in Atlas because there are some internal connections are also made in order to make the cluster running, these may include the connections from:
Connections made from a client.
Internal connections between primary and secondaries.
As it is a hosted service and everything is being monitored so connections from the monitoring agent.
As automation works, so the connections from the automation agent as well.
Hence whenever a new cluster is being created in Atlas, you will always see some connections in the metrics Page even though no client is being connected.
Does Meteor have a setting to timeout and retry if its MongoDB does not give a response in x seconds? Wondering if anyone has tried this.
I am interested in running a MongoDB database remote to the Meteor production app. The Meteor-to-Mongo connection will be quick, just 3-9 milliseconds away, but I also want to understand how Meteor (and NodeJS) would react to a brief network outage. Would the app hang while waiting for a long timeout period? How can I force a 1 second timeout/retry to avoid a hang?
You can specify timeout in the mongo URL:
MONGO_URL=mongodb://host:port/db?connectTimeoutMS=60000&socketTimeoutMS=60000
but let's say you have a network outage, what does a short timeout give you?
your app will hang anyways...
To get high availability, look into replica sets.
https://docs.mongodb.com/manual/tutorial/deploy-replica-set/
We're running Node (v0.10.38) with Express (4.0.0), proxied through nginx (1.2.1), which usually works great. Recently, however, we switched to a new server setup. Now, roughly 30 minutes after starting up the server, the server starts returning 504s (Gateway Timeout). Accessing Node directly from the server (bypassing nginx) also times out. Every so often, we got a series of ETIMEDOUT errors from redis, but connecting to the redis server from the server works from the command line. Furthermore, the server started returning 504s even before redis errors came up anyways. Anyways, after updating our redis middleware (connect-redis) to the newest version, these errors stopped, but the 504s still occurred. However, after disabling the connection to redis in our code for 10 hours, no 504 occurred. We've tried sending a redis ping periodically to prevent the error, believing that to be the cause, but 504s continue. When not connecting to redis, the server doesn't 504, so it is likely tied to redis in some way. Anything else we can try?
Sorry if there's not much to work with. We don't have that much either, and are eager to solve this issue as soon as possible. If there's any more specifics needed, I can update the question. Thank you.
Still don't know the root cause, but we ended up fixing this by pinging Redis every minute so that the connection wouldn't get killed.