This applies to non-user facing backend applications communicating with each other through HTTP. I'm wondering if there is a guideline for a maximum timeout for a synchronous HTTP request. For example, let's say a request can take up to 10 minutes to complete. Can I simply create a worker thread on the client and, in the worker thread, invoke the request synchronously? Or should I implement the request asynchronously, to return HTTP 202 Accepted and spin off a worker thread on the server side to complete the request and figure out a way to send the results back, presumable through a messaging framework?
One of my concerns is it safe to keep an socket open for an extended period of time?
How long a socket connection can remain open (without activity) depends on the (quality of the) network infrastructure.
A client HTTP request waiting for an answer from a server results in an open socket connection without any data going through that connection for a while. A proxy server might decide to close such inactive connections after 5 minutes. Similarly, a firewall can decide to close connections that are open for more than 30 minutes, active or not.
But since you are in the backend, these cases can be tested (just let the server thread handling the request sleep for a certain time before giving an answer). Once it is verified that socket connections are not closed by different network components, it is safe to rely on socket connections to remain open. Keep in mind though that network cables can be unplugged and servers can crash - you will always need a strategy to handle disruptions.
As for synchronous and asynchronous: both are feasable and both have advantages and disadvantages. But what is right for you depends on a whole lot more than just the reliability of socket connections.
Related
What happens to incoming requests when a nodejs server is blocked? There are times when the server will be blocked because it is chewing through something computationally expensive, or perhaps doing some synchronous IO (e.g. writing to a sqlite database). This is best described with an example:
given a server like this:
const { execSync } = require('child_process')
const express = require('express')
const app = express()
const port = 3000
// a synchronous (blocking) sleep function
function sleep(ms) {
execSync(`sleep ${ms / 1000}`)
}
app.get('/block', (req, res) => {
sleep(req.query.ms)
res.send(`Process blocked for ${req.query.ms}ms.`)
})
app.get('/time', (req, res) => res.send(new Date()))
app.listen(port, () => console.log(`Example app listening at http://localhost:${port}`))
I can block the nodejs process like so:
# block the server for two seconds
curl http://localhost:3000/block\?ms\=2000
and while it is blocked attempt to make another request to the server:
curl http://localhost:3000/time
the second request will hang until the blocking call is completed, and then respond with the expected datetime. My question is, what specifically is happening to the request while the nodejs process is blocked?
Does node read in the request using some low level c++ and put it into a queue? Is backpressure involved here?
Is the unix kernel involved here? Does it know to put a request on some kind of queue while a server refuses to respond?
Is it just as simple as curl waiting on a response from a socket indefinitely?
What happens if the server is blocked and 10,000 new requests hit the server? Will they all be serviced as soon as the server becomes unblocked? (assuming there is no load balancer or other timeout mechanisms in between the client & server)
Finally, I understand that blocking nodejs is bad practice but I am not asking about best practices. I want to understand what nodejs does under stressful circumstances like those described here.
In the OS, the TCP stack has a queue for incoming data or connections that is waiting to be picked up by the appropriate host application if the host application is too busy to pick it up right now. Depending upon OS and configuration, that inbound queue will fill up at some point and clients attempting to connect would get an error. I'm not aware of any separate thread in nodejs that picks these up into its own queue and there probably isn't any reason for nodejs to do so as the TCP stack already implements an inbound connection queue on its own.
If you're blocking the nodejs process long enough for 10,000 incoming requests to arrive, you have much bigger problems and need to solve the blocking problem at its core. Nodejs has threads, child processes and clustering all of which can be employed as relief for a blocking calculation.
For data sent on an existing, already-opened TCP connection, there is back pressure (at the TCP level). For new incoming connections, there really isn't such a thing as back pressure. The new incoming connection is either accepted or its not. This is one cause of what we sometimes observe as ERR_CONNECTION_REFUSED.
Some related discussion here: What can be the reason of connection refused errors.
Does node read in the request using some low level c++ and put it into a queue? Is backpressure involved here?
Node itself does not do this (that I'm aware of). The OS TCP stack has a queue for inbound data and incoming connection requests.
Is the unix kernel involved here? Does it know to put a request on some kind of queue while a server refuses to respond?
The TCP stack (in the OS) does have a queue for both incoming data arriving on an existing connection and for inbound connection requests. This queue is of a finite (and partially configurable) size.
Is it just as simple as curl waiting on a response from a socket indefinitely?
No. If the queue for inbound connection requests on the server is full, the connection request will be rejected. If the queue is not full, then it is just a matter of waiting long enough for it to succeed. Most client-side libraries will use some sort of timeout and give up after a time in case something happened that causes there to never be a response sent back.
What happens if the server is blocked and 10,000 new requests hit the server? Will they all be serviced as soon as the server becomes unblocked? (assuming there is no load balancer or other timeout mechanisms in between the client & server)
The target host will queue the inbound connection requests up to some limit (which varies by OS and configuration) and then will reject ones that come after that.
Some other relevant articles:
How TCP backlog works in Linux
What is "backlog" in TCP connections?
TCP Connection Backlog and a Struggling Server
The more of these types of articles you read, the more you will also discover a tradeoff between quickly accepting lots of connections and defending against various types of DOS attacks. It seems a balance has to be drawn.
I am making a tracking system and i would like to know, if i have 1000 cars (clients) transmitting via sockets(tcp) at an interval of 5 seconds. Should the client open ,send then close the socket. Or should client keep the socket open though out as it transmits.
Depends on many things. For example, if there is a maximum number a server can handle sockets at same time, then you better close them in case you are going to have lots of requests. At the same time, if a live and fast connection really matters to you (1 request per 5 sec is normal, not too high not too low in my opinion) then live socket connections are better for you. Note that they also give you power in server side to broadcast messages to clients at any times, while with none persistent connections you have to broadcast messages as response to each 5 second request.
The tags you used suggests me you are trying to choose between websocket or HTTP. Finally, I should clarify that it really depends on your needs. With HTTP you can serve your logic to more clients, while with websocket you have to deal with server loads a little harder while you have advantage of sending messages to clients and faster tracking, and handshake just happens once.
On the Server side for websockets there is already an ping/pong implementation where the server sends a ping and client replies with a pong to let the server node whether a client is connected or not. But there isn't something implemented in reverse to let the client know if the server is still connected to them.
There are two ways to go about this I have read:
Every client sends a message to server every x seconds and whenever
an error is thrown when sending, that means the server is down, so
reconnect.
Server sends a message to every client every x seconds, the client receives this message and updates a variable on the client, and on the client side you have a thread that constantly checks every x seconds which checks if this variable has changed, if it hasn't in a while it means it hasn't received a message from the server and you can assume the server is down so reestablish a connection.
You can achieve trying to figure out on client side whether the server is still online using either methods. The first one you'll be sending traffic to the server whereas the second one you'll be sending traffic out of the server. Both seem easy enough to implement but I'm not so sure which is the better way in terms of being the more efficient/cost effective.
Server upload speeds are higher than client upload speeds, but server CPUs are an expensive resource while client CPUs are relatively cheap. Unloading logic onto the client is a more cost-effective approach...
Having said that, servers must implement this specific logic (actually, all ping/timeout logic), otherwise they might be left with "half-open" sockets that drain resources but aren't connected to any client.
Remember that sockets (file descriptors) are a limited resource. Not only do they use memory even when no traffic is present, but they prevent new clients from connecting when the resource is maxed out.
Hence, servers must clear out dead sockets, either using timeouts or by implementing ping.
P.S.
I'm not a node.js expert, but this type of logic should be implemented using the Websocket protocol ping rather than by your application. You should probably look into the node.js server / websocket framework and check how to enable ping-ing.
You should set pings to accommodate your specific environment. i.e., if you host on Heroku, than Heroku will implement a timeout of ~55 seconds and your pings should be sent before this timeout occurs.
I'm trying to set up a server that can handle a high sustained amount of simultaneous requests. I found that at a certain point, the server won't be able to recycle "old" TCP connections quickly enough to accommodate extreme amounts of requests.
Do websockets eliminate or decrease the amount of tcp connections that a server needs to handle, and are they a good alternative to "normal" requests?
Websockets are persistent connections so it really depends on what you're talking about. The way socket.io uses XHR is different from a typical ajax call in that it hangs onto the request for as long as possible before sending a response. It's a technique called long-polling and It's trying to simulate a persistent connection by never letting go of the request. When the request is about to timeout it sends a response and a new request is initiated immediately which it hangs onto yet again, and the cycle continues.
So I guess if you're getting flooded with connections because of ajax calls then that's probably because your client code is polling the server at some sort of interval. This means that even idle clients will be hitting your server with fury because of this polling. If that's the case then yes, socket.io will reduce your number of connections because it tries to hang onto one single connection per client for as long as possible.
These days I recommend socket.io over doing plain ajax requests. Socket.io is designed to be performant with whatever transport it settles on. The way it gracefully degrades based on what connection is possible is great and means your server will be overloaded as little as possible while still reaching as wide an audience as it can.
I am using node.js Request module to make multiple post requests.
Does this module have a connection pool ?
Can we manage this connection pool ?
can we close open connections ?
How do we handle the socket hang up error
Request does not have a connection pool. However, the http module (which request uses) does:
In node 0.5.3+ there is a new implementation of the HTTP Agent which is used for pooling sockets used in HTTP client requests.
By default, there is a limit of 5 concurrent connections per host. There is an issue with the current agent implementation that causes hang up errors to occur when you try to open too many connections.
You can either:
increase the maximum number of connections: http.globalAgent.maxSockets.
disable the agent entirely: pass {pool: false} to request.
There are several reasons for having an HTTP agent in the first place:
it prevents you from accidentally opening thousands of connections to a host (would be perceived as an attack).
connections in the pool will be kept opened for HTTP 1.1 keepalive.
most of the time, maxSockets really depends on the host you're targetting. node.js will be perfectly happy to open 1000 concurrent connections if the other host handles it.
The behavior of the agent is explained in the node.js doc:
The current HTTP Agent also defaults client requests to using Connection:keep-alive. If no pending HTTP requests are waiting on a socket to become free the socket is closed. This means that node's pool has the benefit of keep-alive when under load but still does not require developers to manually close the HTTP clients using keep-alive.
The asynchronous architecture of node.js is what makes it very cheap to open new connections.