I am trying to create a simplistic TCP client and server. Conceptually, I know that a TCP socket is the same on both the client and server side (at least, this is how it is in python). However, the steps after creating a socket are different. Ie, for clients, the socket establishes a TCP connection to the server. On the server side, the socket is bound to a specific port, and waits for connections, and when it gets a req, it creates a new connected socket. (correct me if I got anything wrong, I'm new to networking).
My question is if there's any difference between the net.Socket and net.Server classes. Did node.js separate the two, and net.Server is explicitly meant for servers? Is it still possible to use net.Socket to make the TCP server socket?
What is the difference between net.createServer() and net.createConnection()?
net.createConnection() initiates an outbound TCP connection to some other host or server.
net.createServer() sets up a server that will accept incoming TCP connections from other hosts or processes.
These are opposite ends of enabling a connection.
My question is if there's any difference between the net.Socket and net.Server classes.
Yes, there's a huge difference between them as neither is a substitute for the other. A server listens for inbound connections.
A client then creates a TCP socket and attempts to connect to a server that is listening for inbound connections on the port and IP address that the server is listening on. During the connection process, the server follows the TCP handshake process to enable the creation of a TCP socket that connects the client and server. That TCP socket is then bidirectional so either end can then send data to the other.
Nodejs uses the net.Socket class as the nodejs object to represent a TCP socket so when you initiate a connection from a client, you get a net.Socket object that represents your TCP connection to some other server. When you are a server and someone connects to you, you get a net.Socket object that represents your TCP connection to the client that connected to you. Those two objects are different ends of a TCP connection and both ends do not have to be nodejs endpoints - they can be any language or tool that can make a standard TCP connection.
Did node.js separate the two, and net.Server is explicitly meant for servers? Is it still possible to use net.Socket to make the TCP server socket?
Yes, net.Server is exclusively for servers to set up a listener for inbound connections on a specific port on your host.
net.socket by itself cannot listen to incoming connections (you use an instance on net.Server for that. It is either used to initiate a TCP connection to some server or it is created as part of of some client connecting to your server.
Related
I'm confused about how TCP packets are routed on the OS level after TCP connection has been established.
Suppose a sever is listening passively on a TCP port for incoming traffic. The server receives a connection request, accepts it, and created a socket for source_ip:port and destination_ip:port, e.g. (x.x.x.x:9999, 127.0.0.1:8080).
How are subsequent packets sent from x.x.x.x:9999 to 127.0.0.1:8080 routed on the server? Does the OS route them to the process binding port 8080, which in turn routes them to file descriptor with source x.x.x.x:9999? If yes, does this mean that each application much implement logic to route packets of established connections to the appropriate file descriptor?
Or does the OS route them directly to the open socket? If yes, how does the OS know which socket to route to since incoming request use well known port number (8080 in this case)?
I read the docs, concerning the .listen() method, used in express. I can USE the method and setup a server that is listening to HTTP requests.
However, since I am fairly new to coding, I find it difficult to grasp whats really happening when using the .listen() method. The high level explanation "listening for connections" didn't help me.
I think, this could be made easier if I could actually see the function instead of only calling it.
Any help is very much appreciated
In a nutshell, the Express app.listen() method creates an http server object and then configures it to receive incoming TCP connections on a specific port and IP address so that when clients request a connection to that port and send an http request, the server can receive that http request and process it, sending a response. The code in app.listen() is shown below later in the answer - though all it does is call down to one further layer down in the http server object.
Here are the lower level details for how that works.
When a server wishes to start listening for incoming connections, it informs the local TCP stack by creating a socket and binding to a particular port and IP address. That essentially reserves that incoming port for this particular server (no other server will be allowed to also bind to that port). So, for example, on a regular http server on the default port, you would bind to port 80. This type of bound socket is used for incoming connections only, not for two-way communications with a client.
Then, the server informs the TCP stack that it is ready for incoming connections. At the TCP level, this is referred to as listen. Within nodejs, the bind and listen steps are combined into the one step called listen.
From then on, whenever the local TCP stack receives an incoming connecting request whose destination is the IP address and port that the server bound to, then that incoming connection will be accepted and inserted into a queue for the server that is configured for that IP address and port. There will typically be a maximum number of incoming connections that can be queued in this way and, if that number is exceeded, then the connection will be refused. This manages load and protects the host if the server gets "backed up" and is behind on processing incoming connections.
The server will then be informed by the TCP stack for each new incoming connection. Once the server accepts that connection, then it can start reading any data that the client has sent over the socket. In the case of an HTTP server working with the HTTP protocol, this would be the initial request protocol, method, version, headers and any body data. For different types of servers, the data would be in a different format.
Here's a useful diagram of the server:
Source: https://medium.com/javarevisited/fundamentals-of-socket-programming-in-java-bc9acc30eaf4
The server creates a socket used for the server to accept new connections..
It binds that socket to a specific IP address and port so it will only be informed about incoming connections targeted to that IP address and port.
It listens on that port to inform the TCP stack it is ready to accept incoming connections.
When it is notified of an incoming connection, it accepts that incoming connection.
Then it can read and write to that new connection over the new socket.
Then, sometime later, the incoming socket is closed to complete the client transaction.
The app.listen() method in Express encapsulates these steps and a few others. Internally (within Express), the code looks like this:
app.listen = function listen() {
var server = http.createServer(this);
return server.listen.apply(server, arguments);
};
You can see that method here in the open source repository.
To get an http server ready for steps 1-6 above, this creates the http server object within nodejs and then registers the app as the request listener for that server object (so it will be notified of incoming http requests).
Then, the call to server.listen() encapsulates steps 1-3 above.
Step 4 happens inside the http server object implementation and the app object is called when a new connection has been established and a new HTTP request is available. The http server reads the initial request and parses the http protocol and that initial request is already made available to the app for routing to the appropriate handler.
Then, subsequent calls such as res.send() or res.json() write a response back on the http socket and close the socket or res.end() will close it directly (steps 5 and 6 above).
Some other useful references:
Why is bind() used in TCP? Why is it used only on server side and not in client side? - Helps explain how a port and IP address define the TCP endpoint represented by a server. This port has to be known by the client so it can specifically request to connect to that port. The client end of the socket also has an IP address and a port, but its port can be dynamically assigned, thus the client does not have to bind to a specific port itself. The four pieces of data [server IP, server port, client IP, client port] define a specific TCP connection.
How TCP sockets work - has a good section about how new connections to a server work.
Understanding socket and port in TCP - talks about active and passive sockets. Passive sockets are sockets in "listen" mode used to accept incoming connections. Active sockets are two-way communications channels between two TCP endpoints.
Transmission Control Protocol (TCP) - more details on the various aspects of TCP from initiating a listening server, initiating a client connection to that server, through packet transmission to closing the socket.
There are a gazillion other references on the topic on the web. You can probably find 1000 articles on any single aspect of TCP that you might want more info about.
I think, this could be made easier if I could actually see the function instead of only calling it.
The underlying code for listen is inside the operating system's TCP stack and is not part of nodejs or Express. Express relies on the nodejs http server object as its interface to that and the nodejs http server object uses native code (built into nodejs) to call libuv (which is a cross platform C library that nodejs uses for networking and other things). Then, libuv talks to the underlying operating system APIs to reach the actual TCP stack on that target host. All of this is to put the server socket into listen mode so it can be notified of new incoming client connections to that target IP address and port.
Here's some doc on the related portions of the Linux TCP API if you want to see what the underlying TCP interface and description of that interface is:
socket() - https://linux.die.net/man/7/socket
bind() - https://linux.die.net/man/2/bind
listen() - https://linux.die.net/man/2/listen
And, portions of the libuv library that nodejs uses for networking:
TCP handles - http://docs.libuv.org/en/v1.x/tcp.html
Server listen() and accept() - http://docs.libuv.org/en/v1.x/stream.html#c.uv_listen
I have a tcp server running. A client connects to the server and send packet periodically. For TCP server, this incoming connections turns to be CONNECTED, and the server socket still listens for other connections.
Say this client suddenly get powered off, no FIN sent to server. When it powers up again, it still use the same port to connect, but server doesn't reply to SYNC request. It just ignores incoming request, since there exists a connection with this port.
How to let server close the old connection and accept new one?
My tcp server runs on Ubuntu 14.04, it's a Java program using ServerSocket.
That's not correct, a server can accept multiple connections and will accept a new connection from a rebooted client as long as it's connecting from a different port (and that's usually the case). If your program is not accepting it it's because you haven't called accept() a second time. This probably means that your application is only handling one blocking operation per time (for example, it might be stuck in a read() operation on the connected socket). The solution for this is to simultaneously read from the connected sockets and accept new connections. This might be done using an I/O multiplexer, like select(), or multiple threads.
I intend to deploy a nodejs app on Heroku which is both an HTTP and TCP server. I can see that I can map my application to a routed port using process.env.PORT. However, this would be just one port, yes? I couldn't map both my HTTP server and TCP server to the same port. Is there a way that I can do this, possibly by getting a second routed port?
Please note, my TCP client applications are not necessarily going to be nodejs (probably Python), so I need something lower level than socket.io and websockets. I was going to use net.
TCP and HTTP are in different layers.
HTTP is under the Application Layer.
TCP is under the Transportation Layer.
An HTTP client initiates a request by establishing a Transmission Control Protocol (TCP) connection to a particular port on a server (typically port 80).
In computer networks, every application is getting/asking from the operating system a port that it can listen to.
If you have 2 TCP servers- one is HTTP server and the other one is another server, they can't listen to the same port, unless you have two NI because of the TCP protocol operations.
I would like to write a daemon that is available on a large number of AF_INET-SOCK_STREAM and AF_INET-SOCK_DGRAM sockets for network debugging purposes.
To avoid excessive resource usage I want to avoid opening a large number of ports on the application layer but try to multiplex the connections per socket type on lower layers.
Knowledge of the original incoming port on the application layer is a requirement.
I have successfully implemented a daemon that listens on an AF_INET SOCK_STREAM socket that is multiplexed by an iptables REDIRECT rule. The original incoming port of the connection can be retrieved by calling getsockopt with SO_ORIGINAL_DST. As I understand this does not work with AF_INET SOCK_DGRAM.
I have also successfully implemented a daemon that listens on an AF_INET SOCK_DGRAM socket that is multiplexed by an iptables TPROXY rule. The original incoming port of the connection can be retrieved by using recvmsg() and consuming the available ancillary message containing information about the connection before multiplexing. As I understand this does not work with AF_INET SOCK_STREAM.
Is there a transport-layer-agnostic way of multiplexing such socket connections and retrieving information about the original incoming port? Possibly even suitable for protocols like SCTP or DCCP?
With TCP you have a connection. The target port for this connection is the same for all packets inside this connection. In this case each connected socket (result from accept) equals to a single connection and the incoming port is a property of this socket. It does not matter in this case if the listening socket will accept connections on multiple ports, all what matters is the connected socket.
With UDP you don't have a connection. Instead the same socket is used to receive packages from multiple clients and in your case to multiple incoming ports. Source and destination IP and port are thus a property of each packet and not of the socket.
That's why you need different interfaces to retrieve the original incoming port: a socket based for TCP and a packet based for UDP.