I intend to deploy a nodejs app on Heroku which is both an HTTP and TCP server. I can see that I can map my application to a routed port using process.env.PORT. However, this would be just one port, yes? I couldn't map both my HTTP server and TCP server to the same port. Is there a way that I can do this, possibly by getting a second routed port?
Please note, my TCP client applications are not necessarily going to be nodejs (probably Python), so I need something lower level than socket.io and websockets. I was going to use net.
TCP and HTTP are in different layers.
HTTP is under the Application Layer.
TCP is under the Transportation Layer.
An HTTP client initiates a request by establishing a Transmission Control Protocol (TCP) connection to a particular port on a server (typically port 80).
In computer networks, every application is getting/asking from the operating system a port that it can listen to.
If you have 2 TCP servers- one is HTTP server and the other one is another server, they can't listen to the same port, unless you have two NI because of the TCP protocol operations.
Related
I'm confused about how TCP packets are routed on the OS level after TCP connection has been established.
Suppose a sever is listening passively on a TCP port for incoming traffic. The server receives a connection request, accepts it, and created a socket for source_ip:port and destination_ip:port, e.g. (x.x.x.x:9999, 127.0.0.1:8080).
How are subsequent packets sent from x.x.x.x:9999 to 127.0.0.1:8080 routed on the server? Does the OS route them to the process binding port 8080, which in turn routes them to file descriptor with source x.x.x.x:9999? If yes, does this mean that each application much implement logic to route packets of established connections to the appropriate file descriptor?
Or does the OS route them directly to the open socket? If yes, how does the OS know which socket to route to since incoming request use well known port number (8080 in this case)?
I read the docs, concerning the .listen() method, used in express. I can USE the method and setup a server that is listening to HTTP requests.
However, since I am fairly new to coding, I find it difficult to grasp whats really happening when using the .listen() method. The high level explanation "listening for connections" didn't help me.
I think, this could be made easier if I could actually see the function instead of only calling it.
Any help is very much appreciated
In a nutshell, the Express app.listen() method creates an http server object and then configures it to receive incoming TCP connections on a specific port and IP address so that when clients request a connection to that port and send an http request, the server can receive that http request and process it, sending a response. The code in app.listen() is shown below later in the answer - though all it does is call down to one further layer down in the http server object.
Here are the lower level details for how that works.
When a server wishes to start listening for incoming connections, it informs the local TCP stack by creating a socket and binding to a particular port and IP address. That essentially reserves that incoming port for this particular server (no other server will be allowed to also bind to that port). So, for example, on a regular http server on the default port, you would bind to port 80. This type of bound socket is used for incoming connections only, not for two-way communications with a client.
Then, the server informs the TCP stack that it is ready for incoming connections. At the TCP level, this is referred to as listen. Within nodejs, the bind and listen steps are combined into the one step called listen.
From then on, whenever the local TCP stack receives an incoming connecting request whose destination is the IP address and port that the server bound to, then that incoming connection will be accepted and inserted into a queue for the server that is configured for that IP address and port. There will typically be a maximum number of incoming connections that can be queued in this way and, if that number is exceeded, then the connection will be refused. This manages load and protects the host if the server gets "backed up" and is behind on processing incoming connections.
The server will then be informed by the TCP stack for each new incoming connection. Once the server accepts that connection, then it can start reading any data that the client has sent over the socket. In the case of an HTTP server working with the HTTP protocol, this would be the initial request protocol, method, version, headers and any body data. For different types of servers, the data would be in a different format.
Here's a useful diagram of the server:
Source: https://medium.com/javarevisited/fundamentals-of-socket-programming-in-java-bc9acc30eaf4
The server creates a socket used for the server to accept new connections..
It binds that socket to a specific IP address and port so it will only be informed about incoming connections targeted to that IP address and port.
It listens on that port to inform the TCP stack it is ready to accept incoming connections.
When it is notified of an incoming connection, it accepts that incoming connection.
Then it can read and write to that new connection over the new socket.
Then, sometime later, the incoming socket is closed to complete the client transaction.
The app.listen() method in Express encapsulates these steps and a few others. Internally (within Express), the code looks like this:
app.listen = function listen() {
var server = http.createServer(this);
return server.listen.apply(server, arguments);
};
You can see that method here in the open source repository.
To get an http server ready for steps 1-6 above, this creates the http server object within nodejs and then registers the app as the request listener for that server object (so it will be notified of incoming http requests).
Then, the call to server.listen() encapsulates steps 1-3 above.
Step 4 happens inside the http server object implementation and the app object is called when a new connection has been established and a new HTTP request is available. The http server reads the initial request and parses the http protocol and that initial request is already made available to the app for routing to the appropriate handler.
Then, subsequent calls such as res.send() or res.json() write a response back on the http socket and close the socket or res.end() will close it directly (steps 5 and 6 above).
Some other useful references:
Why is bind() used in TCP? Why is it used only on server side and not in client side? - Helps explain how a port and IP address define the TCP endpoint represented by a server. This port has to be known by the client so it can specifically request to connect to that port. The client end of the socket also has an IP address and a port, but its port can be dynamically assigned, thus the client does not have to bind to a specific port itself. The four pieces of data [server IP, server port, client IP, client port] define a specific TCP connection.
How TCP sockets work - has a good section about how new connections to a server work.
Understanding socket and port in TCP - talks about active and passive sockets. Passive sockets are sockets in "listen" mode used to accept incoming connections. Active sockets are two-way communications channels between two TCP endpoints.
Transmission Control Protocol (TCP) - more details on the various aspects of TCP from initiating a listening server, initiating a client connection to that server, through packet transmission to closing the socket.
There are a gazillion other references on the topic on the web. You can probably find 1000 articles on any single aspect of TCP that you might want more info about.
I think, this could be made easier if I could actually see the function instead of only calling it.
The underlying code for listen is inside the operating system's TCP stack and is not part of nodejs or Express. Express relies on the nodejs http server object as its interface to that and the nodejs http server object uses native code (built into nodejs) to call libuv (which is a cross platform C library that nodejs uses for networking and other things). Then, libuv talks to the underlying operating system APIs to reach the actual TCP stack on that target host. All of this is to put the server socket into listen mode so it can be notified of new incoming client connections to that target IP address and port.
Here's some doc on the related portions of the Linux TCP API if you want to see what the underlying TCP interface and description of that interface is:
socket() - https://linux.die.net/man/7/socket
bind() - https://linux.die.net/man/2/bind
listen() - https://linux.die.net/man/2/listen
And, portions of the libuv library that nodejs uses for networking:
TCP handles - http://docs.libuv.org/en/v1.x/tcp.html
Server listen() and accept() - http://docs.libuv.org/en/v1.x/stream.html#c.uv_listen
Hello, 👋
I was wondering how services (like mysql, apache, mongoDB) are bind against a port in the server/local machine. How does this work?
I'm guessing that when the service starts, it tries to connect to the port and if possible, the service is "paused" until the OS receives a request against the selected port. Is there any documentation out explaining how this works?
Thank you!
May I help you?
This is a list of TCP and UDP port numbers used by protocols for operation of network applications.
The Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP) only need one port for duplex, bidirectional traffic. They usually use port numbers that match the services of the corresponding TCP or UDP implementation, if they exist.
I am trying to create a simplistic TCP client and server. Conceptually, I know that a TCP socket is the same on both the client and server side (at least, this is how it is in python). However, the steps after creating a socket are different. Ie, for clients, the socket establishes a TCP connection to the server. On the server side, the socket is bound to a specific port, and waits for connections, and when it gets a req, it creates a new connected socket. (correct me if I got anything wrong, I'm new to networking).
My question is if there's any difference between the net.Socket and net.Server classes. Did node.js separate the two, and net.Server is explicitly meant for servers? Is it still possible to use net.Socket to make the TCP server socket?
What is the difference between net.createServer() and net.createConnection()?
net.createConnection() initiates an outbound TCP connection to some other host or server.
net.createServer() sets up a server that will accept incoming TCP connections from other hosts or processes.
These are opposite ends of enabling a connection.
My question is if there's any difference between the net.Socket and net.Server classes.
Yes, there's a huge difference between them as neither is a substitute for the other. A server listens for inbound connections.
A client then creates a TCP socket and attempts to connect to a server that is listening for inbound connections on the port and IP address that the server is listening on. During the connection process, the server follows the TCP handshake process to enable the creation of a TCP socket that connects the client and server. That TCP socket is then bidirectional so either end can then send data to the other.
Nodejs uses the net.Socket class as the nodejs object to represent a TCP socket so when you initiate a connection from a client, you get a net.Socket object that represents your TCP connection to some other server. When you are a server and someone connects to you, you get a net.Socket object that represents your TCP connection to the client that connected to you. Those two objects are different ends of a TCP connection and both ends do not have to be nodejs endpoints - they can be any language or tool that can make a standard TCP connection.
Did node.js separate the two, and net.Server is explicitly meant for servers? Is it still possible to use net.Socket to make the TCP server socket?
Yes, net.Server is exclusively for servers to set up a listener for inbound connections on a specific port on your host.
net.socket by itself cannot listen to incoming connections (you use an instance on net.Server for that. It is either used to initiate a TCP connection to some server or it is created as part of of some client connecting to your server.
I have to implement a server to server communication protocol using a SINGLE PERSISTENT TCP connection. The server at both the ends of this connection are implemented using "multi-threaded and asynchronous event-driven model". Both these servers are implemented in C++ and Pthreads on Linux. Server A always sends requests to Server B and Server B responds with a response. Server B doesn't send any requests to Server A, it just responds to the requests it receives. Could some one post me a sample code for this communication? Could you help me with the code for both Server A and Server B? Or please point me to any old answers or any websites where i can find a prototype code. Thanks in advance.
TCP servers cannot open connections to TCP servers. There is no IP protocol for that. One of the two servers must run a TCP client as a subsystem. The exact mechanics of how you do that depend on your client<>server protocol - the 'server-client' could log in to the 'client-server' with a unique username/password, or could use a different server listening port.
It's up to you:)