There will be no human being in the loop, and both endpoints are autonomous Node.js applications operating as independent services.
Endpoint A is responsible for contacting Endpoint B via secure web socket, and maintaining that connection 24/7/365.
Both endpoints will initiate messages independently (without human intervention), and both endpoints will have an API (RESTful or otherwise) to receive and process messages. You might say that each endpoint is both a client of, and a server to, the other endpoint.
I am considering frameworks like Sails.js and LoopBack (implemented on both endpoints), as well as simply passing JSON messages over ws, but remain unclear what the most idiomatic approach would be.
Web Sockets have a lot of overhead for connecting to browsers and what not, since they try to remain compatible with HTTP. If you're just connecting a pair of servers, a simple TCP connection will suffice. You can use the net module for this.
Now, once you have that connection, how do you initiate communication? You could go through the trouble of making your own protocol, but I don't recommend it. I found that a simple RPC was easiest. You can use the rpc-stream package over any duplex stream (including your TCP socket).
For my own application, I actually installed socket.io-client and let my servers use it for RPC. Although if I were to do it again, I would use rpc-stream to skip all the overhead required for setting up a Web Socket connection.
Related
I'm making a nodejs application that will act a server for other sites in different countries as the data being transmitted will be business related data. I would like to know how I can safely/securely send this data.
I am currently using socket.io to act as my main server (Master) on other sites there are (Slave) servers that handle the data from the master server.
I have got this working in a local environment but want to deploy this in the other sites.
I have tried to Google this to see if anyone else has done this but came across socket.io sessions but I don't know if this will fit with (Server->Server) connections.
Any help or experience would be grateful.
For server-server communication where you control both ends of the communication you can use WebSocket over HTTPS, you can use TCP over SSH tunnel or any other encrypted tunnel. You can use a PubSub service, a queue service etc. There are a lot of ways you can do it. Just make sure that the communication is encrypted either natively by the protocols you use or with VPN or tunnels that connect your servers in remote locations.
Socket.io is usually used as a replacement for WebSocket where there is no native support in the browser. It is rarely used for server to server communication. See this answer for more details:
Differences between socket.io and websockets
If you want a higher level framework with focus on real-time data then see ActionHero:
https://www.actionherojs.com/
For other options of sending real-time data between servers you can use some shared resource like a Redis database or some pub/sub service like Faye or Kafka, or a queue service like ZeroMQ or RabbitMQ. This is what is usually done to make things like that work across multiple instances of the server or multiple locations. You could also use a CouchDB changes feed, or a similar feature of RethinkDB to make sure that all of your instances get all the data as soon as it is posted by any one of them. See:
http://docs.couchdb.org/en/2.0.0/api/database/changes.html
https://rethinkdb.com/docs/changefeeds/javascript/
https://redis.io/topics/pubsub
https://faye.jcoglan.com/
https://kafka.apache.org/
Everything that uses HTTP is easy to encrypt with HTTPS. Everything else can be encrypted with a tunnel or VPN.
Good tools that can add encryption for protocols that are not encrypted themselves (like e.g. the Redis protocol) are:
http://www.tarsnap.com/spiped.html
https://www.stunnel.org/index.html
https://openvpn.net/
https://forwardhq.com/help/ssh-tunneling-how-to
See also:
https://en.wikipedia.org/wiki/Tunneling_protocol
Note that some hosting services may give you preconfigured tunnels or internal network interfaces that pass data encrypted between your servers located in different data centers of that provider. Some providers give you tools and tutorials to that easily as well.
I am having trouble understanding the difference between net.createserver and http.createserver in node.js.
I have read the documentation for both methods located at these two urls
https://nodejs.org/api/net.html#/net_net,
https://nodejs.org/api/http.html#/http_class_http_server.
I understand that http.createserver creates an http server. However, the documentation says that net.createserver creates a tcp server. I understand that tcp is the transmission protocol that http is on top of and that http servers are set up to read http request headers. I also understand the concept of even emitters in node.js pretty well. However, I don't understand this notion of a tcp server and why one would be made in node.js. The context is I am coding a chat application example in the "node.js in action" book.
http.createServer() sets up a server that handles the HTTP protocol, which is indeed transmitted over tcp. net.createServer() creates a server that simply understands when a TCP connection has happened, and data has been transmitted, and so on, but doesn't know anything about whether a valid HTTP request has been received, etc.
If you are writing a web server, favor http.createServer() over net.createServer() as it will save you a lot of work. If you are writing some other kind of server, do not use http.createServer().
I don't know much of a Node.js, but I know something about networks. HTTP is a protocol that works on 7th (Application) layer of model OSI. TCP is protocol that works on 4th (Transport) layer of model OSI. As you said, yes HTTP works on top of the TCP. The option of creating HTTP server by http.createServer() is there so you don't have to implement it by yourself by using net.createServer(). The protocol TCP might by used by lot of applications, you might create your own, or implement some different protocol than HTTP, for example: FTP, DNS, SMTP, Telnet and much much more.
Straight from the Node Net documentation. NET is the basic bare-bones server you can create. It's particularly useful for setting up a cluster of servers and allows simple connections but on that you'll want communication protocols, namely HTTP, which HTTP is in fact a NET server at it's core.
The net module provides an asynchronous network API for creating stream-based TCP or IPC servers (net.createServer()) and clients (net.createConnection()).
And from the HTTP documentation. HTTP is the common way to transmit large sets of data as requested by the client and then a response is generated. It's the standard way of communicating over the internet and introduces the concept of handshakes and is done through REST protocol, you know the usual request and response way of communicating.
The HTTP interfaces in Node.js are designed to support many features of the protocol which have been traditionally difficult to use. In particular, large, possibly chunk-encoded, messages. The interface is careful to never buffer entire requests or responses — the user is able to stream data.
Websockets are an upgrade over the HTTP headers and offer low latency and less server load and are a much more minimal conversation. If you're talking peer to peer communication, that's the way you'll want to go.
I am using Socket.IO with a MEAN stack and it's been excellent for low latency and bidirectional communication, but what would be the major draw back for using it for relatively static data as well as dynamic?
My assumption is that it would be more apt for sending more dynamic content. That being said, once a socket connection is established, how relevant is the amount of communication being done? Is there a time where it would be more appropriate to use http instead when a connection is constantly established throughout the user's direct interaction with the application?
Thanks!
WebSockets are a bidirectional data exchange within a HTTP connection. So the question is not if you use HTTP or WebSockets, because there is no WebSockets without HTTP. WebSockets are often confused with simple (BSD) sockets, but WebSockets are actually a socket-like layer inside a HTTP connection which is inside a TCP connection which uses "real" sockets. Or for anybody familiar with OSI layers: it as a layer 4 (transport) encapsulated inside layer 7 (application) and the main reason for doing it this strange way instead of using layer 4 directly is that plain sockets to ports outside of HTTP, SMTP and a few other protocols are no longer possible because of all the port blocking firewalls.
So the question should be more if you use simple HTTP or if you need to use WebSockets (inside HTTP).
With simple HTTP the client sends a request and the server sends the response back. The format is well defined and browser and server transparently support compression, caching and other optimizations. But this simple request-response pattern is limited, because there is no way to push data from server to client or to have a more (BSD) socket like behavior where both client and server can send any data at any time. There are various more or less good workarounds for this, like long polling.
WebSockets gives you a bidirectional communication, which makes it possible for the server to push data to the client or to send data in both directions at any time. And once the WebSocket connection is established by upgrading an existing HTTP connection the overhead for the data itself is very small, much smaller then with a full new HTTP request. While this sounds good you loose all the advantages of simple request-response HTTP like caching at the client or in proxies. And because client and server need resources to keep the underlying TCP connection open it needs more resources, which can be relevant for a busy server. Also, WebSockets might give you more trouble with middleboxes (like proxies or firewalls) then simple HTTP does.
In summary: if you don't need the advantages of WebSockets stay with simple request-response HTTP.
Given a standard Node.js HTTP library, or an existing REST client library, what would be the most feasible way to allow such a library to perform those HTTP requests over the top of my own protocol?
To put this another way: I aim provide a module which looks like a HTTP client. It accepts HTTP requests headers, and returns HTTP responses. What options should I consider to adapt an existing REST library to work with my 'pseudo' HTTP client module, as opposed to the standard Node library HTTP client?
Further background information
I wish to create a server application (based on Node.js) which makes HTTP REST requests to a remote embedded device. However, due to NAT, it is not possible for the application server to make client TCP connections directly to the remote device. Therefore, to get around NAT, I will devise my own proprietary protocol which involves the remote device initiating a persistent connection to the application server. Then, once that persistent connection is established, the Node.js application shall be able to make HTTP requests back over that persistent connection to the networked device.
My objective is therefore to create a Node.js module which acts as a 'bridge' layer between incoming socket connections from the networked devices, and the main application which makes REST requests. The aim is that the application would make REST requests as if it were making HTTP client requests to a server, when in fact the HTTP requests and responses are being conveyed on top of the proprietary protocol.
An option I'm presently considering is for my 'bridge' module to implement an interface that mimics that of http.request(options,[callback]) and somehow enforce a REST client library to use this interface instead of the Node HTTP client. Supposedly at minimum I'd have to lightly modify whichever REST client library I'd use to achieve this.
As explained above, I'm essentially trying to create my own form of NAT traversal using an intermediary server. The intermediary server would provide the front-end UI to users, and make back-end data requests to the embedded networked devices. Connections between embedded devices and application server would be persistent, and initiated from the embedded devices, to avoid the usual NAT headaches (i.e. the requirement to configure port forwarding).
Though I mentioned earlier I'd achieve the device-to-server connection using my own protocol over a raw socket connection, the mechanism I'm actually experimenting with right now is to use plain HTTP together with long-polling. The embedded device initiates a HTTP connection to the application server and delayed responses are used to convey data back to the device when the server has something to send. I would then 'tunnel' HTTP requests going in the reverse direction over the top of this.
Therefore, in simple terms, my 'bridge' layer is something that accepts HTTP connections inwards from both sides (outside device connections, and inside web application REST requests). By using long-polling it would effectively convey requests and responses between the connected clients.
Instead of replacing the http layer, create a man-in-the-middle. Create an http server in node that is the target for all of the rest requests. It then transfers the request onto the proprietary protocol and handles the response by translating back to rest.
This way you don't have to hack the rest code and can even swap it out for another library if needed.
I have a program in an embedded device that outputs an xml string to a socket. The embedded device has lighthttpd has a web server. I want to use a web based client (no flash/silverlight) to connect to the socket and pull the xml data every second.
I looked at Node.js with Socket.io to get what I want to do, but I am not clear about how to proceed. Searching through the Node.js and Socket.io documentation and examples I see standard client-server behavior, nothing regarding what I am trying to do.
Basically, the web server is just there to accept a connection from a client on the socket that the embedded application is outputting data to. Basically the web server's purpose is to just let the client retrieve data from the raw tcp socket that the embedded application is writing to. Please advice.
I solved the problem using Websockify, which acts as bridge between a TCP Socket and a browser.
The html client will connect to a websocket, and Websockify will listen on the websocket port and transmit data between the websocket and the tcp socket.
Web browsers have the ability to do HTTP requests (which can be web page requests or Ajax requests for data) and webSocket connections. You will need to pick one of these two mechanisms if you're sticking with stock browser access.
If the lighthttpd web server in the embedded device does not support webSockets, then your choice will like be an Ajax call from the browser to your server. This is basically just an HTTP request that make return something different than a web page (often JSON data) and is designed to fetch data from the server into a web client.
If the lighthttpd web server does support webSockets, then you could use a webSocket connection to fetch the data too. This has an advantage of being a persistent connection and allows for the server to directly send data to the client (without the client even requesting more data) whenever it wants to (more efficient for constant updates).
An Ajax connection is generally not persistent. A client sends an Ajax request, the server returns the answer and the connection is closed. The next request starts a new Ajax request.
Either Ajax requests or webSocket connections should work just fine for your use. All browsers still in use support Ajax. WebSockets are supported in modern browsers (IE10 and higher).
Once you decide upon a client connection strategy, then you'd build your web app on the embedded device that served as the middleman between the browser and the data on the embedded device. It would collect the appropriate data from the embedded device and then be able to send that to browser clients that connected and requested the data.
I'm not sure exactly why you mentioned node.js. In this circumstance, it would be used as the web server and the environment for building your app and the logic that collects the data from your device and feeds it to the requesting web browser, but it sounds like you already have lighthttpd for this purpose. Personally, I recommend node.js if it works in your environment. Combined with socket.io (for webSocket support), it's a very nice way to connect browsers directly to an embedded device. I have an attic fan controller written in node.js and running on a Raspberry Pi. The node.js app monitors temperature probes and controls relays that switch attic fans and node.js also serves as a web server for me to administer and monitor the node.js. All-in-all, it's a pretty slick environment if you already know and like programming in Javascript and there's a rich set of add-in modules to extend its capabilities available through NPM. If, however, your embedded device isn't a common device that there is already support for node.js on or it doesn't already have node.js on it, then you'd be facing a porting tasks to make node.js run on it which might be more work than using some other development environment that already runs on the device like lighthttpd.