My question could be flagged as "opinion based" but I am wondering which approach is the best for my application as I am able to do it in both ways.
I am building chat application in which users and conversations are saved in MongoDB. I will have my react application consuming API/APIs. The question is - is it better to have REST API and Socket.io applications running separate? For example:
Have REST API running on port 3005
Have Socket.io running on port 3006
React Application consuming these 2 separately and basically they will not know about each other. My endpoints in REST API endpoints and socket.io will be invoked only in front-end.
On the other hand, I can have my socket.io application and REST API working together in 1 big application. I think it is possible to make it working without problems.
To sum up, at first glance I would take the first approach - more cleaner and easy to maintain. But I would like to hear other opinions or if somebody had a similar project. Usually how the things are made in this kind of projects when you have socket.io and REST API?
I would check the pros and cons for both scenario. For example code and resource reusability is better if you have a single application and you don't have to care about which versions are compatible with each other. On the other hand one error can kill both applications, so from security perspective it is better to have separate applications. I think the decision depends on what pros and cons are important to you.
you can make a separate file for socket.io logic like this:
// socket.mjs file
import { Server } from "socket.io"
let io = new Server()
const socketApi = {
io: io
}
io.on('connection',(socket)=>{
console.log('client connected:', socket.id)
socket.join('modbus-room')
socket.on('app-server', data=>{
console.log('**************')
console.log(data)
io.to('modbus-room').emit('modbus-client', data)
})
socket.on('disconnect',(reason)=>{
console.log(reason)
})
})
export default socketApi
and add it to your project like this:
// index.js or main file
//...
import socketApi from "../socket.mjs";
//...
//
/**
* Create HTTP server.
*/
const server = http.createServer(app);
socketApi.io.attach(server);
//
I am developing a web application with a small team, and after researching and studying a bit we discovered it is a good practice to separate back-end and front-end projects. So we will develop the back-end as a REST API with hapijs and mysql database, and the front-end using angularjs.
But in the production environment they must be at the same server, right? How do we deploy them to the same server if they are in separate repositories?
We are a fairly new team, starting our adventures in web development, so we are studying a lot to get things right.
Our technology stack will be:
Hapijs for the webserver
sequelize for orm
socket.io for chat functions
mocha for unit testing
angularjs for frontend
We will use microsoft azure for hosting our web app.
Thank You for the answers and help.
They don't have to be in the same server. It is perfectly fine to have the backend in a different server, it is also handy if you need to scale your backend/frontend but not the other.
There are a few possibilities:
You could use a message broker like RabbitMQ to communicate between the two micro-services
Your frontend app could expose the url of your backend and you use that in your AJAX requests, this requires your backend to enable CORS. Not a fan of this approach.
Use relative urls in your frontend and then pipe the requests with a particular pattern (like /api/*) to your backend. Is your AngularJs app served by a Node.js server or is it a Hapi.js server too?
If Node.js something like
:
app.all(['/api/*', '/fe/*'], function(req, res) {
console.log('[' + req.method + ']: ' + PROXY + req.url);
req.pipe(request({
url: PROXY + req.url,
method: req.method,
body: req.body,
rejectUnauthorized: false,
withCredentials: true
}))
.on('error', function(e) {
res.status(500).send(e);
})
.pipe(res);
});
Where PROXY_URL is the url/ip of your backend server.
Haven't done it in hapi.js but it should also be possible.
I'm sure there are more options, those are the ones I'm familiar with.
As a beginner team you might have come across so many questions. The best thing to do is to
seperate the front end and back end as seperate units. That will lead to better full stack development process. This https://www.freecodecamp.org/news/lessons-learned-from-deploying-my-first-full-stack-web-application-34f94ec0a286/
article explains the process using react as Front end and Express js for backend.
It is well designed for beginners. It includes detailed explanation about your question.
As you guys are starting out now I think you can handle this by creating two server instances with hapi. This addresses your requirement to have only one server in production:
const server = new Hapi.Server();
server.connection({ port: 80, labels: 'front-end' });
server.connection({ port: 8080, labels: 'api' });
This is really easy to implement, however, it comes with a downside: you need to accept down time form both the client and server app whenever you rollout updates.
You can find more info in this post: https://futurestud.io/blog/hapi-how-to-run-separate-frontend-and-backend-servers-within-one-project
In regards to deployment, whatever you use to build a release (continuous integration tools, manual scripts, etc.) can be git pushed to azure: https://azure.microsoft.com/en-us/documentation/articles/web-sites-publish-source-control. For instance, a manual script would fetch your code from the two separate repositories (front-end and back-end), copy the relevant files, update config values and push the end result to git.
I have the node db connection working okay I think:
var marklogic = require('marklogic');
var db = marklogic.createDatabaseClient({ ...
And now I want to do cts.search
How do I require cts?
You don't. The cts functions are for running queries within the database. Working with the Node.js Client API, you'll use the query builder. There's an example in the feature introduction, with more detail in the Getting Started tutorial and the Node.js Application Developer's Guide.
You may be mixing up the Node.js Client API with Server-side JavaScript. The former is used to have middle-tier Node.js code interact with MarkLogic. The latter is for writing REST API extensions or transformations that run within the database (and can be called from the Node API).
I had the requirement to build up a REST API in node.js and was looking for a more light-weight framework than express.js which probably avoids the unwanted features and would act like a custom-built framework for building REST APIs. Restify from its intro is recommended for the same case.
Reading Why use restify and not express? seemed like restify is a good choice.
But the surprise came when I tried out both with a load.
I made a sample REST API on Restify and flooded it with 1000 requests per second. Surprise to me the route started not responding after a while. The same app built on express.js handled all.
I am currently applying the load to API via
var FnPush = setInterval(function() {
for(i=0;i<1000;i++)
SendMsg(makeMsg(i));
}, 1000);
function SendMsg(msg) {
var post_data = querystring.stringify(msg);
var post_options = {
host: target.host,
port: target.port,
path: target.path,
agent: false,
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Content-Length': post_data.length,
"connection": "close"
}
};
var post_req = http.request(post_options, function(res) {});
post_req.write(post_data);
post_req.on('error', function(e) {
});
post_req.end();
}
Does the results I have got seem sensible? And if so is express more efficient than restify in this scenario? Or is there any error in the way I tested them out?
updated in response to comments
behavior of restify
when fed with a load of more than 1000 req.s it stopped processing in just 1 sec receiving till 1015 req.s and then doing nothing. ie. the counter i implemented for counting incoming requests stopped increment after 1015.
when fed with a load of even 100 reqs. per second it received till 1015 and gone non responsive after that.
Corrigendum: this information is now wrong, keep scrolling!
there was an issue with the script causing the Restify test to be conducted on an unintended route. This caused the connection to be kept alive causing improved performance due to reduced overhead.
This is 2015 and I think the situation has changed a lot since. Raygun.io has posted a recent benchmark comparing hapi, express and restify.
It says:
We also identified that Restify keeps connections alive which removes the overhead of creating a connection each time when getting called from the same client. To be fair, we have also tested Restify with the configuration flag of closing the connection. You’ll see a substantial decrease in throughput in that scenario for obvious reasons.
Looks like Restify is a winner here for easier service deployments. Especially if you’re building a service that receives lots of requests from the same clients and want to move quickly. You of course get a lot more bang for buck than naked Node since you have features like DTrace support baked in.
This is 2017 and the latest performance test by Raygun.io comparing hapi, express, restify and Koa.
It shows that Koa is faster than other frameworks, but as this question is about express and restify, Express is faster than restify.
And it is written in the post
This shows that indeed Restify is slower than reported in my initial
test.
According to the Node Knockout description:
restify is a node.js module purpose built to create REST web services in Node. restify makes lots of the hard problems of building such a service, like versioning, error handling and content-negotiation easier. It also provides built in DTrace probes that you get for free to quickly find out where your application’s performance problems lie. Lastly, it provides a robust client API that handles retry/backoff for you on failed connections, along with some other niceties.
Performance issues and bugs can probably be fixed. Maybe that description will be adequate motivation.
I ran into a similar problem benchmarking multiple frameworks on OS X via ab. Some of the stacks died, consistently, after around the 1000th request.
I bumped the limit significantly, and the problem disappeared.
You can you check your maxfiles is at with ulimit, (or launchctl limit < OS X only) and see what the maximum is.
Hope that helps.
In 2021, benchmarking done by Fastify (https://www.fastify.io/benchmarks/) indicates that Restify is now slightly faster than Express.
The code used to run the benchmark can be found here https://github.com/fastify/benchmarks/.
i was confused with express or restify or perfectAPI. even tried developing a module in all of them. the main requirement was to make a RESTapi. but finally ended up with express, tested my self with the request per second made on all the framework, the express gave better result than others. Though in some cases restify outshines express but express seams to win the race. I thumbs up for express. And yes i also came across locomotive js, some MVC framework build on top of express. If anyone looking for complete MVC app using express and jade, go for locomotive.
What are the differences between socket.io and websockets in
node.js?
Are they both server push technologies?
The only differences I felt was,
socket.io allowed me to send/emit messages by specifying an event name.
In the case of socket.io a message from server will reach on all clients, but for the same in websockets I was forced to keep an array of all connections and loop through it to send messages to all clients.
Also,
I wonder why web inspectors (like Chrome/firebug/fiddler) are unable to catch these messages (from socket.io/websocket) from server?
Please clarify this.
Misconceptions
There are few common misconceptions regarding WebSocket and Socket.IO:
The first misconception is that using Socket.IO is significantly easier than using WebSocket which doesn't seem to be the case. See examples below.
The second misconception is that WebSocket is not widely supported in the browsers. See below for more info.
The third misconception is that Socket.IO downgrades the connection as a fallback on older browsers. It actually assumes that the browser is old and starts an AJAX connection to the server, that gets later upgraded on browsers supporting WebSocket, after some traffic is exchanged. See below for details.
My experiment
I wrote an npm module to demonstrate the difference between WebSocket and Socket.IO:
https://www.npmjs.com/package/websocket-vs-socket.io
https://github.com/rsp/node-websocket-vs-socket.io
It is a simple example of server-side and client-side code - the client connects to the server using either WebSocket or Socket.IO and the server sends three messages in 1s intervals, which are added to the DOM by the client.
Server-side
Compare the server-side example of using WebSocket and Socket.IO to do the same in an Express.js app:
WebSocket Server
WebSocket server example using Express.js:
var path = require('path');
var app = require('express')();
var ws = require('express-ws')(app);
app.get('/', (req, res) => {
console.error('express connection');
res.sendFile(path.join(__dirname, 'ws.html'));
});
app.ws('/', (s, req) => {
console.error('websocket connection');
for (var t = 0; t < 3; t++)
setTimeout(() => s.send('message from server', ()=>{}), 1000*t);
});
app.listen(3001, () => console.error('listening on http://localhost:3001/'));
console.error('websocket example');
Source: https://github.com/rsp/node-websocket-vs-socket.io/blob/master/ws.js
Socket.IO Server
Socket.IO server example using Express.js:
var path = require('path');
var app = require('express')();
var http = require('http').Server(app);
var io = require('socket.io')(http);
app.get('/', (req, res) => {
console.error('express connection');
res.sendFile(path.join(__dirname, 'si.html'));
});
io.on('connection', s => {
console.error('socket.io connection');
for (var t = 0; t < 3; t++)
setTimeout(() => s.emit('message', 'message from server'), 1000*t);
});
http.listen(3002, () => console.error('listening on http://localhost:3002/'));
console.error('socket.io example');
Source: https://github.com/rsp/node-websocket-vs-socket.io/blob/master/si.js
Client-side
Compare the client-side example of using WebSocket and Socket.IO to do the same in the browser:
WebSocket Client
WebSocket client example using vanilla JavaScript:
var l = document.getElementById('l');
var log = function (m) {
var i = document.createElement('li');
i.innerText = new Date().toISOString()+' '+m;
l.appendChild(i);
}
log('opening websocket connection');
var s = new WebSocket('ws://'+window.location.host+'/');
s.addEventListener('error', function (m) { log("error"); });
s.addEventListener('open', function (m) { log("websocket connection open"); });
s.addEventListener('message', function (m) { log(m.data); });
Source: https://github.com/rsp/node-websocket-vs-socket.io/blob/master/ws.html
Socket.IO Client
Socket.IO client example using vanilla JavaScript:
var l = document.getElementById('l');
var log = function (m) {
var i = document.createElement('li');
i.innerText = new Date().toISOString()+' '+m;
l.appendChild(i);
}
log('opening socket.io connection');
var s = io();
s.on('connect_error', function (m) { log("error"); });
s.on('connect', function (m) { log("socket.io connection open"); });
s.on('message', function (m) { log(m); });
Source: https://github.com/rsp/node-websocket-vs-socket.io/blob/master/si.html
Network traffic
To see the difference in network traffic you can run my test. Here are the results that I got:
WebSocket Results
2 requests, 1.50 KB, 0.05 s
From those 2 requests:
HTML page itself
connection upgrade to WebSocket
(The connection upgrade request is visible on the developer tools with a 101 Switching Protocols response.)
Socket.IO Results
6 requests, 181.56 KB, 0.25 s
From those 6 requests:
the HTML page itself
Socket.IO's JavaScript (180 kilobytes)
first long polling AJAX request
second long polling AJAX request
third long polling AJAX request
connection upgrade to WebSocket
Screenshots
WebSocket results that I got on localhost:
Socket.IO results that I got on localhost:
Test yourself
Quick start:
# Install:
npm i -g websocket-vs-socket.io
# Run the server:
websocket-vs-socket.io
Open http://localhost:3001/ in your browser, open developer tools with Shift+Ctrl+I, open the Network tab and reload the page with Ctrl+R to see the network traffic for the WebSocket version.
Open http://localhost:3002/ in your browser, open developer tools with Shift+Ctrl+I, open the Network tab and reload the page with Ctrl+R to see the network traffic for the Socket.IO version.
To uninstall:
# Uninstall:
npm rm -g websocket-vs-socket.io
Browser compatibility
As of June 2016 WebSocket works on everything except Opera Mini, including IE higher than 9.
This is the browser compatibility of WebSocket on Can I Use as of June 2016:
See http://caniuse.com/websockets for up-to-date info.
Its advantages are that it simplifies the usage of WebSockets as you described in #2, and probably more importantly it provides fail-overs to other protocols in the event that WebSockets are not supported on the browser or server. I would avoid using WebSockets directly unless you are very familiar with what environments they don't work and you are capable of working around those limitations.
This is a good read on both WebSockets and Socket.IO.
http://davidwalsh.name/websocket
tl;dr;
Comparing them is like comparing Restaurant food (maybe expensive sometimes, and maybe not 100% you want it) with homemade food, where you have to gather and grow each one of the ingredients on your own.
Maybe if you just want to eat an apple, the latter is better. But if you want something complicated and you're alone, it's really not worth cooking and making all the ingredients by yourself.
I've worked with both of these. Here is my experience.
SocketIO
Has autoconnect
Has namespaces
Has rooms
Has subscriptions service
Has a pre-designed protocol of communication
(talking about the protocol to subscribe, unsubscribe or send a message to a specific room, you must all design them yourself in websockets)
Has good logging support
Has integration with services such as redis
Has fallback in case WS is not supported (well, it's more and more rare circumstance though)
It's a library. Which means, it's actually helping your cause in every way. Websockets is a protocol, not a library, which SocketIO uses anyway.
The whole architecture is supported and designed by someone who is not you, thus you dont have to spend time designing and implementing anything from the above, but you can go straight to coding business rules.
Has a community because it's a library (you can't have a community for HTTP or Websockets :P They're just standards/protocols)
Websockets
You have the absolute control, depending on who you are, this can be very good or very bad
It's as light as it gets (remember, its a protocol, not a library)
You design your own architecture & protocol
Has no autoconnect, you implement it yourself if yo want it
Has no subscription service, you design it
Has no logging, you implement it
Has no fallback support
Has no rooms, or namespaces. If you want such concepts, you implement them yourself
Has no support for anything, you will be the one who implements everything
You first have to focus on the technical parts and designing everything that comes and goes from and to your Websockets
You have to debug your designs first, and this is going to take you a long time
Obviously, you can see I'm biased to SocketIO. I would love to say so, but I'm really really not.
I'm really battling not to use SocketIO. I dont wanna use it. I like designing my own stuff and solving my own problems myself.
But if you want to have a business and not just a 1000 lines project, and you're going to choose Websockets, you're going to have to implement every single thing yourself. You have to debug everything. You have to make your own subscription service. Your own protocol. Your own everything. And you have to make sure everything is quite sophisticated. And you'll make A LOT of mistakes along the way. You'll spend tons of time designing and debugging everything. I did and still do. I'm using websockets and the reason I'm here is because they're unbearable for a one guy trying to deal with solving business rules for his startup and instead dealing with Websocket designing jargon.
Choosing Websockets for a big application ain't an easy option if you're a one guy army or a small team trying to implement complex features. I've wrote more code in Websockets than I ever wrote with SocketIO in the past, for ten times simpler things than I did with SocketIO.
All I have to say is ... Choose SocketIO if you want a finished product and design. (unless you want something very simple in functionality)
Im going to provide an argument against using socket.io.
I think using socket.io solely because it has fallbacks isnt a good idea. Let IE8 RIP.
In the past there have been many cases where new versions of NodeJS has broken socket.io. You can check these lists for examples... https://github.com/socketio/socket.io/issues?q=install+error
If you go to develop an Android app or something that needs to work with your existing app, you would probably be okay working with WS right away, socket.io might give you some trouble there...
Plus the WS module for Node.JS is amazingly simple to use.
Using Socket.IO is basically like using jQuery - you want to support older browsers, you need to write less code and the library will provide with fallbacks. Socket.io uses the websockets technology if available, and if not, checks the best communication type available and uses it.
https://socket.io/docs/#What-Socket-IO-is-not (with my emphasis)
What Socket.IO is not
Socket.IO is NOT a WebSocket implementation. Although Socket.IO indeed uses WebSocket as a transport when possible, it adds some metadata to each packet: the packet type, the namespace and the packet id when a message acknowledgement is needed. That is why a WebSocket client will not be able to successfully connect to a Socket.IO server, and a Socket.IO client will not be able to connect to a WebSocket server either. Please see the protocol specification here.
// WARNING: the client will NOT be able to connect!
const client = io('ws://echo.websocket.org');
I would like provide one more answer in 2021. socket.io has become actively maintained again since 2020 Sept. During 2019 to 2020 Aug(almost 2 years) there was basically no activity at all and I had thought the project may be dead.
Socket.io also published an article called Why Socket.IO in 2020?, except for a fallback to HTTP long-polling, I think these 2 features are what socket.io provides and websocket lacks of
auto-reconnection
a way to broadcast data to a given set of clients (rooms/namespace)
One more feature I find socket.io convenient is for ws server development, especially I use docker for my server deployment. Because I always start more than 1 server instances, cross ws server communication is a must and socket.io provide https://socket.io/docs/v4/redis-adapter/ for it.
With redis-adapter, scaling server process to multiple nodes is easy while load balance for ws server is hard. Check here https://socket.io/docs/v4/using-multiple-nodes/ for further information.
Even if modern browsers support WebSockets now, I think there is no need to throw SocketIO away and it still has its place in any nowadays project. It's easy to understand, and personally, I learned how WebSockets work thanks to SocketIO.
As said in this topic, there's a plenty of integration libraries for Angular, React, etc. and definition types for TypeScript and other programming languages.
The other point I would add to the differences between Socket.io and WebSockets is that clustering with Socket.io is not a big deal. Socket.io offers Adapters that can be used to link it with Redis to enhance scalability. You have ioredis and socket.io-redis for example.
Yes I know, SocketCluster exists, but that's off-topic.
Socket.IO uses WebSocket and when WebSocket is not available uses fallback algo to make real time connections.
TLDR:
'Socket.io' is an application layer specification that can be implemented on top of/using the application layer specification 'websockets'.
websocket spec
socket.io spec
I think the simple answer here is in basic web technology definitions:
Specification: A documented standard detailing the requirements for a program to achieve in order to be labeled as "an implimentation of some sepc." It is important to achieve this rubber stamp when building programs, because any program is only as good at the machine executing the code. Programming is fundamentally built upon specifications, and if, they are not followed code will not execute correctly. However, a specification does nothing. It is just a text document.
Implementation: This is actual, executable code that accomplishes what the specification says to do.
Application Layer - System that defines messages and handshakes sent over transport. This is the stuff you have to know when working with HTTP/Websockets/Socketio. It defines how the connections will be made, authenticated, data will be sent, and how it will arrive.