I am using socket.io with nodejs on the server side to provide two-way communication between my web application and the server.
I am using the socket.io API to issue commands and receive responses, but I am not sure if there is a more methodical way of defining a "language" for sending commands to the server and receiving results from it.
For sending commands to the server, I am emitting events from the web application like the following (I am using pseudo code below):
socket.emit('commandRequest', {
msg_id: '...'
username: '...',
command: '...'
});
The server evaluates the command and emits responses like the following:
socket.on('commandRequest', (data) => {
// parse and execute data.command
socket.emit('commandResponse', {
msg_id: data.msg_id,
username: data.username,
response: ...,
error: ...
});
})
Finally, the web application is listening to responses from the server and it updates the app content accordingly:
socket.on('commandResponse', (data) => {
if (data.error) {
...
} else {
// interpret data.response
}
})
So I am using the commandRequest/commandResponse event naming paradigm and the event data structure has corresponding {command: ...} and {response: ...} properties.
Is there a more formal way of defining a request/response "language" that can be used for more complex client/server interactions? Something similar to what REST APIs achieve with HTTP request, but using web sockets?
You could try looking into Primus and its Responder plugin as described in this blog post: Request-Response Oriented Websockets
An excerpt from the post:
It allows to request the opposite peer for a response using a simple API known from other libraries.
Primus Responder adds two things to Primus' core functionality:
A new event called request is emitted as soon as a peer requests a response.
The new method named writeAndWait(data, fn) writes data to a peer and runs given callback function as the peer sends its response.
Primus Responder is available on the client too.
Related
I started using Azure recently and It has been an overwhelming experience. I started experimenting with eventhubs and I'm basically following the official tutorials on how to send and receive messages from eventhubs using nodejs.
Everything worked perfectly so I built a small web app (static frontend app) and I connected it with a node backend, where the communication with eventhubs occurs. So basically my app is built like this:
frontend <----> node server <-----> eventhubs
As you can see it is very simple. The node server is fetching data from eventhubs and sending it forward to the frontend, where the values are shown. It is a cool experience and I'm enjoying MS Azure until this error occured:
azure.eventhub.common.EventHubError: ErrorCodes.ResourceLimitExceeded: Exceeded the maximum number of allowed receivers per partition in a consumer group which is 5. List of connected receivers - nil, nil, nil, nil, nil.
This error is really confusing. Im using the default consumer group and only one app. I never tried to access this consumer group from another app. It said the limit is 5, I'm using only one app so it should be fine or am I missing something? I'm not checking what is happening here.
I wasted too much time googling and researching about this but I didn't get it. At the end, I thought that maybe every time I deploy the app (my frontend and my node server) on azure, this would be counted as one consumer and since I deployed the app more than 5 times then this error is showing up. Am I right or this is nonsense?
Edit
I'm using websockets as a communication protocol between my app (frontend) and my node server (backend). The node server is using the default consumer group ( I didn't change nothing), I just followed this official example from Microsoft. I'm basically using the code from MS docs that's why I didn't post any code snippet from my node server and since the error happens in backend and not frontend then it will not be helpful if I posted any frontend code.
So to wrap up, I'm using websocket to connect front & backend. It works perfectly for a day or two and then this error starts to happen. Sometimes I open more than one client (for example a client from the browser and client from my smartphone).
I think I don't understand the concept of this consumer group. Like is every client a consumer? so if I open my app (the same app) in 5 different tabs in my browser, do I have 5 consumers then?
I didn't quite understand the answer below and what is meant by "pooling client", therefore, I will try to post code examples here to show you what I'm trying to do.
Code snippets
Here is the function I'm using on the server side to communicate with eventhubs and receive/consume a message
async function receiveEventhubMessage(socket, eventHubName, connectionString) {
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log("[ consumer ] Message received : " + event.body);
io.emit('msg-received', event.body);
}
},
processError: async (err, context) => {
console.log(`Error : ${err}`);
}
}
);
If you notice, I'm giving the eventhub and connection string as an argument in order to be able to change that. Now in the frontend, I have a list of multiple topics and each topic have its own eventhubname but they have the same eventhub namespace.
Here is an example of two eventhubnames that I have:
{
"EventHubName": "eh-test-command"
"EventHubName": "eh-test-telemetry"
}
If the user chooses to send a command (from the frontend, I just have a list of buttons that the user can click to fire an event over websockets) then the CommandEventHubName will be sent from the frontend to the node server. The server will receive that eventhubname and switch the consumerClient in the function I posted above.
Here is the code where I'm calling that:
// io is a socket.io object
io.on('connection', socket => {
socket.on('onUserChoice', choice => {
// choice is an object sent from the frontend based on what the user choosed. e.g if the user choosed command then choice = {"EventhubName": "eh-test-command", "payload": "whatever"}
receiveEventhubMessage(socket, choice.EventHubName, choice.EventHubNameSpace)
.catch(err => console.log(`[ consumerClient ] Error while receiving eventhub messages: ${err}`));
}
}
The app I'm building will be extending in the future to a real use case in the automotive field, that's why this is important for me. Therefore, I'm trying to figure out how can I switch between eventhubs without creating a new consumerClient each time the eventhubname changes?
I must say that I didn't understand the example with the "pooling client". I am seeking more elaboration or, ideally, a minimal example just to put me on the way.
Based on the conversation in the issue, it would seem that the root cause of this is that your backend is creating a new EventHubConsumerClient for each request coming from your frontend. Because each client will open a dedicated connection to the service, if you have more than 5 requests for the same Event Hub instance using the same consumer group, you'll exceed the quota.
To get around this, you'll want to consider pooling your EventHubConsumerClient instances so that you're starting with one per Event Hub instance. You can safely use the pooled client to handle a request for your frontend by calling subscribe. This will allow you to share the connection amongst multiple frontend requests.
The key idea being that your consumerClient is not created for every request, but shares an instance among requests. Using your snippet to illustrate the simplest approach, you'd end up hoisting your client creation to outside the function to receive. It may look something like:
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
async function receiveEventhubMessage(socket, eventHubName, connectionString) {
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log("[ consumer ] Message received : " + event.body);
io.emit('msg-received', event.body);
}
},
processError: async (err, context) => {
console.log(`Error : ${err}`);
}
}
);
That said, the above may not be adequate for your environment depending on the architecture of the application. If whatever is hosting receiveEventHubMessage is created dynamically for each request, nothing changes. In that case, you'd want to consider something like a singleton or dependency injection to help extend the lifespan.
If you end up having issues scaling to meet your requests, you can consider increasing the number of clients for each Event Hub and/or spreading requests out to different consumer groups.
Ok, this is not what you think it is, I am not asking for help with the async/wait pattern or asynchronous programming I am well versed with those. I am however querying whether something is possible within a Node.JS Express service.
The Scenario
I have a web service which is developed in Node.JS and uses Express.JS to expose some REST endpoints that a client can connect to and send a POST request. For the most part these are Synchronous and will create a SOAP message and send that on to an external service and receive an immediate response which can then be returned to the client, all really simple stuff which is already implemented. So what's your point I hear you say, I am coming to that.
I have a couple of POST interactions that will build a SOAP message to send to an Asynchronous external endpoint where the response will be received asynchronously through an inbound endpoint.
Option 1: What I am looking for in these cases is to be able to build the SOAP message, create a listener (so I can listen for the response to my request), and then send the request to the external service which immediately returns a 200.
Option 2: When I setup the service I want to also setup and listen for incoming requests from the external service whilst also listening for REST requests from the internal service.
The Question
Is either option possible in Node and Express? and, if so how would one achieve this?
NOTE: I know its possible in C# using WCF or a Listener but I would like to avoid this and use Node.JS so any help would be greatly appreciated.
First of all check node-soap if it fits your needs.
Option 1: What I am looking for in these cases is to be able to build the SOAP message, create a listener (so I can listen for the response to my request), and then send the request to the external service which immediately returns a 200.
Here's a very basic non-soap service implementation.
let request = require('request-promise');
let express = require('express');
let app = express();
//Validate the parameters for the request
function validateRequest(req) { ... }
//Transform the request to match the internal API endpoint
function transformRequest(req) { ... }
app.post('/external', function(req, res) {
if(!validateRequest(req))
return res.status(400).json({success: false, error: 'Bad request format');
res.status(200).send();
let callbackUrl = req.query.callback;
let transformedRequest = transformRequest(req);
let internalServiceUrl = 'https://internal.service.com/internal'
request.post(internalServiceUrl, {body: transformedRequest}).then(function (internalResponse){
//Return some of the internal response?
return request.get(callbackUrl, {success: true, processed: true});
}).catch(function (e) {
request.get(callbackUrl, {success: false, error: e});
});
});
Option 2: When I setup the service I want to also setup and listen for incoming requests from the external service whilst also listening for REST requests from the internal service.
There is no "listening" in http. Check socket.io if you need realtime listening. It uses websockets.
Your other option is to poll the internal service (say if you want to check for its availability).
In this node.js forum app https://github.com/designcreateplay/NodeBB, which allows you to follow other users on the site, I noticed that it seems to use sockets to communicate information I would have been expected to be communicated via an ajax post request. For example, when you click the button to follow another user (which this forum software allows), the socket on the client emits an 'api:user.follow' event, which is then listened for on the server, as you can see below.
Can you explain why someone would or wouldn't want to use sockets instead of ajax for this type of functionality? I assume there's pros and cons to each, but I don't know them.
client
followBtn.on('click', function() {
socket.emit('api:user.follow', {
uid: theirid
}, function(success) {
if (success) {
followBtn.addClass('hide');
unfollowBtn.removeClass('hide');
app.alertSuccess('You are now following ' + username + '!');
} else {
app.alertError('There was an error following' + username + '!');
}
});
return false;
});
Server
socket.on('api:user.follow', function(data, callback) {
if (uid) {
user.follow(uid, data.uid, callback);
}
});
NodeBB doesn't use sockets: it uses Socket.io, which is a library that enables you to use "real-time" communication between browsers and the server.
You can easily find AJAX vs. Socket.io comparisons on the internet, but here are some differences:
Socket.io uses one persistent connection between the client and the server; whereas each AJAX request uses a new connection, which means sending the lengthy HTTP headers, cookies, etc.
Socket.io messages can't be cached, AJAX requests can be
Socket.io provides namespaces, volatile messages, broadcasts...
In the particular case of a forum, the main advantage of using Socket.io is to provide real-time functionalities such as instantly displaying new messages on a thread, which are automatically pushed from the server to the client. If some messages could be sent using AJAX (such as following a user), I suppose the developers don't want to introduce the mental overhead of using two different technologies to communicate between the client and the server, as Socket.io can also handle those messages just fine.
Edit: As pointed in the comments, Socket.io selects a transport protocol, depending on the browser. Websocket is preferred, but it can switch to AJAX long polling or an iframe if needed.
I am trying to add some custom information to my socket object on connect, so that when I disconnect the socket, I can read that custom information.
IE:
// (Client)
socket.on('connect', function(data){
socket.customInfo = 'customdata';
});
// (server)
socket.on('disconnect', function () {
console.log(socket.customInfo);
});
Since it is JavaScript you can freely add attributes to any object (just as you did). However socket.io does give you a built-in way to do that (so you won't have to worry about naming conflicts):
socket.set('nickname', name, function () {
socket.emit('ready');
});
socket.get('nickname', function (err, name) {
console.log('Chat message by ', name);
});
Note that this is only on one side (either client or server). Obviously you can't share data between client and server without communication (that's what your example suggests).
The socket in your browser and the socket in the server won't share the same properties if you set them.
Basically you have set the data only at the client side (which is in your browsers memory NOT on the server).
For anyone still looking for an answer, there are a couple of things you can do to share data from the client to the server without actually sending a message.
Add a custom property to the auth property in the client socketIo options. This will be available to the server event handlers in socket.handshake.auth.xxxx.
Add a custom header to the transportOptions.polling.extraHeaders property in the client socketIo options. This will ONLY be presented when the socket.io client is connected via polling and not when "upgraded" to websockets (as you can't have custom headers then).
Add a custom query property to the client socketIo options. I don't recommend this since it potentially exposes the data to intermediate proxies.
I'd like to add a live functionality to a PHP based forum - new posts would be automatically shown to users as soon as they are created.
What I find a bit confusing is the interaction between the PHP code and NodeJS+socket.io.
How would I go about informing the NodeJS server about new posts and have the server inform the clients that are watching the thread in which the post was posted?
Edit
Tried the following code, and it seems to work, my only question is whether this is considered a good solution, as it looks kind of messy to me.
I use socket.io to listen on port 81 to clients, and the server running om port 82 is only intended to be used by the forum - when a new post is created, a PHP script sends a POST request to localhost on port 82, along with the data.
Is this ok?
var io = require('socket.io').listen(81);
io.sockets.on('connection', function(socket) {
socket.on('init', function(threadid) {
socket.join(threadid);
});
});
var forumserver = require('http').createServer(function(req, res) {
if (res.socket.remoteAddress == '127.0.0.1' && req.method == 'POST') {
req.on('data', function(chunk) {
data = JSON.parse(chunk.toString());
io.sockets.in(data.threadid).emit('new-post', data.content);
});
}
res.end();
}).listen(82);
Your solution of a HTTP server running on a special port is exactly the solution I ended up with when faced with a similar problem. The PHP app simply uses curl to POST to the Node server, which then pushes a message out to socket.io.
However, your HTTP server implementation is broken. The data event is a Stream event; Streams do not emit messages, they emit chunks of data. In other words, the request entity data may be split up and emitted in two chunks.
If the data event emitted a partial chunk of data, JSON.parse would almost assuredly throw an exception, and your Node server would crash.
You either need to manually buffer data, or (my recommendation) use a more robust framework for your HTTP server like Express:
var express = require('express'), forumserver = express();
forumserver.use(express.bodyParser()); // handles buffering and parsing of the
// request entity for you
forumserver.post('/post/:threadid', function(req, res) {
io.sockets.in(req.params.threadid).emit('new-post', req.body.content);
res.send(204); // HTTP 204 No Content (empty response)
});
forumserver.listen(82);
PHP simply needs to post to http​://localhost:82/post/1234 with an entity body containing content. (JSON, URL-encoded, or multipart-encoded entities are acceptable.) Make sure your firewall blocks port 82 on your public interface.
Regarding the PHP code / forum's interaction with Node.JS, you probably need to create an API endpoint of sorts that can listen for changes made to the forum. Depending on your forum software, you would want to hook into the process of creating a new post and perform the API callback to Node.js at this time.
Socket.io out of the box is geared towards visitors of the site being connected on the frontend via Javascript. Upon the Node server receiving notification of a new post update, it would then notify connected clients of this new post and its details, at which point it would probably add new HTML to the DOM of the page the visitor is viewing.
You may want to arrange the Socket.io part of things so that users only subscribe to specific events being emitted by them being in a specific room such as "subforum123" so that they only receive notifications of applicable posts.