I have two clients. A client sends an HTTP request to node.js express server and this writes in the DB then waits that value changes. It changes when the other client sends an HTTP request to node.js express and it UPDATE the record in the DB. Then the first request sends a response JSON to the first client.
The idea is that the first client process waits a signal from the second client process without using busy wait (like reads from deatabase every 100ms). Here the code (uniqueid is a value that is PRIMARY KEY of that table's row):
var foo = function(uniqueid, client, cb){
var query="UPDATE... WHERE uniqueid=" + db.escape(uniqueid) + ";";
db.query(query, function(err, result, fields){
if(client==1){ //if we are client1 we wait
waitClient2(uniqueid, function(callback){
//Then read the new value
getValue(uniqueid, function(v){
cb(v);
});
});
}else{ //else unlock client1
unlockClient1(uniqueid);
//Then read the new value
getValue(uniqueid, function(v){
cb(v);
});
}
});
}
var waitClient2 = function(uniqueid, cb){
//wait(uniqueid)
//cb(0) for unlock foo
}
var unlockClient1 = function(uniqueid){
//signal(uniqueid)
}
Is there a way to send a message (signal) from a client request to another client request so the second can unlocks itself editing two latest functions?
I appreciate other ideas too.
Thank you.
Related
I am currently creating a websocket server for a mobile front. and in some cases I need to send a json only to a specific user and after several attempts I still haven't managed to get Websocket to work so I can send my json to one client at a time.
i'm using this library : github.com/websockets/ws
To explain my problem i have several products that contain several variables that need to be refreshed in real time. when a user connects to a product he will receive only the json of that product and the other users will receive the json of the other products they are currently on. that's why i want to send a specific json to a user to enable this
I would like to know if any of you know how to fix the problem as I'm starting to block on it.
Thank you very much.
const opp = new WebSocket.Server({port: 10001});
let user = 0;
let lookup = [];
opp.on('connection', function connection(op) {
lookup.push(op.id);
let id = "";
op.on('message', function incoming(message) {
console.log('received: %s', message);
id = message;
query = {
text: "SELECT id,state,users_list_name,user_win,timer_stamp FROM products WHERE id = " + parseInt(id) + " AND circle = 1 ORDER BY CASE WHEN state = \'available\' THEN \'1\' WHEN state = \'soon\' THEN \'2\' WHEN state = \'expired\' THEN \'3\' END",
};
});
client.connect();
const interval = setInterval(function ping() {
client.query(query, (err, res) => {
if (err) {
console.log(err.toString());
console.log(query);
} else {
console.log(lookup);
for (let i = 0; i < lookup.length; i++){
console.log("########################");
lookup[i].send(JSON.stringify(res.rows));
}
}
});
}, 300);
});```
OK. Still trying to understand the actual spec you're shooting for. But, assuming the following (based on your answers to my prior questions):
A client connects using a webSocket.
When they send a message over that webSocket, that message is an id of something that can be looked up in your database and that they want regular updates for.
Those updates for that particular id should be sent only to that specific client that requested it.
If a different client connects and specifies some id, they should get updates for that id only.
When a client sends a new message that specifies a different id, their updates should now be only for that new id.
Updates for the id that one client requested are sent only to that one client (not to the other clients).
If that's what you really want, here's a way to structure that.
const wss = new WebSocket.Server({port: 10001});
// make database connection that all users share
client.connect();
wss.on('connection', function connection(ws) {
// these variables are unique to each ws connection
let interval, query;
// when webSocket closes, stop any current interval timer associated with this webSocket
ws.on('close', function() {
if (interval) {
clearInterval(interval);
}
});
// when we get an id, start querying for updates on that id
ws.on('message', function incoming(id) {
console.log(`received: ${id}`);
query = {
text: "SELECT id,state,users_list_name,user_win,timer_stamp FROM products WHERE id = " + parseInt(id) + " AND circle = 1 ORDER BY CASE WHEN state = \'available\' THEN \'1\' WHEN state = \'soon\' THEN \'2\' WHEN state = \'expired\' THEN \'3\' END",
};
// if interval is not already going, start it
if (!interval) {
interval = setInterval(function() {
client.query(query, (err, res) => {
if (err) {
console.log(err);
console.log(query);
} else {
// send data to just the one client that this timer is for
ws.send(JSON.stringify(res.rows));
}
});
}, 300);
}
});
});
Now, some comments:
Polling the database on a short time interval with a separate polling loop for every single client simply will not scale at all. You will have serious database scale issues. You really need a better design here, but since we don't know the overall requirements and architecture of your application, we don't have enough info to know what to suggest. Probably you want to leverage notifications in a database that tell you when data has changed rather than you polling it on a short interval on behalf of every single client.
I could find no reason for the lookup data structure. Your comments say that you want to send updates to ONE specific client, the one that requested that id. That can be done with ws.send().
This code assumes that the client variable represents a connection to your database that each of the setIntervals for each connected client can all share. That's why that code was moved out of the wss.on('connection', ...) event handler.
I switched to the more common terminology of wss to refer to the server instance and ws to refer to the webSocket for a particular connected client.
ws.send() is how you send to a connected client. I still don't know what you were doing with op.id. Looking at the doc for the ws library, that doesn't appear to be something you can use to send to.
Your code (and this code) creates a separate setInterval() timer for every webSocket client that connects and it uses a very short interval time. This will not scale. At worst, the interval time needs to be lengthened into multiple seconds (depending upon desired target scale). At best, you need to stop polling the database entirely and use some other mechanism in the database for getting notifications when data has been changed.
I built a simple API endpoint with NodeJS using Sails.js.
When someone access my API endpoint, the server starts to wait for data and whenever a new data appears, he broadcasts it using sockets. Each client should receive his own stream of data based on his user input.
var Cap = require('cap').Cap;
collect: function (req, res) {
var iface = req.param("ip");
var c = new Cap(),
device = Cap.findDevice(ip);
c.on('data', function(myData) {
sails.sockets.blast('message', {"host": myData});
});
});
The response do not complete (I never send a res.json() - what actually happens is that the browser keep loading - but the above functionality works).
2 Problems:
I'm trying to subscribe and unsubscribe to to this API endpoint from my client (using RxJS). When I subscribe, I start to receive data via sockets - but I can't unsubscribe to the API endpoint (the browser expect the request to be completed).
Each client should subscribe to his own socket room based on the request IP parameter ( see updated code ). Currently it blasts the message to everyone.
How I can create a stream/service-like API endpoint with Sails.js that will emit new data to each user based on his input?
My goal is to be able to subscribe / unsubscribe to this API endpoint from each client.
Revised Answer
Let's assume your API endpoint is defined in config/routes.js like this:
...
'get /collect': 'SomeController.collectSubscribe',
'delete /collect': 'SomeController.collectUnsubscribe',
Since each Cap instance is tied to one device, we need one instance for each subscription. Instead of using the sails join/leave methods, we keep track of Cap instances in memory and just broadcast to the request socket's id. This works because Sails sockets are subscribed to their own ids by default.
In api/controllers/SomeController.js:
// In order for the `Cap` instances to persist after `collectSubscribe` finishes, we store them all in an Object, associated with which socket the were created for.
var caps = {/* req.socket.id: <instance of Cap>, */};
module.exports = {
...
collectSubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest("I need a websocket! Help!");
if (!!caps[req.socket.id]) return res.badRequest("Dude, you are already subscribed.");
caps[req.socket.id] = new Cap();
var c = caps[req.socket.id]; // remember that `c` is a reference to our new `Cap`, not a copy.
var device = c.findDevice(req.param('ip'));
c.open(device, ...);
c.on('data', function(myData) {
sails.sockets.broadcast(req.socket.id, 'message', {host: myData});
});
return res.ok();
},
collectUnsubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest("I need a websocket! Help!");
if (!caps[req.socket.id]) return res.badRequest("I can't unsubscribe you unless you actually subscribe first.");
caps[req.socket.id].removeAllListeners('data');
delete caps[req.socket.id];
return res.ok();
}
}
Basically, it goes like this: when a browser request triggers collectSubscribe, a new Cap instance listens to the provided IP. When the browser triggers collectUnsubscribe, the server retreives that Cap instance, tells it to stop listening, and then deletes it.
Production Considerations: please be aware that the list of Caps is NOT PERSISTENT (since it is stored in memory and not a DB)! So if your server is turned off and rebooted (due to lightning storm, etc), the list will be cleared, but considering that all websocket connections will be dropped anyway, I don't see any need to worry about this.
Old Answer, Kept for Reference
You can use sails.sockets.join(req, room) and sails.sockets.leave(req, room) to manage socket rooms. Essentially you have a room called "collect", and only sockets joined in that room will receive a sails.sockets.broadcast(room, eventName, data).
More info on how to user sails.sockets here.
In api/controllers/SomeController.js:
collectSubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest();
sails.sockets.join(req, 'collect');
return res.ok();
},
collectUnsubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest();
sails.sockets.leave(req, 'collect');
return res.ok();
}
Finally, we need to tell the server to broadcast messages to our 'collect' room.
Note that this only need to happen once, so you can do this in a file under the config/ directory.
For this example, I'll put this in config/sockets.js
module.exports = {
// ...
};
c.on('data', function(myData) {
var eventName = 'message';
var data = {host: myData};
sails.sockets.broadcast('collect', eventName, data);
});
I am assuming that c is accessible here; If not, you could define it as sails.c = ... to make it globally accessible.
I'm building a small node server that generates PDF files (using Nightmare.js). Each request calls createPage to generate one pdf.
The incoming request tend to all come around the same time, overloading the PC this is running on.
I need to buffer the incoming requests to delay execution of some requests till some of the current requests have completed. How do I do this?
function createPage(o, final) {
//generate pdf files
}
http.createServer(function (request, response) {
var body = [];
request.on('data', function (chunk) {
body.push(chunk);
}).on('end', function () {
body = Buffer.concat(body).toString();
var json = JSON.parse(body);
createPage(json, function (status) {
if (status === true) {
response.writeHead(200, { 'Content-Length': 0 });
console.log('status good');
} else {
response.writeHead(500, { 'Content-Type': 'text/html' });
response.write(' ' + status);
}
response.end('\nEnd of Request \n');
});
});
}).listen(8007);
If I understand correctly, you want to continually accept http requests but throttle the rate at which createPage is invoked. If so, you probably need to consider a slightly different design. In this current design, every next client will have to wait longer than the previous one to find out if their request has succeeded or failed.
Approach 1:
use a queue (rabbitmq, aws sqs, zeromq, kafka, etc).
Here's the basic workflow:
receive the request
generate a unique id
put a message on the queue that includes the data and the unique id
return the unique id to the client
the client periodically checks for the completion of the task using the unique id
Approach 2:
Use a queue with message duplexing.
receive the request
generate a correlation id and relate it to the http transaction
send message on queue to worker with correlation id
when worker completes, it sends the response back with the correlation id
server uses correlation id to find the http transaction and send the appropriate response to the client
I have a NodeJS API web server (let's call it WS1) that receives RESTful HTTP requests from clients, and to respond needs to first query another local server (let's call it WS2).
The flow is pretty much like this:
WS1 receives an HTTP request from a client and parses it.
WS1 sends a request to WS2 for some information.
When WS1 receives the response from WS2, it finishes processing the original request and sends a response back to the client.
Until now all communication between WS1 and WS2 has been done through HTTP requests, since the two machines are on the same local network.
To speed things up though I'm considering to start using zmq instead. I've looked at the patterns they show on the docs, but still haven't figured out a concurrency problem.
WS1 can send many requests per second to WS2, and there's no guarantee that WS2 replies in the same order as it receives the requests, since some async operations can internally take longer than others.
So, using zmq with NodeJS, how do I make sure that when WS1 receives a message from WS2 it knows to what original client request it belongs to? Is there a built-in mechanism to take care of it?
Thanks!
0MQ is an interesting tool set that helps abstract socket communication. There are mechanism (should you choose the correct socket types) that allow the server to respond to the right client, and it is handled within the confines of 0mq.
The basic API types are:
PUSH-PULL
PUB-SUB
REQUEST-REPLY
IF you want to be able to have one machine respond to the originator, then I believe you want REQ-REP api type.
then you need to consider the multi-plexing on each side to get the connectors correct. But keep it one to one for simplicity sake at first:
Sample Client (from http://zguide.zeromq.org/js:rrclient
// Hello World client in Node.js
// Connects REQ socket to tcp://localhost:5559
// Sends "Hello" to server, expects "World" back
var zmq = require('zmq')
, requester = zmq.socket('req');
requester.connect('tcp://localhost:5559');
var replyNbr = 0;
requester.on('message', function(msg) {
console.log('got reply', replyNbr, msg.toString());
replyNbr += 1;
});
for (var i = 0; i < 10; ++i) {
requester.send("Hello");
}
sample server (from http://zguide.zeromq.org/js:rrserver)
// Hello World server in Node.js
// Connects REP socket to tcp://*:5560
// Expects "Hello" from client, replies with "World"
var zmq = require('zmq')
, responder = zmq.socket('rep');
responder.connect('tcp://localhost:5560');
responder.on('message', function(msg) {
console.log('received request:', msg.toString());
setTimeout(function() {
responder.send("World");
}, 1000);
});
The routing of the reply back to the client is handled automatically by 0MQ. it is part of the message (although I don't remember if you see the address buffer in these examples - it maybe abstracted away). Here is what the request envelope looks like:
it is the first frame, which allows 0MQ to be able to reply to the correct client.
Once that is running you can then consider 1..* *..1 and ... All it really does is require you to change the socket types to DEALER and ROUTER where appropriate.
I ended up implementing some sort of "middleware" to support this functionality with zmq.
In the example below for simplicity I've used Express with Node >= v4.0.0 (supporting native JS promises), but you can obviously substitute it with any HTTP server you like (these days I prefer Koa) and promises library you prefer. This is the code for the two servers.
WS1 (requester)
var zmq = require('zmq');
var mem = {};
var requester = zmq.socket('req');
requester.on("message", function(reply) {
reply = reply.toString().split('*');
mem[reply.pop()](reply);
});
requester.connect("tcp://localhost:5555");
var app = require('express')();
app.get('/', function (req, res) {
var id = Date.now() + Math.random();
new Promise(function (resolve, reject) {
mem[id] = function (reply) {
reply[0] === 'success' ? resolve(reply[1]) : reject(reply[1]);
}
})
.then(function (data) {
res.send(data);
})
.catch(function (err) {
console.log(err);
res.send(500);
})
requester.send(id + '*' + message);
});
var server = app.listen(3000);
WS2 (responder)
var zmq = require('zmq');
var responder = zmq.socket('rep');
responder.on('message', function(message) {
message = message.split('*');
var reqId = message[0];
// Do whatever async stuff you need with message[1]
// Then at the end of your callbacks you'll have something like this
if (err) {
responder.send('err' + '*' + JSON.stringify(err) + '*' + reqId);
} else {
responder.send('success' + '*' + JSON.stringify(yourData) + '*' + reqId);
}
});
responder.bind('tcp://*:5555');
I'm really new to node.js so please bear with me if I'm making a obvious mistake.
To understand node.js, i'm trying to create a webserver that basically:
1) update the page with appending "hello world" everytime the root url (localhost:8000/) is hit.
2) user can go to another url (localhost:8000/getChatData) and it will display all the data built up from the url (localhost:8000/) being triggered
Problem I'm experiencing:
1) I'm having issue with displaying that data on the rendered page. I have a timer that should call get_data() ever second and update the screen with the data variable that stores the appended output. Specifically this line below response.simpleText(200, data); isn't working correctly.
The file
// Load the node-router library by creationix
var server = require('C:\\Personal\\ChatPrototype\\node\\node-router').getServer();
var data = null;
// Configure our HTTP server to respond with Hello World the root request
server.get("/", function (request, response) {
if(data != null)
{
data = data + "hello world\n";
}
else
{
data = "hellow world\n";
}
response.writeHead(200, {'Content-Type': 'text/plain'});
console.log(data);
response.simpleText(200, data);
response.end();
});
// Configure our HTTP server to respond with Hello World the root request
server.get("/getChatData", function (request, response) {
setInterval( function() { get_data(response); }, 1000 );
});
function get_data(response)
{
if(data != null)
{
response.writeHead(200, {'Content-Type': 'text/plain'});
response.simpleText(200, data);
console.log("data:" + data);
response.end();
}
else
{
console.log("no data");
}
}
// Listen on port 8080 on localhost
server.listen(8000, "localhost");
If there is a better way to do this, please let me know. The goal is to basically have a way for a server to call a url to update a variable and have another html page to report/display the updated data dynamically every second.
Thanks,
D
The client server model works by a client sending a request to the server and the server in return sends a response. The server can not send a response to the client that the client hasn't asked for. The client initiates the request. Therefore you cannot have the server changing the response object on an interval.
The client will not get these changes to the requests. How something like this is usually handled as through AJAX the initial response from the server sends Javascript code to the client that initiates requests to the server on an interval.
setTimeout accepts function without parameter which is obvious as it will be executed later in time. All values you need in that function should be available at the point of time. In you case, the response object that you are trying to pass, is a local instance which has scope only inside the server.get's callback (where you set the setTimeout).
There are several ways you can resolve this issue. you can keep a copy of the response instance in the outer scope where get_data belongs or you can move the get_data entirely inside and remove setTimeout. The first solution is not recommended as if getChatData is called several times in 1sec the last copy will be prevailing.
But my suggestion would be to keep the data in database and show it once getChatData is called.