I'm trying to write a simple asynchronous TCP/IP client that runs alongside a Meteor server for communicating to a remote server and posting data to MongoDB. I got it working using net.on callbacks, but the code was messy and it was failing at random times. I decided to try writing it using fibers/Futures to clean it up so I could focus on the failures. The code currently looks like:
var Future = Npm.require('fibers/future'), wait = Future.wait;
var coreComm = function(coreClient) {
console.log('coreCommm started')
try {
var running = true
while (running) {
console.log('calling onF.wait()')
var ev = onF.wait();
console.log('ev received', ev)
switch(ev[0]) {
default:
console.log('unknown event from coreClient: ', ev)
break;
case 'readable':
console.log('read', ev)
break;
}
}
} catch(err) {
console.log('comm error: ', err)
}
}.future()
function tryConnect(options) {
var connect = new Future
onF = new Future
coreClient = net.connect(options, function() {
console.log('connected,')
connect.return()
})
connect.wait()
coreClient.on('readable',
function() { console.log('readable event,'); onF.return(['readable'])})
console.log('coreClient connected to core');
coreComm(coreClient)
}
Meteor.startup(function () {
tryConnect({port: 9987});
}
The output when a message is sent looks like:
=> Meteor server running on: http://localhost:3000/
I2038-10:42:18.160(-5)? starting
I2038-10:42:18.392(-5)? connected,
I2038-10:42:18.398(-5)? coreClient connected to core
I2038-10:42:18.402(-5)? coreCommm started
I2038-10:42:18.409(-5)? calling onF.wait()
I2038-10:42:18.413(-5)? readable event,
As far as I can tell, the message is received from the remote server, the readable event is sent, I call onF.return(...) and nothing happens except Meteor goes to 100% CPU.
Any suggestions as to why the onF.wait() call isn't returning like it's suppose to?
Related
Probably this is already answered question but somehow the solutions that I found would not work. It is better to describe.
I have an embedded linux device which has a C based application running and nodejs is running as a web server. I use TCP socket for inter-process communication.
Main problem is that C application is designed to run in a syncronous way but not nodejs.
Web page sends requests to nodejs and nodejs asks to C application and wait for application result to send response to web page.
Usually this is not a problem since everything runs very fast. But in the embed side, now there are slow processes thus it makes nodejs wait.
var counterVal = 0;
app.post('/ajax.html', jsonParser, async function (req, res) {
var q = url.parse(req.url, true);
var formData = req.body;
var cMessage = { counter : counterVal };
var message = { ...q.query, ...formData, ...cMessage };
await remoteSocket.write(JSON.stringify(message), function ()
{
remoteSocket.on('data', function (e) {
try
{
remoteSocket._events.data = function (data) { };
var ret = JSON.parse(e);
if (message.counter == ret.counter)
{
res.send(e);
}
else
{
logError("received error : "+ message.counter + ": " + ret.counter);
}
}
catch (err) {
logError(err);
}
});
});
});
I use this code to handle web requests. Counter value is used as handshake. Some of the requests gets into logError because of mismatch.
But when i look into wireshark records packet by packet, there is no mismatch. ever request has correct response.
app.post is async so data listener is registered twice. Then response for a packet triggers two listeners but one listener is for next packet. Therefore is gives a mismatch.
What is a way to make app.post to run in sync mode?
I'm trying to implement and existing solution in node.js, specifically, using express.js framework. Now, the existing solution works as follows:
server exposes a GET service that clients can connect to
when a client calls the GET service, the client number increments (a global variable) and then the number of clients is checked;
if there are not at least 3 clients connected, the service is in endless loop, waiting for other clients to connect
if (or rather, when) the rest of the two clients connect, the service sends respond to everyone that enough clients are connected (a 'true' value).
So what basically happens is, the client connects and the connection is active (in a loop) until enough clients connect, then and only then there is a response (to all clients at the same time).
Now I'm not expert in these architectures, but from what I think, this is not a correct or good solution. My initial thought was: this must be solved with sockets. However, since the existing solution works like that (it's not written in node.js), I tried to emulate such behaviour:
var number = (function(){
var count = 0;
return {
increase: function() {
count++;
},
get: function(){
return count;
}
};
})();
app.get('/test', function(req, res){
number.increase();
while (number.get() < 3) {
//hold it here, until enough clients connect
}
res.json(number.get());
});
Now while I think that this is not a correct solution, I have a couple of questions:
Is there any alternative to solving this issue, besides using sockets?
Why does this "logic" work in C#, but not in express.js? The code above hangs, no other request is processed.
I know node.js is single-threaded, but what if we have a more conventional service that responds immediately, and there are 20 requests all at the same time?
I would probably use an event emitter for this:
var EventEmitter = require('events').EventEmitter;
var emitter = new EventEmitter();
app.get('/', function(req, res) {
// Increase the number
number.increase();
// Get the current value
var current = number.get();
// If it's less than 3, wait for the event emitter to trigger.
if (current < 3) {
return emitter.once('got3', function() {
return res.json(number.get());
});
}
// If it's exactly 3, emit the event so we wake up other listeners.
if (current === 3) {
emitter.emit('got3');
}
// Fall through.
return res.json(current);
});
I would like to stress that #Plato is correct in stating that browsers may timeout when a response takes too much time to complete.
EDIT: as an aside, some explanation on the return emitter.once(...).
The code above can be rewritten like so:
if (current < 3) {
emitter.once('got3', function() {
res.json(number.get());
});
} else if (current === 3) {
emitter.emit('got3');
res.json(number.get());
} else {
res.json(number.get());
}
But instead of using those if/else statements, I return from the request handler after creating the event listener. Since request handlers are asynchronous, their return value is discarded, so you can return anything (or nothing). As an alternative, I could also have used this:
if (current < 3) {
emitter.once(...);
return;
}
if (current === 3) {
...etc...
Also, even though you return from the request handler function, the event listener is still referencing the res variable, so the request handler scope is maintained by Node until res.json() in the event listener callback is called.
Your http approach should work
You are blocking the event loop so node refuses to do any other work while it is in the while loop
You're really close, you just need to check every now and then instead of constantly. I do this below with process.nextTick() but setTimeout() would also work:
var number = (function(){
var count = 0;
return {
increase: function() {
count++;
},
get: function(){
return count;
}
};
})();
function waitFor3(callback){
var n = number.get();
if(n < 3){
setImmediate(function(){
waitFor3(callback)
})
} else {
callback(n)
}
}
function bump(){
number.increase();
console.log('waiting');
waitFor3(function(){
console.log('done');
})
}
setInterval(bump, 2000);
/*
app.get('/test', function(req, res){
number.increase();
waitFor3(function(){
res.json(number.get());
})
});
*/
Sorry if the question title is a tad ambiguous, but I'm not entirely sure how to word it.
I'm writing an NPM module that talks to a json-rpc api - this is the current setup.
// The module
function MyModule(config) {
// do some connection stuff here
connected = true
}
MyModule.prototype.sendCommand = function() {
if(connected) {
// do command
} else {
// output an error
}
}
module.exports = MyModule;
// The script interacting with the module
var MyModule = require('./MyModule');
var config = {
// config stuff
};
var mod = new MyModule(config);
var mod.sendCommand;
The command won't send, as at this point it hasn't connected, I assume this is due to NodeJS' asynchronous, non-blocking architecture and that I perhaps need to use promises to wait for a response from the API, where would I implement this? Do I do it in my module or do I do it in the script interacting with the module?
You will need to use either a callback or promises or something like that to indicate when the connection is complete so you can then use the connection in further code that is started via that callback.
Though it is generally not considered the best practice to do asynchronous stuff in a constructor, it can be done:
function MyModule(config, completionCallback) {
// do some connection stuff here
connected = true
completionCallback(this);
}
var mod = new MyModule(config, function(mod) {
// object has finished connecting
// further code can run here that uses the connection
mod.sendCommand(...);
});
A more common design pattern is to not put the connecting in the constructor, but to add a method just for that:
function MyModule(config) {
}
MyModule.prototype.connect = function(fn) {
// code here that does the connection and calls
// fn callback when connected
}
var mod = new MyModule(config);
mod.connect(function() {
// object has finished connecting
// further code can run here that uses the connection
mod.sendCommand(...);
});
don't use promises, use node's programming model where you don't "call functions" but you "call functions with a result handler for dealing with the data once it's actually available":
MyModule.prototype.sendCommand = function(handler) {
if(connected) {
// run stuff, obtain results, send that on:
handler(false, result);
} else {
// output an error, although really we should
// just try to connect if we're not, and say
// there's an error only when it actually fails.
handler(new Error("ohonoes"));
}
}
and then you call the function as
var MyModule = require('./MyModule');
var mod = ...
mod.sendCommand(function(err, result) {
// we'll eventually get here, at which point:
if (err) { return console.error(err); }
run();
more();
code();
withResult(result);
});
I have code to log every connection to my HTTP-server on a socket level and also log any incoming data.
This code was originally written for NodeJS 0.8 and works good there.
No my project is migrated to 0.10.24 and socket logging code stopped working.
Here is my code:
var netLogStream = fs.createWriteStream('net.log');
(function(f) {
net.Server.prototype.listen = function(port) {
var rv = f.apply(this, arguments); // (1)
rv.on('connection', function(socket) { // (2)
socket.on('data', function(data) {
data.toString().split('\n').forEach(function(line) { // (3)
netLogStream.write('... some logging here ... ' + line);
});
});
});
return rv;
};
})(net.Server.prototype.listen);
On 0.10 I can get to (1) and get Socket instance on (2) but I never get to (3). Same time my whole application works fine without any issues.
ADD: My server is created with Express#3.4.x
I'm not sure why the results are different between node v0.8 and v0.10, but if I had to guess, I'd be looking at the return value of net.Server.prototype.listen.
According to the documentation, this is an asynchronous method which emits the 'listen' event and invokes its callback when the listening is bound. You're not looking for that event, but rather, capturing the return value of listen, which for an async function, may not be well-defined. It's obviously not null or undefined since you don't get a runtime error, but the return value may not be the same between v0.8 and v0.10.
I honestly don't know for sure because I don't do low-level socket coding, but I have 2 suggestions to try:
Since the connection event is emitted from the Server object, perhaps you need this.on instead of rv.on.
Setup the connection event listener before you invoke listen just to minimize risk of race conditions.
Try this and see what happens:
var netLogStream = fs.createWriteStream('net.log');
(function(f) {
net.Server.prototype.listen = function(port) {
this.on('connection', function(socket) { // (2)
socket.on('data', function(data) {
data.toString().split('\n').forEach(function(line) { // (3)
netLogStream.write('... some logging here ... ' + line);
});
});
});
return f.apply(this, arguments); // (1)
};
})(net.Server.prototype.listen);
Trying to send data from a serial device to web clients. I am using a serial to network proxy, ser2Net to make the data available to a server that acts on the data and sends a manipulated version of the data to web clients. The clients specify the location of the ser2net host and port. The core of this action is coded in node.js as shown here:
function getDataStream(socket, dataSourcePort, host) {
var dataStream = net.createConnection(dataSourcePort, host),
dataLine = "";
dataStream.on('error', function(error){
socket.emit('error',{message:"Source not found on host:"+ host + " port:"+dataSourcePort});
console.log(error);
});
dataStream.on('connect', function(){
socket.emit('connected',{message:"Data Source Found"});
});
dataStream.on('close', function(){
console.log("Close socket");
});
dataStream.on('end',function(){
console.log('socket ended');
dataConnection.emit('lost',{connectInfo:{host:host,port:dataSourcePort}});
});
dataStream.on('data', function(data) {
// Collect a line from the host
line += data.toString();
// Split collected data by delimiter
line.split(delimiter).forEach(function (part, i, array) {
if (i !== array.length-1) { // Fully delimited line.
//push on to buffer and emit when bufferSendCommand is present
dataLine = part.trim();
buffer.push(part.trim());
if(part.substring(0, bufferSendCommand.length) == bufferSendCommand){
gotALine.emit('new', buffer);
buffer=[];
}
}
else {
// Last split part might be partial. We can't announce it just yet.
line = part;
}
});
});
return dataStream;
}
io.sockets.on('connection', function(socket){
var stream = getDataStream(socket, dataSourcePort, host);
//dispense incoming data from data server
gotALine.on('new', function(buffer){
socket.emit('feed', {feedLines: buffer});
});
dataConnection.on('lost', function(connectInfo){
setTimeout(function(){
console.log("Trying --- to reconnect ");
stream = getDataStream(socket, connectInfo.port, connectInfo.host);
},5000);
});
// Handle Client request to change stream
socket.on('message',function(data) {
var clientMessage = JSON.parse(data);
if('connectString' in clientMessage
&& clientMessage.connectString.dataHost !== ''
&& clientMessage.connectString.dataPort !== '') {
stream.destroy();
stream = getDataStream(socket,
clientMessage.connectString.dataPort,
clientMessage.connectString.dataHost);
}
});
});
This works well enough until the serial device drops off and ser2net stops sending data. My attempt to catch the end of the socket and reconnect is not working. The event gets emitted properly but the setTimeout only goes once. I would like to find a way to keep on trying to reconnect while sending a message to the client informing or retry attempts. I am node.js newbie and this may not be the best way to do this. Any suggestions would be appreciated.
Ok I think I figured it out in the dataStream.on('data' ... I added a setTimeout
clearTimeout(connectionMonitor);
connectionMonitor = setTimeout(function(){doReconnect(socket);}, someThresholdTime);
The timeout executes if data stops coming in, as it is repeatedly cleared each time data comes in. The doReconnect function keeps trying to connect and sends a message to the client saying something bad is going on.