I am creating a node application that will send UDP packets to another machine. Everything appears to work fine when packets are sent about 100/second. When that number gets higher 500/second the packets stop sending. I don't think they are being droped as Wireshark does not show them leaving my computer.
const dgram = require("dgram");
var sent = 0;
var server = dgram.createSocket("udp4");
function sendCommand(text){
// Put command into a buffer
var buffer = new Buffer(text);
// Send the command
server.send(buffer, 0, buffer.length, 12345, 19.5.6.45, (err) => {
sent++;
});
}
The sendCommand() function is what is called internally. Like I said at 100 calls per second no problem. When it gets higher it just stops working.
The "sent" var stops incrementing so this leads me to think node dgram is faulty and server.send() is not firing correctly. It doesn't seem to recover unless I restart the program.
This is my first time posting on Stack Overflow, I apologise if I did anything wrong.
Edit:
Here is where the function is firing from.
// Loop through the channels
for (var i = 0; i < ids.length; i++) {
// Check if the channel is locked
if (!toBool(data.locks[i])) {
// Send the UDP command
sendCommand(ids[i]);
}
}
Related
First time posting here so sorry if the question is too vague or missing info. Anyway:
I'm trying to send OSC (open sound control) or MQTT messages when I hit specific notes on my midi controller. When I try to send a OSC message when I get a midi input, I get this unpredictable high latency, varying from about 200-300 milliseconds up to 10 seconds. Here is my test-code:
var easymidi = require('easymidi');
var osc = require("osc");
easymidi.getInputs().forEach((midiDevice)=> {
if (midiDevice.includes('Maschine')){
inputMidi = midiDevice
console.log(`Midi device input found: ${inputMidi}`)
}
})
var inputMikro = new easymidi.Input(inputMidi)
var rmeOSC = new osc.UDPPort ({
remoteAddress: "192.168.10.148",
remotePort: 9001
});
rmeOSC.open();
inputMikro.on('cc', function (msg) {
if (msg.controller == 7){
// This is instant
console.log(msg)
// This is what becomes unpredictable and slow
rmeOSC.send({
address: "/1/mastervolume",
args: [
{
type: "f",
value: msg.value / 127
}
]
});
}
}
I've tried to zero in what makes it slow. When just logging to console inside inputMikro.on('cc', ... ) event listener, it's instant. If I send that midi data to another midi device in that same event listener (also using easymidi library), it's also instant. But sending OSC or MQTT messages there creates that unpredictable latency. I've also tried setting an interval every 20 to send OSC just to make sure it's not a limitation of the OSC library in itself:
var vol = 0
var sendOsc = (()=>{
vol++
if (vol == 127){
vol = 0
}
rmeOSC.send({
address: "/1/mastervolume",
args: [
{
type: "f",
value: vol / 127
}
]
});
})
setInterval(sendOsc, 20)
That works great, same with MQTT. I also tried creating a separate volume variable with an event listener that listens for changes in that variable, and update that variable inside the inputMikro.on('cc', ... ) listener. Same there, if i just log to console whenever that variable changes, it's instant, but if I try to send OSC or MQTT messages when the variable changes, I get that latency.
I'm out of ideas and have no idea what's going on. I'd very much appreciate any insight on how I can fix this. I hope my question is clear enough.
I'm using socket.io like this
Client:
socket.on('response', function(i){
console.log(i);
});
socket.emit('request', whateverdata);
Server:
socket.on('request', function(whateverdata){
for (i=0; i<10000; i++){
console.log(i);
socket.emit('response', i);
}
console.log("done!");
});
I need output like this when putting the two terminals side by side:
Server Client
0 0
1 1
. (etc) .
. .
9998 9998
9999 9999
done!
But instead I am getting this:
Server Client
0
1
. (etc)
.
9998
9999
done!
0
1
.
. (etc)
9998
9999
Why?
Shouldn't Socket.IO / Node emit the message immediately, not wait for the loop to complete before emitting any of them?
Notes:
The for loop is very long and computationally slow.
This question is referring to the socket.io library, not websockets in general.
Due to latency, waiting for confirmation from the client before sending each response is not possible
The order that the messages are received is not important, only that they are received as quickly as possible
The server emits them all in a loop and it takes a small bit of time for them to get to the client and get processed by the client in another process. This should not be surprising.
It is also possible that the single-threaded nature of Javascript in node.js prevents the emits from actually getting sent until your Javascript loop finishes. That would take detailed examination of socket.io code to know for sure if that is an issue. As I said before if you want to 1,1 then 2,2 then 3,3 instead of 1,2,3 sent, then 1,2,3 received you have to write code to force that.
If you want the client to receive the first before the server sends the 2nd, then you have to make the client send a response to the first and have the server not send the 2nd until it receives the response from the first. This is all async networking. You don't control the order of events in different processes unless you write specific code to force a particular sequence.
Also, how do you have client and server in the same console anyway? Unless you are writing out precise timestamps, you wouldn't be able to tell exactly what event came before the other in two separate processes.
One thing you could try is to send 10, then do a setTimeout(fn, 1) to send the next 10 and so on. That would give JS a chance to breathe and perhaps process some other events that are waiting for you to finish to allow the packets to get sent.
There's another networking issue too. By default TCP tries to batch up your sends (at the lowest TCP level). Each time you send, it sets a short timer and doesn't actually send until that timer fires. If more data arrives before the timer fires, it just adds that data to the "pending" packet and sets the timer again. This is referred to as the Nagle's algorithm. You can disable this "feature" on a per-socket basis with socket.setNoDelay(). You have to call that on the actual TCP socket.
I am seeing some discussion that Nagle's algorithm may already be turned off for socket.io (by default). Not sure yet.
In stepping through the process of socket.io's .emit(), there are some cases where the socket is marked as not yet writable. In those cases, the packets are added to a buffer and will be processed "later" on some future tick of the event loop. I cannot see exactly what puts the socket temporarily in this state, but I've definitely seen it happen in the debugger. When it's that way, a tight loop of .emit() will just buffer and won't send until you let other events in the event loop process. This is why doing setTimeout(fn, 0) every so often to keep sending will then let the prior packets process. There's some other event that needs to get processed before socket.io makes the socket writable again.
The issue occurs in the flush() method in engine.io (the transport layer for socket.io). Here's the code for .flush():
Socket.prototype.flush = function () {
if ('closed' !== this.readyState &&
this.transport.writable &&
this.writeBuffer.length) {
debug('flushing buffer to transport');
this.emit('flush', this.writeBuffer);
this.server.emit('flush', this, this.writeBuffer);
var wbuf = this.writeBuffer;
this.writeBuffer = [];
if (!this.transport.supportsFraming) {
this.sentCallbackFn.push(this.packetsFn);
} else {
this.sentCallbackFn.push.apply(this.sentCallbackFn, this.packetsFn);
}
this.packetsFn = [];
this.transport.send(wbuf);
this.emit('drain');
this.server.emit('drain', this);
}
};
What happens sometimes is that this.transport.writable is false. And, when that happens, it does not send the data yet. It will be sent on some future tick of the event loop.
From what I can tell, it looks like the issue may be here in the WebSocket code:
WebSocket.prototype.send = function (packets) {
var self = this;
for (var i = 0; i < packets.length; i++) {
var packet = packets[i];
parser.encodePacket(packet, self.supportsBinary, send);
}
function send (data) {
debug('writing "%s"', data);
// always creates a new object since ws modifies it
var opts = {};
if (packet.options) {
opts.compress = packet.options.compress;
}
if (self.perMessageDeflate) {
var len = 'string' === typeof data ? Buffer.byteLength(data) : data.length;
if (len < self.perMessageDeflate.threshold) {
opts.compress = false;
}
}
self.writable = false;
self.socket.send(data, opts, onEnd);
}
function onEnd (err) {
if (err) return self.onError('write error', err.stack);
self.writable = true;
self.emit('drain');
}
};
Where you can see that the .writable property is set to false when some data is sent until it gets confirmation that the data has been written. So, when rapidly sending data in a loop, it may not be letting the event come through that signals that the data has been successfully sent. When you do a setTimeout() to let some things in the event loop get processed that confirmation event comes through and the .writable property gets set to true again so data can again be sent immediately.
To be honest, socket.io is built of so many abstract layers across dozens of modules that it's very difficult code to debug or analyze on GitHub so it's hard to be sure of the exact explanation. I did definitely see the .writable flag as false in the debugger which did cause a delay so this seems like a plausible explanation to me. I hope this helps.
I'm using the code below to try send OSC messages to a computer on the network. I'm using a package called osc.
I'm unable to send messages to the machine running the OSC server and receive the error below when attempting to send OSC messages:
Error: Uncaught, unspecified "error" event. (Can't send packets on a closed osc.Port object. Please open (or reopen) this Port by calling open().)
Code
let osc = require('osc');
let oscUDP = new osc.UDPPort({
remoteAddress: "192.168.1.5",
remotePort: 8004
});
oscUDP.send({
address: "/carrier/frequency",
args: 440
});
oscUDP.open();
If I put oscUDP.open() before the send call I get a different error:
Error: send EINVAL 192.168.1.5:8004
at Object.exports._errnoException (util.js:1007:11)
at exports._exceptionWithHostPort (util.js:1030:20)
at SendWrap.afterSend [as oncomplete] (dgram.js:402:11)
I am running OSCulator on OSX as the server. The code above lives on a different machine. When I run nmap on the IP address the port is open:
nmap 192.168.1.5 -p 8004
Starting Nmap 6.40 ( http://nmap.org ) at 2016-08-30 08:22 BST
Nmap scan report for 192.168.1.5
Host is up (0.13s latency).
PORT STATE SERVICE
8004/tcp open unknown
If I use osc-cli the messages are received on the machine running the OSC server:
osc --host 192.168.1.5:8004 /test 1 2 3
So it would seem the problem isn't with closed ports at all as the messages are sent and received when using osc-cli.
Any ideas?
I know I'm coming to this quite late, and it looks like you found a different library that works for you, but I thought a response might be helpful for others who are facing this issue. I'm the developer of osc.js, the original library you were trying to use.
First off, as background information, osc.js is factored into two different layers:
The low-level API that provides functions for reading and writing OSC messages and bundles to/from Typed Arrays.
The higher-level, event-based Port API, which provides a collection of platform-specific transport objects, which offer an easy way to do bidirectional communication over protocols like UDP, Web Sockets, etc.
In the case of your example code, you were trying to send an OSC message on your UDPPort object prior to it being ready. When you open() a Port, it may need to perform asynchronous operations such as opening up a socket, etc. As a result, it fires an event (aptly called ready) when the Port is all set to be used. Until ready fires, you won't be able to send or receive OSC packets.
So in the case of your original code, it looks like you were assuming that this line was synchronous and that you could call send() immediately afterwards:
oscUDP.open();
Instead, you just needed to listen for the ready event prior to attempting to send a message on the Port. Like this:
oscUDP.on("ready", function () {
oscUDP.send({
address: "/carrier/frequency",
args: 440
});
});
The osc.js Node.js example illustrates this pattern. But when I saw your question, I realized that the sample code in the osc.js README was a bit ambiguous in this regard. I have improved the event documentation and the inline README sample code to be more clear in this regard. Sorry for the confusion.
There are cases, perhaps such as yours, where the higher-level API isn't quite what you need. osc.js also provides functions for easily encoding an OSC packet as a Uint8Array, which can be converted into a Node.js buffers. So you could have done something similar to your solution just by using osc.js' osc.writeMessage() function. It has always been quite well documented, fortunately. Here's your example, modified to use osc.js' low-level API:
const dgram = require('dgram');
const client = dgram.createSocket('udp4');
const osc = require('osc');
const HOST = '192.168.1.5';
const PORT = 8004;
process.on('SIGINT', function() {
client.close();
});
let oscNoteMessage = function(note, value) {
var message = osc.writeMessage({
address: '/note/' + note,
args: [
{
type: 'i',
value: value
}
]
});
return Buffer.from(message);
}
let noteOn = function(note) {
return oscNoteMessage(note, 1);
}
let noteOff = function(note) {
return oscNoteMessage(note, 0);
}
let send = function(message) {
client.send(message, PORT, HOST, function(err, bytes) {
if(err) throw new Error(err);
})
}
send(noteOn('c'));
setTimeout(function() {
send(noteOff('c'));
}, 1000);
Anyway, I'm glad you were able to come up with a solution that works for your project, and I hope this response helps other users who may encounter similar issues. And of course, feel free to ask questions or file issues on the osc.js issue tracker.
Best regards, and apologies for the trouble you experienced using the library!
I figured it's actually pretty easy to send OSC data over UDP without the need for any packages except a2r-osc which is used for encoding OSC data.
I'm posting the solution incase anyone else in interested:
const dgram = require('dgram');
const client = dgram.createSocket('udp4');
const osc = require('a2r-osc');
const HOST = '192.168.1.5';
const PORT = 8004;
process.on('SIGINT', function() {
client.close();
});
let noteOn = function(note) {
return new osc.Message('/note/' + note, 'i', 1).toBuffer();
}
let noteOff = function(note) {
return new osc.Message('/note/' + note, 'i', 0).toBuffer();
}
let send = function(message) {
client.send(message, PORT, HOST, function(err, bytes) {
if(err) throw new Error(err);
})
}
send(noteOn('c'));
setTimeout(function() {
send(noteOff('c'));
}, 1000);
I am trying to receive and send data from a vacuum gauge (previous Model of https://www.pfeiffer-vacuum.com/en/products/measurement/digiline/gauges/?detailPdoId=13238&request_locale=en_US) with a computer (Linux 16.04) via an USB-to-RS485-Interface (the half-duplex USB485-STISO from http://www.hjelmslund.dk/). When I send a request to the gauge using a specific protocol it is supposed to answer to the request and I should be able to receive it with the interface. I managed to send data but whenever I send data, it seems that nothing comes back. I'm trying to do this with Node.js. The Code that I used so far is:
function pack(address, action, parameter, data) {
var length = String('00' + data.length.toString()).slice(-2);
var bufferAsString = address + action + parameter + length + data;
var check = 0;
for (var i = 0; i < bufferAsString.length; ++i) {
check += bufferAsString.charCodeAt(i)
}
var checkSum = String('000' + String(check % 256)).slice(-3);
var buffer = Buffer.from(bufferAsString + checkSum),
carriageReturn = Buffer.from('\r');
return Buffer.concat([buffer, carriageReturn]);
}
var serialPort = require('serialport');
var SerialPort = serialPort.SerialPort;
var port = new SerialPort('/dev/ttyUSB0', {
baudrate: 9600,
dataBits: 8,
stopBits: 1,
parity: 'none'
}, false);
port.open(function(err) {
if (err) {
return console.log('Error opening port: ', err.message);
}
console.log(port.isOpen());
port.on('data', function (data) {
console.log('Data: ' + data);
});
port.on('close', function() {
console.log('port closed')
});
var sendBuffer = pack('001', '00', '740', '=?');
setInterval(function() {
port.write(sendBuffer, function(err, bytes) {
console.log('send' + bytes)
});
port.drain();
}, 1000)
});
That is supposed to send a request every second to the gauge to measure the pressure. I know that the request is being send since the TxD-Led blinks shortly every second. But I receive no answer to that request.
I also tried other methods of sending data (mostly via python and the terminal) but with similar success. The green lamp for sending always flashes up but then nothing happens and no answer is received.
I am at a loss as to what to try next and would really appreciate any help that you could give me.
UPDATE:
Ok so I seem to have found one possible error in the whole thing. I was working with an oszilloscope to capture the signal that is going out of the interface when I send something. I started with single ascii-characters to see if the most basic signals are cominng out right. For ascii '0' the signal that is being sent is 10000011001, for ascii '1' it is 10100011001. So those are almost what I would expect, except that there seem to be 2 startbits. Normally I would expect there to be only 1 startbit. Is there a way to change the amount of startbits sent?
Here are the outputs of the Oszilloscope:
this is a communication problem:
1 check the protocol-based communications parameters like baud rate, parity, start-/stop-bits they have to be consistent
(if you use UART protocol on RS-485 other protocols like MODBUS, Profibus,... are also possible, this is a difference to normal RS-232)
If the gauge uses 9600 baud for communication you can not use 115200 baud in your command. In the nodejs code you do not set any parameter (i assume you use the UART protocol because of your nodejs). If the gauge uses any other protocol the nodejs code will also not work, despite that there are not set any parameters like baud rate, parity,... in the code
https://en.wikipedia.org/wiki/RS-485
for other protocols node js serial module can not be used
http://libmodbus.org/
http://www.pbmaster.org/
2 check the proprietary commands you send to the gauge. When i want to read out the data of my multimeter i have to send an ASCII 'D' = 0100 0100 (bin) to get an answer (endianness ?) If i send any other value the multimeter stays silent.
http://electronicdesign.com/what-s-difference-between/what-s-difference-between-rs-232-and-rs-485-serial-interfaces
Unless you have DE pulled high and R︦E︦ tied to ground, your conversation will be rather one-sided.
And if you do wire as above, you need to be able to deal with your own echo in the received data.
I am pretty new to Node.Js and I'm using tcp sockets to communicate with a client. Since the received data is fragmented I noticed that it prints "ondata" to the console more than once. I need to be able to read all the data and concatenate it in order to implement the other functions. I read the following http://blog.nodejs.org/2012/12/20/streams2/ and thought I can use socket.on('end',...) for this purpose. But it never prints "end" to the console.
Here is my code:
Client.prototype.send = function send(req, cb) {
var self = this;
var buffer = protocol.encodeRequest(req);
var header = new Buffer(16);
var packet = Buffer.concat([ header, buffer ], 16 + buffer.length);
function cleanup() {
self.socket.removeListener('data', ondata);
self.socket.removeListener('error', onerror);
}
var body = '';
function ondata() {
var chunk = this.read() || '';
body += chunk;
console.log('ondata');
}
self.socket.on('readable', ondata);
self.socket.on('end', function() {
console.log('end');
});
function onerror(err) {
cleanup();
cb(err);
}
self.socket.on('error', onerror);
self.socket.write(packet);
};
The end event will handle the FIN package of the TCP protocol (in other words: will handle the close package)
Event: 'end'#
Emitted when the other end of the socket sends a FIN packet.
By default (allowHalfOpen == false) the socket will destroy its file descriptor once it has written out its pending write queue. However, by setting allowHalfOpen == true the socket will not automatically end() its side allowing the user to write arbitrary amounts of data, with the caveat that the user is required to end() their side now.
About FIN package: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Connection_termination
The solution
I understand your problem, the network communication have some data transfer gaps and it split your message in some packages. You just want read your fully content.
For solve this problem i will recommend you create a protocol. Just send a number with the size of your message before and while the size of your concatenated message was less than total of your message size, keep concatenating :)
I have created a lib yesterday to simplify that issue: https://www.npmjs.com/package/node-easysocket
I hope it helps :)