First time posting here so sorry if the question is too vague or missing info. Anyway:
I'm trying to send OSC (open sound control) or MQTT messages when I hit specific notes on my midi controller. When I try to send a OSC message when I get a midi input, I get this unpredictable high latency, varying from about 200-300 milliseconds up to 10 seconds. Here is my test-code:
var easymidi = require('easymidi');
var osc = require("osc");
easymidi.getInputs().forEach((midiDevice)=> {
if (midiDevice.includes('Maschine')){
inputMidi = midiDevice
console.log(`Midi device input found: ${inputMidi}`)
}
})
var inputMikro = new easymidi.Input(inputMidi)
var rmeOSC = new osc.UDPPort ({
remoteAddress: "192.168.10.148",
remotePort: 9001
});
rmeOSC.open();
inputMikro.on('cc', function (msg) {
if (msg.controller == 7){
// This is instant
console.log(msg)
// This is what becomes unpredictable and slow
rmeOSC.send({
address: "/1/mastervolume",
args: [
{
type: "f",
value: msg.value / 127
}
]
});
}
}
I've tried to zero in what makes it slow. When just logging to console inside inputMikro.on('cc', ... ) event listener, it's instant. If I send that midi data to another midi device in that same event listener (also using easymidi library), it's also instant. But sending OSC or MQTT messages there creates that unpredictable latency. I've also tried setting an interval every 20 to send OSC just to make sure it's not a limitation of the OSC library in itself:
var vol = 0
var sendOsc = (()=>{
vol++
if (vol == 127){
vol = 0
}
rmeOSC.send({
address: "/1/mastervolume",
args: [
{
type: "f",
value: vol / 127
}
]
});
})
setInterval(sendOsc, 20)
That works great, same with MQTT. I also tried creating a separate volume variable with an event listener that listens for changes in that variable, and update that variable inside the inputMikro.on('cc', ... ) listener. Same there, if i just log to console whenever that variable changes, it's instant, but if I try to send OSC or MQTT messages when the variable changes, I get that latency.
I'm out of ideas and have no idea what's going on. I'd very much appreciate any insight on how I can fix this. I hope my question is clear enough.
Related
I'm using socket.io like this
Client:
socket.on('response', function(i){
console.log(i);
});
socket.emit('request', whateverdata);
Server:
socket.on('request', function(whateverdata){
for (i=0; i<10000; i++){
console.log(i);
socket.emit('response', i);
}
console.log("done!");
});
I need output like this when putting the two terminals side by side:
Server Client
0 0
1 1
. (etc) .
. .
9998 9998
9999 9999
done!
But instead I am getting this:
Server Client
0
1
. (etc)
.
9998
9999
done!
0
1
.
. (etc)
9998
9999
Why?
Shouldn't Socket.IO / Node emit the message immediately, not wait for the loop to complete before emitting any of them?
Notes:
The for loop is very long and computationally slow.
This question is referring to the socket.io library, not websockets in general.
Due to latency, waiting for confirmation from the client before sending each response is not possible
The order that the messages are received is not important, only that they are received as quickly as possible
The server emits them all in a loop and it takes a small bit of time for them to get to the client and get processed by the client in another process. This should not be surprising.
It is also possible that the single-threaded nature of Javascript in node.js prevents the emits from actually getting sent until your Javascript loop finishes. That would take detailed examination of socket.io code to know for sure if that is an issue. As I said before if you want to 1,1 then 2,2 then 3,3 instead of 1,2,3 sent, then 1,2,3 received you have to write code to force that.
If you want the client to receive the first before the server sends the 2nd, then you have to make the client send a response to the first and have the server not send the 2nd until it receives the response from the first. This is all async networking. You don't control the order of events in different processes unless you write specific code to force a particular sequence.
Also, how do you have client and server in the same console anyway? Unless you are writing out precise timestamps, you wouldn't be able to tell exactly what event came before the other in two separate processes.
One thing you could try is to send 10, then do a setTimeout(fn, 1) to send the next 10 and so on. That would give JS a chance to breathe and perhaps process some other events that are waiting for you to finish to allow the packets to get sent.
There's another networking issue too. By default TCP tries to batch up your sends (at the lowest TCP level). Each time you send, it sets a short timer and doesn't actually send until that timer fires. If more data arrives before the timer fires, it just adds that data to the "pending" packet and sets the timer again. This is referred to as the Nagle's algorithm. You can disable this "feature" on a per-socket basis with socket.setNoDelay(). You have to call that on the actual TCP socket.
I am seeing some discussion that Nagle's algorithm may already be turned off for socket.io (by default). Not sure yet.
In stepping through the process of socket.io's .emit(), there are some cases where the socket is marked as not yet writable. In those cases, the packets are added to a buffer and will be processed "later" on some future tick of the event loop. I cannot see exactly what puts the socket temporarily in this state, but I've definitely seen it happen in the debugger. When it's that way, a tight loop of .emit() will just buffer and won't send until you let other events in the event loop process. This is why doing setTimeout(fn, 0) every so often to keep sending will then let the prior packets process. There's some other event that needs to get processed before socket.io makes the socket writable again.
The issue occurs in the flush() method in engine.io (the transport layer for socket.io). Here's the code for .flush():
Socket.prototype.flush = function () {
if ('closed' !== this.readyState &&
this.transport.writable &&
this.writeBuffer.length) {
debug('flushing buffer to transport');
this.emit('flush', this.writeBuffer);
this.server.emit('flush', this, this.writeBuffer);
var wbuf = this.writeBuffer;
this.writeBuffer = [];
if (!this.transport.supportsFraming) {
this.sentCallbackFn.push(this.packetsFn);
} else {
this.sentCallbackFn.push.apply(this.sentCallbackFn, this.packetsFn);
}
this.packetsFn = [];
this.transport.send(wbuf);
this.emit('drain');
this.server.emit('drain', this);
}
};
What happens sometimes is that this.transport.writable is false. And, when that happens, it does not send the data yet. It will be sent on some future tick of the event loop.
From what I can tell, it looks like the issue may be here in the WebSocket code:
WebSocket.prototype.send = function (packets) {
var self = this;
for (var i = 0; i < packets.length; i++) {
var packet = packets[i];
parser.encodePacket(packet, self.supportsBinary, send);
}
function send (data) {
debug('writing "%s"', data);
// always creates a new object since ws modifies it
var opts = {};
if (packet.options) {
opts.compress = packet.options.compress;
}
if (self.perMessageDeflate) {
var len = 'string' === typeof data ? Buffer.byteLength(data) : data.length;
if (len < self.perMessageDeflate.threshold) {
opts.compress = false;
}
}
self.writable = false;
self.socket.send(data, opts, onEnd);
}
function onEnd (err) {
if (err) return self.onError('write error', err.stack);
self.writable = true;
self.emit('drain');
}
};
Where you can see that the .writable property is set to false when some data is sent until it gets confirmation that the data has been written. So, when rapidly sending data in a loop, it may not be letting the event come through that signals that the data has been successfully sent. When you do a setTimeout() to let some things in the event loop get processed that confirmation event comes through and the .writable property gets set to true again so data can again be sent immediately.
To be honest, socket.io is built of so many abstract layers across dozens of modules that it's very difficult code to debug or analyze on GitHub so it's hard to be sure of the exact explanation. I did definitely see the .writable flag as false in the debugger which did cause a delay so this seems like a plausible explanation to me. I hope this helps.
I'm using the code below to try send OSC messages to a computer on the network. I'm using a package called osc.
I'm unable to send messages to the machine running the OSC server and receive the error below when attempting to send OSC messages:
Error: Uncaught, unspecified "error" event. (Can't send packets on a closed osc.Port object. Please open (or reopen) this Port by calling open().)
Code
let osc = require('osc');
let oscUDP = new osc.UDPPort({
remoteAddress: "192.168.1.5",
remotePort: 8004
});
oscUDP.send({
address: "/carrier/frequency",
args: 440
});
oscUDP.open();
If I put oscUDP.open() before the send call I get a different error:
Error: send EINVAL 192.168.1.5:8004
at Object.exports._errnoException (util.js:1007:11)
at exports._exceptionWithHostPort (util.js:1030:20)
at SendWrap.afterSend [as oncomplete] (dgram.js:402:11)
I am running OSCulator on OSX as the server. The code above lives on a different machine. When I run nmap on the IP address the port is open:
nmap 192.168.1.5 -p 8004
Starting Nmap 6.40 ( http://nmap.org ) at 2016-08-30 08:22 BST
Nmap scan report for 192.168.1.5
Host is up (0.13s latency).
PORT STATE SERVICE
8004/tcp open unknown
If I use osc-cli the messages are received on the machine running the OSC server:
osc --host 192.168.1.5:8004 /test 1 2 3
So it would seem the problem isn't with closed ports at all as the messages are sent and received when using osc-cli.
Any ideas?
I know I'm coming to this quite late, and it looks like you found a different library that works for you, but I thought a response might be helpful for others who are facing this issue. I'm the developer of osc.js, the original library you were trying to use.
First off, as background information, osc.js is factored into two different layers:
The low-level API that provides functions for reading and writing OSC messages and bundles to/from Typed Arrays.
The higher-level, event-based Port API, which provides a collection of platform-specific transport objects, which offer an easy way to do bidirectional communication over protocols like UDP, Web Sockets, etc.
In the case of your example code, you were trying to send an OSC message on your UDPPort object prior to it being ready. When you open() a Port, it may need to perform asynchronous operations such as opening up a socket, etc. As a result, it fires an event (aptly called ready) when the Port is all set to be used. Until ready fires, you won't be able to send or receive OSC packets.
So in the case of your original code, it looks like you were assuming that this line was synchronous and that you could call send() immediately afterwards:
oscUDP.open();
Instead, you just needed to listen for the ready event prior to attempting to send a message on the Port. Like this:
oscUDP.on("ready", function () {
oscUDP.send({
address: "/carrier/frequency",
args: 440
});
});
The osc.js Node.js example illustrates this pattern. But when I saw your question, I realized that the sample code in the osc.js README was a bit ambiguous in this regard. I have improved the event documentation and the inline README sample code to be more clear in this regard. Sorry for the confusion.
There are cases, perhaps such as yours, where the higher-level API isn't quite what you need. osc.js also provides functions for easily encoding an OSC packet as a Uint8Array, which can be converted into a Node.js buffers. So you could have done something similar to your solution just by using osc.js' osc.writeMessage() function. It has always been quite well documented, fortunately. Here's your example, modified to use osc.js' low-level API:
const dgram = require('dgram');
const client = dgram.createSocket('udp4');
const osc = require('osc');
const HOST = '192.168.1.5';
const PORT = 8004;
process.on('SIGINT', function() {
client.close();
});
let oscNoteMessage = function(note, value) {
var message = osc.writeMessage({
address: '/note/' + note,
args: [
{
type: 'i',
value: value
}
]
});
return Buffer.from(message);
}
let noteOn = function(note) {
return oscNoteMessage(note, 1);
}
let noteOff = function(note) {
return oscNoteMessage(note, 0);
}
let send = function(message) {
client.send(message, PORT, HOST, function(err, bytes) {
if(err) throw new Error(err);
})
}
send(noteOn('c'));
setTimeout(function() {
send(noteOff('c'));
}, 1000);
Anyway, I'm glad you were able to come up with a solution that works for your project, and I hope this response helps other users who may encounter similar issues. And of course, feel free to ask questions or file issues on the osc.js issue tracker.
Best regards, and apologies for the trouble you experienced using the library!
I figured it's actually pretty easy to send OSC data over UDP without the need for any packages except a2r-osc which is used for encoding OSC data.
I'm posting the solution incase anyone else in interested:
const dgram = require('dgram');
const client = dgram.createSocket('udp4');
const osc = require('a2r-osc');
const HOST = '192.168.1.5';
const PORT = 8004;
process.on('SIGINT', function() {
client.close();
});
let noteOn = function(note) {
return new osc.Message('/note/' + note, 'i', 1).toBuffer();
}
let noteOff = function(note) {
return new osc.Message('/note/' + note, 'i', 0).toBuffer();
}
let send = function(message) {
client.send(message, PORT, HOST, function(err, bytes) {
if(err) throw new Error(err);
})
}
send(noteOn('c'));
setTimeout(function() {
send(noteOff('c'));
}, 1000);
I am trying to receive and send data from a vacuum gauge (previous Model of https://www.pfeiffer-vacuum.com/en/products/measurement/digiline/gauges/?detailPdoId=13238&request_locale=en_US) with a computer (Linux 16.04) via an USB-to-RS485-Interface (the half-duplex USB485-STISO from http://www.hjelmslund.dk/). When I send a request to the gauge using a specific protocol it is supposed to answer to the request and I should be able to receive it with the interface. I managed to send data but whenever I send data, it seems that nothing comes back. I'm trying to do this with Node.js. The Code that I used so far is:
function pack(address, action, parameter, data) {
var length = String('00' + data.length.toString()).slice(-2);
var bufferAsString = address + action + parameter + length + data;
var check = 0;
for (var i = 0; i < bufferAsString.length; ++i) {
check += bufferAsString.charCodeAt(i)
}
var checkSum = String('000' + String(check % 256)).slice(-3);
var buffer = Buffer.from(bufferAsString + checkSum),
carriageReturn = Buffer.from('\r');
return Buffer.concat([buffer, carriageReturn]);
}
var serialPort = require('serialport');
var SerialPort = serialPort.SerialPort;
var port = new SerialPort('/dev/ttyUSB0', {
baudrate: 9600,
dataBits: 8,
stopBits: 1,
parity: 'none'
}, false);
port.open(function(err) {
if (err) {
return console.log('Error opening port: ', err.message);
}
console.log(port.isOpen());
port.on('data', function (data) {
console.log('Data: ' + data);
});
port.on('close', function() {
console.log('port closed')
});
var sendBuffer = pack('001', '00', '740', '=?');
setInterval(function() {
port.write(sendBuffer, function(err, bytes) {
console.log('send' + bytes)
});
port.drain();
}, 1000)
});
That is supposed to send a request every second to the gauge to measure the pressure. I know that the request is being send since the TxD-Led blinks shortly every second. But I receive no answer to that request.
I also tried other methods of sending data (mostly via python and the terminal) but with similar success. The green lamp for sending always flashes up but then nothing happens and no answer is received.
I am at a loss as to what to try next and would really appreciate any help that you could give me.
UPDATE:
Ok so I seem to have found one possible error in the whole thing. I was working with an oszilloscope to capture the signal that is going out of the interface when I send something. I started with single ascii-characters to see if the most basic signals are cominng out right. For ascii '0' the signal that is being sent is 10000011001, for ascii '1' it is 10100011001. So those are almost what I would expect, except that there seem to be 2 startbits. Normally I would expect there to be only 1 startbit. Is there a way to change the amount of startbits sent?
Here are the outputs of the Oszilloscope:
this is a communication problem:
1 check the protocol-based communications parameters like baud rate, parity, start-/stop-bits they have to be consistent
(if you use UART protocol on RS-485 other protocols like MODBUS, Profibus,... are also possible, this is a difference to normal RS-232)
If the gauge uses 9600 baud for communication you can not use 115200 baud in your command. In the nodejs code you do not set any parameter (i assume you use the UART protocol because of your nodejs). If the gauge uses any other protocol the nodejs code will also not work, despite that there are not set any parameters like baud rate, parity,... in the code
https://en.wikipedia.org/wiki/RS-485
for other protocols node js serial module can not be used
http://libmodbus.org/
http://www.pbmaster.org/
2 check the proprietary commands you send to the gauge. When i want to read out the data of my multimeter i have to send an ASCII 'D' = 0100 0100 (bin) to get an answer (endianness ?) If i send any other value the multimeter stays silent.
http://electronicdesign.com/what-s-difference-between/what-s-difference-between-rs-232-and-rs-485-serial-interfaces
Unless you have DE pulled high and R︦E︦ tied to ground, your conversation will be rather one-sided.
And if you do wire as above, you need to be able to deal with your own echo in the received data.
I can't make following code to work:
"use strict";
let kafka = require('kafka-node');
var conf = require('./providers/Config');
let client = new kafka.Client(conf.kafka.connectionString, conf.kafka.clientName);
let consumer = new kafka.HighLevelConsumer(client, [ { topic: conf.kafka.readTopic } ], { groupId: conf.kafka.clientName, paused: true });
let threads = 0;
consumer.on('message', function(message) {
threads++;
if (threads > 10) consumer.pause();
if (threads > 50) process.exit(1);
console.log(threads + " >>> " + message.value);
});
consumer.resume();
I see 50 messages in console and process exits by termination statement.
What I'm trying to understand, is that is it my code broken or package broken? Or maybe I'm just doing something wrong? Does anyone was able to make kafka consumer work with pause/resume? I tried several versions of kafka-node, but all of them behave same way. Thanks!
You are already using pause and resume in your code, so obviously they work. ;)
It's because pause doesn't pause the consumption of messages. It pauses the fetching of messages. I'm guessing you already fetched the first 50 in one throw before you receive the first message and call pause.
For kicks, I just tested pause() and resume() in the Node REPL and they work as expected:
var kafka = require('kafka-node');
var client = new kafka.Client('localhost:2181');
var consumer = new kafka.HighLevelConsumer(client, [{topic: 'MyTest'}]);
consumer.on('message', (msg) => { console.log(JSON.stringify(msg)) });
Then I go into another window and run:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic MyTest
And type some stuff, and it shows up in the first window. Then in the first window, I type: consumer.pause(); And type some more in the second window. Nothing appears in the first window. Then I run consumer.resume()in the first window, and the delayed messages appear.
BTW, you should be able to play with the Kafka config property fetch.message.max.bytes and control how many messages can be fetched at one time. For example, if you had fixed-width messages of 500 bytes, set fetch.message.max.bytes to something less than 1000 (but greater than 500!) to only receive a single message per fetch. But note that this might not fix the problem entirely -- I am fairly new to Node, but it is asynchronous, and I suspect a second fetch could get kicked off before you processed the first fetch completely (or at all).
I am creating a node application that will send UDP packets to another machine. Everything appears to work fine when packets are sent about 100/second. When that number gets higher 500/second the packets stop sending. I don't think they are being droped as Wireshark does not show them leaving my computer.
const dgram = require("dgram");
var sent = 0;
var server = dgram.createSocket("udp4");
function sendCommand(text){
// Put command into a buffer
var buffer = new Buffer(text);
// Send the command
server.send(buffer, 0, buffer.length, 12345, 19.5.6.45, (err) => {
sent++;
});
}
The sendCommand() function is what is called internally. Like I said at 100 calls per second no problem. When it gets higher it just stops working.
The "sent" var stops incrementing so this leads me to think node dgram is faulty and server.send() is not firing correctly. It doesn't seem to recover unless I restart the program.
This is my first time posting on Stack Overflow, I apologise if I did anything wrong.
Edit:
Here is where the function is firing from.
// Loop through the channels
for (var i = 0; i < ids.length; i++) {
// Check if the channel is locked
if (!toBool(data.locks[i])) {
// Send the UDP command
sendCommand(ids[i]);
}
}