Python3 paho-mqtt speed vs nodejs mqtt - node.js

Question

I have conducted some speed tests for MQTT in Python3 and Node.js, using af QoS level 0 and have found Node.js to be remarkably faster than the Python3 implementation.
How can this be?
I'm open to using either framework as bridge on the server side to handle data from multiple clients. However, I'm losing my confidence that I should be using Python3 for anything on the server.
Running code snippets.
Python3:
import paho.mqtt.client as mqtt
import logging
import time
import threading
import json
import sys
class MqttAdaptor(threading.Thread):
def __init__(self,topic, type=None):
threading.Thread.__init__(self)
self.topic = topic
self.client = None
self.type = type
def run(self):
self.client = mqtt.Client(self.type)
self.client.on_connect = self.on_connect
self.client.on_disconnect = self.on_disconnect
if self.type is not None:
self.client.connect("localhost", 1883, 60)
self.client.on_message = self.on_message
self.client.loop_forever()
else:
self.client.connect_async("localhost", 1883, 60)
self.client.loop_start()
# The callback for when the client receives a CONNACK response from the server.
def on_connect(self,client, userdata, flags, rc):
self.client.subscribe(self.topic)
def on_disconnect(self, client, userdata, rc):
if rc != 0:
print("Unexpected disconnection from local MQTT broker")
# The callback for when a PUBLISH message is received from the server.
def on_message(self,client, userdata, msg):
jsonMsg = ""
try:
jsonMsg = json.loads(msg.payload)
if jsonMsg['rssi'] is not None:
jsonMsg['rssi'] = round(jsonMsg['rssi']*3.3 * 100000)/ 10000
except:
pass
print(json.dumps(jsonMsg))
def publish(self,topic, payload, qos=0,retain=False):
self.client.publish(topic,payload,qos,retain)
def close(self):
if self.client is not None:
self.client.loop_stop()
self.client.disconnect()
if __name__=="__main__":
topic = '/test/+/input/+'
subber = MqttAdaptor(topic,'sub')
subber.start()
topic = None
test = MqttAdaptor(topic)
test.run()
print("start")
while True:
data = sys.stdin.readline()
if not len(data):
print("BREAK")
break
msg = data.split('\t')
topic = msg[0]
test.publish(topic,msg[1],0)
print("done")
sys.exit(0)
Node.js:
"use strict";
const fs = require('fs');
const readline = require('readline');
const mqtt = require('mqtt');
const mqttClient = mqtt.connect();
mqttClient.on('connect', () => {
console.error('==== MQTT connected ====');
mqttClient.subscribe('/test/+/input/+');
});
mqttClient.on('close', () => {
console.error('==== MQTT closed ====');
});
mqttClient.on('error', (error) => {
console.error('==== MQTT error ' + error + ' ====');
});
mqttClient.on('offline', () => {
console.error('==== MQTT offline ====');
});
mqttClient.on('reconnect', () => {
console.error('==== MQTT reconnect ====');
});
mqttClient.on('message', (topic, message) => {
const topicSegments = topic.split('/');
topicSegments[topicSegments.length - 2] = 'done';
topic = topicSegments.join('/');
try {
//The message might not always be valid JSON
const json = JSON.parse(message);
//If rssi is null/undefined in input, it should be left untouched
if (json.rssi !== undefined && json.rssi !== null) {
//Multiply by 3 and limit the number of digits after comma to four
json.rssi = Math.round(json.rssi * 3.3 * 10000) / 10000;
}
console.log(topic + "\t" + JSON.stringify(json));
} catch (ex) {
console.error('Error: ' + ex.message);
}
});
const rl = readline.createInterface({
input: process.stdin,
terminal: false,
});
rl.on('line', (line) => {
const lineSegments = line.split("\t");
if (lineSegments.length >= 2) {
const topic = lineSegments[0];
const message = lineSegments[1];
mqttClient.publish(topic, message);
}
});
rl.on('error', () => {
console.error('==== STDIN error ====');
process.exit(0);
});
rl.on('pause', () => {
console.error('==== STDIN paused ====');
process.exit(0);
});
rl.on('close', () => {
console.error('==== STDIN closed ====');
process.exit(0);
});
Both script are run on the command line connecting to the same broker.
They are run using a scripting pipe (node):
time cat test-performance.txt | pv -l -L 20k -q | nodejs index.js | pv -l | wc -l
and (python):
time cat test-performance.txt | pv -l -L 20k -q | python3 mqttTestThread.py | pv -l | wc -l
The test file contains around 2Gb of text in this format:
/test/meny/input/test {"sensor":"A1","data1":"176","time":1534512473545}
As shown in the scripts, I count the number of lines during the time they run. For a small test the Python3 script has a throughput of roughly 3k/sec, while node has a throughput og roughly 20k/sec.
This is a big difference. Does anyone have an idea why? And/or how to get python to run with a comparable throughput?

Answer 1

There are multiple reasons why Node is faster for this task than Python. The main reason is: Python is slooooow. Only the libraries which are implemented in C like numpy or pandas are somewhat fast. But then also just for numeric tasks.
The second reason is, as Nhosko mentioned in a comment, that Node is per default async and therefore faster in I/O bound tasks.
A potential third reason could be that MQTT sends JSON data. JSON stands for Java-Script-Object-Notation and can be natively converted into NodeJS objects.
I wouldn't recommend you to use Python for this task. Python is great for machine learning and data science. For server and I/O bound tasks you may consider using Node or Go.