I am using the node.js mosca MQTT broker for some internet of things (iot) application.
https://github.com/mcollina/mosca
What is the maximum message length that a topic can receive for the mosca broker? What are the factors that constrain the message length?
If I want to increase the message length, is there a configuration parameter I can modify or which part of the code can I change?
It's not entirely clear what you're asking here, so I'll answer both possibilities.
The length of the actual topic string is at most 65536 bytes. This is a limit imposed by the mqtt spec, you can't change it. It is also worth noting that the topic is encoded with utf-8, so you may have less than 65536 characters available.
The payload of the message is limited to 268,435,456 bytes. Again, this is defined by the spec.
If you are routinely approaching either of these limits you should be thinking about whether what you are doing is sensible.
Related
I am currently working on a graduation project where I want to transmit a sessiontoken using BLE. On the server side I am using Node.js and Bleno to create the connection. After the client subscribes to the notification, the server will push the token.
A small part of the code is:
const buf1 = Buffer.from(info, 'utf8');
updateValueCallback(buf1);
At this step, I am using nRF Connect to check if everything is working. My intention works, except I see that only the first 20 characters are transferred. (As much as the packet size)
My question concerns the buffer size. Will, when I finally connect to an Android app, the whole string be transmitted? In this case the underlying protocols will cut the string and reassemble it on the other side. In this case the buffer size doesn't matter. Or must I negotiate the MTU to be the size of the string. In other words must the buffersize be the size of the transmitted package?
In the case the buffer is smaller than the whole string, can the whole string still be transmitted with it?
GATT requires that a notification is maximum MTU - 3 bytes long. The default MTU is 23 so hence the maximum modification value length is 20 bytes by default. By negotiating a larger MTU you can send longer notifications (if your BLE stack supports that).
I haven't used Bleno but all the stack that I have used I needed to slice the data myself 20 bytes at the time. And on receiver side collect them and put them together again.
The stacks have been good to buffer the data and transmit it one chunk at the time. So I have looped the function (as your updateValueCallback()) until all the slices of my data was done.
Hope it works for you.
I need to transfer data over a serial port. In order to ensure integrity of the data, I want a small envelope protocol around each protobuf message. I thought about the following:
message type (1 byte)
message size (2 bytes)
protobuf message (N bytes)
(checksum; optional)
The message type will mostly be a mapping between messages defined in proto files. However, if a message gets corrupted or some bytes are lost, the message size will not be correct and all subsequent bytes cannot be interpreted anymore. One way to solve this would be the introduction of limiters between messages, but for that I need to choose something that is not used by protobuf. Is there a byte sequence that is never used by any protobuf message?
I also thought about a different way. If the master finds out that packages are corrupted, it should reset the communication to a clean start. For that I want the master to send a RESTART command to the slave. The slave should answer with an ACK and then start sending complete messages again. All bytes received between RESTART and ACK are to be discarded by the master. I want to encode ACK and RESTART as special messages. But with that approach I face the same problem: I need to find byte sequences for ACK and RESTART that are not used by any protobuf messages.
Maybe I am also taking the wrong approach - feel free to suggest other approaches to deal with lost bytes.
Is there a byte sequence that is never used by any protobuf message?
No; it is a binary serializer and can contain arbitrary binary payloads (especially in the bytes type). You cannot use sentinel values. Length prefix is fine (your "message size" header), and a checksum may be a pragmatic option. Alternatively, you could impose an artificial sentinel to follow each message (maybe a guid chosen per-connection as part of the initial handshake), and use that to double-check that everything looks correct.
One way to help recover packet synchronization after a rare problem is to use synchronization words in the beginning of the message, and use the checksum to check for valid messages.
This means that you put a constant value, e.g. 0x12345678, before your message type field. Then if a message fails checksum check, you can recover by finding the next 0x12345678 in your data.
Even though that value could sometimes occur in the middle of the message, it doesn't matter much. The checksum check will very probably catch that there isn't a real message at that position, and you can search forwards until you find the next marker.
I need to create a monitor, which will log information about packet missing using ZeroMQ ipc. Actually I don't really understand everything about it because of there are some LINX, TIPS protocols also. Can you please explain me that and answer the main question?
You could make the application self-monitoring, by including a message serial number in each message structure. The message sender keeps track of the serial number it last sent, and increments it every time it sends a message.
The recipient should then be receiving messages with ever-increasing message serial numbers embedded. If that ever jumps by 2 or more, a message has gone missing.
IPC is not lossy like a network can be - the bytes put in come out the other end. TCP is not lossy either, provided both ends are still running and the network itself hasn't failed. However, depending on the ZMQ pattern used and how it's set up whole messages can be undelivered (for example, if the recipient hasn't connected yet, etc). If that's what you mean by "packet missing", it would be revealed by including an incrementing message serial number.
This question suggests that the Node.js Net module doesn't handle ip-fragmentation:
Node.js how to handle packet fragmentation with net.Server
I can almost not imagine it's true, but I can't find any documentation about this (please forgive me if it shouldn't be too hard to find information about this :-) ). Is it true?
If not: thank you, that's a real day-safer to me :-).
If it is true: how to handle this problem if I don't know how large the whole datagram is?
Situation:
I have a TCP connection with an embedded system (Wiznet W5500). The MSS (Maximum Segment Size, mostly equal to MTU - 40) will be set to 536 and data packages are a variable size and may be over 4kb in size. So the data package will be send in multiple segments. Will the 'on data' event be triggered once a segment is received or only when the whole package is received?
Side question: Am I right that the data segment (about what Wiznet is talking in the explanation of the MSS register) equel is to a ip-fragment?
So if I have to send 4000byte (ie payload) and the MSS is set to 536 I will receive consecutively:
segment1: 536bytes payload
segment2: 536bytes payload
segment3: 536bytes payload
segment4: 536bytes payload
segment5: 536bytes payload
segment6: 536bytes payload
segment7: 536bytes payload
segment8: 248bytes payload
May the 'on data' event only be triggered after segment 8 an will the 'data argument' contain the whole package or may it happen that the 'on data' event will be triggered after every separately received segment?
How can I make 100% sure that I get the whole package before I continue to process it?
Solution I thought of:
First 2 byte of the data package is the byte length of the whole length, I keep concattenating received data until I have received as many bytes. If I received more than the package size, I'll assume these successive bytes are the start of a subsequent data package.
I do believe this 'solution' is somewhat tricky and I hope it's not required.
Thanks in advance! If any information is missing: I'm sorry, please feel free to ask for it :-).
I am a big fan of nodeJS, but in this case you should use python with scapy :)
http://www.secdev.org/projects/scapy/
I am left with a few questions after reading the RFC 6520 for Heartbeat:
https://www.rfc-editor.org/rfc/rfc6520
Specifically, I don't understand why a heartbeat needs to include arbitrary payloads or even padding for that matter. From what I can understand, the purpose of the heartbeat is to verify that the other party is still paying attention at the other end of the line.
What does these variable length custom payloads provide that a fixed request and response do not?
E.g.
Alice: still alive?
Bob: still alive!
After all, FTP uses the NOOP command to keep connections alive, which seem to work fine.
There is, in fact, a reason for this payload/padding within RFC 6520
From the document:
The user can use the new HeartbeatRequest message,
which has to be answered by the peer with a HeartbeartResponse
immediately. To perform PMTU discovery, HeartbeatRequest messages
containing padding can be used as probe packets, as described in
[RFC4821].
>In particular, after a number of retransmissions without
receiving a corresponding HeartbeatResponse message having the
expected payload, the DTLS connection SHOULD be terminated.
>When a HeartbeatRequest message is received and sending a
HeartbeatResponse is not prohibited as described elsewhere in this
document, the receiver MUST send a corresponding HeartbeatResponse
message carrying an exact copy of the payload of the received
HeartbeatRequest.
If a received HeartbeatResponse message does not contain the expected
payload, the message MUST be discarded silently. If it does contain
the expected payload, the retransmission timer MUST be stopped.
Credit to pwg at HackerNews. There is a good and relevant discussion there as well.
(The following is not a direct answer, but is here to highlight related comments on another question about Heartbleed.)
There are arguments against the protocol design that allowed an arbitrary limit - either that there should have been no payload (or even echo/heartbeat feature) or that a small finite/fixed payload would have been a better design.
From the comments on the accepted answer in Is the heartbleed bug a manifestation of the classic buffer overflow exploit in C?
(R..) In regards to the last question, I would say any large echo request is malicious. It's consuming server resources (bandwidth, which costs money) to do something completely useless. There's really no valid reason for the heartbeat operation to support any length but zero
(Eric Lippert) Had the designers of the API believed that then they would not have allowed a buffer to be passed at all, so clearly they did not believe that. There must be some by-design reason to support the echo feature; why it was not a fixed-size 4 byte buffer, which seems adequate to me, I do not know.
(R..) .. Nobody thinking from a security standpoint would think that supporting arbitrary echo requests is reasonable. Even if it weren't for the heartbleed overflow issue, there may be cryptographic weaknesses related to having such control over the content the peer sends; this seems unlikely, but in the absence of a strong reason to support a[n echo] feature, a cryptographic system should not support it. It should be as simple as possible.
While I don't know the exact motivation behind this decision, it may have been motivated by the ICMP echo request packets used by the ping utility. In an ICMP echo request, an arbitrary payload of data can be attached to the packet, and the destination server will return exactly that payload if it is reachable and responding to ping requests. This can be used to verify that data is being properly sent across the network and that payloads aren't being corrupted in transit.