I am programming a Node JS client that's querying a third-party server for information. Part of the message that I receive back from the third party server is a UINT 128 (specified in documentation). However, from what I can tell, NodeJS buffers do not support reading 128 bit ints from buffers. I have checked this page and the two most likely functions (reading BIGINTS and reading unspecified byte length ints) both don't support 128 bit integers.
I've already tried installing and using the big-integer module, but if that is the correct method, I couldn't understand how to read the bytes out of the buffer into a big-integer object.
How can I read a UINT128 out of a NodeJS buffer? I'm' willing to use bit arithmetic if necessary, I'm just not sure how.
I did it by using getBigUint64 on a DataView twice, then calling toString on both, concatenating the strings and parsing that string as a BigInt... it's dumb but it seems to work. You may also be able to concat the 64bit BigInts using bitwise operators but I haven't tried.
readUInt128() {
let a = this.readUInt64();
let b = this.readUInt64();
return BigInt(a.toString() + b.toString());
}
readUInt64() {
let result = this.view.getBigUint64(this.pos, this.littleEndian)
this.pos += 8
return result
}
Related
Background
I am reading buffers using the Node.js buffer native API. This API has two functions called readUIntBE and readUIntLE for Big Endian and Little Endian respectively.
https://nodejs.org/api/buffer.html#buffer_buf_readuintbe_offset_bytelength_noassert
Problem
By reading the docs, I stumbled upon the following lines:
byteLength Number of bytes to read. Must satisfy: 0 < byteLength <= 6.
If I understand correctly, this means that I can only read 6 bytes at a time using this function, which makes it useless for my use case, as I need to read a timestamp comprised of 8 bytes.
Questions
Is this a documentation typo?
If not, what is the reason for such an arbitrary limitation?
How do I read 8 bytes in a row ( or how do I read sequences greater than 6 bytes? )
Answer
After asking in the official Node.js repo, I got the following response from one of the members:
No it is not a typo
The byteLength corresponds to e.g. 8bit, 16bit, 24bit, 32bit, 40bit and 48bit. More is not possible since JS numbers are only safe up to Number.MAX_SAFE_INTEGER.
If you want to read 8 bytes, you can read multiple entries by adding the offset.
Source: https://github.com/nodejs/node/issues/20249#issuecomment-383899009
I have a node.js application that contains the following piece of code:
const crypto = require('crypto');
const randomBytes=crypto.randomBytes(15);
Now what I want to do is to read a single each byte from 15 bytes because I want to perform a low level operation such as sending via USB in a manner that this answer explains.
But how I can do that?
The crypto.randomBytes return a buffer in order to achieve that you should put read the data by using these methods:
randomBytesInABuffer.readUInt8(offset);
randomBytesInABuffer.readInt8(offset)
Keep in mind that in lower-level languages and specifically C and C++ the variable types char and unsigned char also have 1 byte length and can be used as 1byte integers too.
I've been working on a server that expects data to be received through a buffer. I have an object which is defined like this and some procedures that modify the buffer in it:
Packet* = ref object
buf*: seq[int8]
#buf*: array[0..4096, int8]
pos*: int
proc newPacket*(size: int): Packet =
result = Packet(buf: newSeq[int8](size))
#result = Packet()
proc sendPacket*(s: AsyncSocket, p: Packet) =
aSyncCheck s.send(addr(p.buf), p.pos)
Now the reason I have two lines commented is because that was the code I originally used, but creating an object that initialises an array with 4096 elements every time probably wasn't very good for performance. However, it works and the seq[int8] version does not.
The strange thing is though, my current code will work perfectly fine if I use the old static buffer buf*: array[0..4096, int8]. In sendPacket, I have made sure to check the data contained in the buffer to make sure both the array and seq[int8] versions are equal, and they are. (Or at least appear to be). In other words, if I were to do var p = createPacket(17) and write to p.buf with exactly 17 bytes, the values of the elements appear to be the same in both versions.
So despite the data appearing to be the same in both versions, I get a different result when calling send when passing the address of the buffer.
In case it matters, the data would be read like this:
result = p.buf[p.pos]
inc(p.pos)
And written to like this:
p.buf[p.pos] = cast[int8](value)
inc(p.pos)
Just a few things I've looked into, which were probably unrelated to my problem anyway: I looked at GC_ref and GC_unref which had no effect on my problem and also looked at maybe trying to use alloc0 where buf is defined as pointer but I couldn't seem to access the data of that pointer and that probably isn't what I should be doing in the first place. Also if I do var data = p.buf and pass the addr of data instead, I get a different result, but still not the intended one.
So I guess what I want to get to the bottom of is:
Why does send work perfectly fine when I use array[0..4096, int8] but not seq[int8] which is initialised with newSeq, even when they appear to contain the same data?
Does my current layout for receiving and writing data even make sense in a language like Nim (or any language for that matter)? Is there a better way?
In order not to initialize the array you can use the noinit pragma like this:
buf* {.noinit.}: array[0..4096, int8]
You are probably taking the pointer to the seq, not the pointer to the data inside the seq, so try using addr(p.buf[0]).
A pos field is useless if you are using the seq version since you have p.buf.len already, but you probably know that already and just left it in for the array. If you want to use the seq and expect large packets, make sure to use newSeqOfCap to only allocate the memory once.
Also, your array is 1 byte too big, it goes from 0 to 4096 inclusively! Instead you can use [0..4095, int8] or just [4096, int8].
Personally I would prefer to use a uint8 type inside of buf, so that you can just put in values from 0 to 255 instead of -128 to 127
Using a seq inside of a ref object means you have two layers of indirection when accessing buf, as well as two objects that the GC will have to clean up. You could just make Packet an alias for seq[uint8] (without ref): type Packet* = seq[uint8]. Or you can use the array version if you want to store some more data inside the Packet later on.
I am trying to convert a Ruby program to NodeJS, but I seem to be getting stuck with buffers.
I have
rounds = header_bytes[120..-1].unpack('L*').first
In Ruby, which headers a buffer (header_bytes), and get's 120-124 (or in this case -1, which is remaining). Then unpacks it into an unsigned 32 bit integer.
I am trying to do the same thing in JS, but it can't seem to get it to work. I have
rounds = header.slice(120,124).toString('ucs2');
I've tried all the different formats in toString and nothing returns the same result as Ruby.
Assuming that header is an instance of Node's Buffer then you have a variety of functions for reading from a buffer as various sizes of integer, including
buf.readUInt32LE
buf.readUInt32BE
These both take an offset from which to read the bytes. The ruby L specifier means native byte order so depending on where this code is running you might need either of those functions, depending on whether you're on a big or little endian platform. For example on an x86 machine you'd do
header.readUInt32LE(120)
Protocols normally specify big or little endian (traditionally network byte order is big endian)
You can check the platform endianness with os.endianness
I have a PHP class for reading Binary data that I'm converting to NodeJS or finding the equivalent of a couple functions in NodeJS. The functions I'm interested in this BinaryReader class are ReadULong and ReadUShort. I believe these mean read Unsigned Long integer (4 bytes) and Unsigned Short integer (2 bytes). As I'm trying to find the equivalent for these in NodeJS, I get confused on which function to use between these:
buf.readUInt16LE(offset, [noAssert])
buf.readUInt16BE(offset, [noAssert])
buf.readUInt32LE(offset, [noAssert])
buf.readUInt32BE(offset, [noAssert])
What would LE or BE stand for in this case?
The Buffer docs are located here but I was unable to find an explanation for those here.
Also I've found a constant on the PHP class that says const DEFAULT_BYTE_ORDER = 'L';. Is this L same as that L in readUInt32LE? Is this whole thing about Byte Orders?
So far I've read these articles:
Good Source at cplusplus.com for looking up variable types.
PHP bytewise tutorial and binary math
How to read binary files byte by byte in Node.js question at stackoverflow
If I could be given a couple more references to read about binary reading that would be much appreciated!
BE and LE stand for big endian and little endian. In big endian, the most significant byte is stored in the smallest address, and in little endian, the least significant byte is stored in the smallest address. That being said, endian does indicate the byte order. You can see the pattern in one of the examples in the documentation:
var buf = new Buffer(2);
buf[0] = 0x3;
buf[1] = 0x4;
buf.readUInt16BE(0);
buf.readUInt16LE(0);
// 0x0304
// 0x0403