I have a large DAT file that holds custom byte information,
And I've been tasked with converting the solution to JavaScript.
It's to make the solution be more single-language and convert to serverless cloud computing.
But, I've run into an issue with converting this test data.
The values supposed to return a float,
But I can't seem to get the number converted correctly.
The sliced buffer output is <Buffer 40 82 e2 31 d6 d7 2e 8d>,
Which is supposed to return 604.274335557087
But actually returns 4.090111255645752.
And I'm at my wits end right now.
Any thoughts?
fs.readFile(file, (err, data) => {
...
// Read other buffer slice() values fine until this step.
// Like: readInt8(), readIntBE(0, 4), readBigUInt64BE(0, 8)
...
let FloatNumber = data.slice(start, end).readFloatBE();
console.log('FloatNumber', FloatNumber);
...
}
const buf = Buffer.from([0x40, 0x82, 0xe2, 0x31, 0xd6, 0xd7, 0x2e, 0x8d]);
console.log( buf.readDoubleBE(0) );
// Prints: 604.274335557087
Related
I'm receiving a high volume of "hex array" strings in the form:
'16 03 03 00 50 40 f2 12 71 0b c0 4f 99 dc 87 6f'
What's the most efficient way of copying them into an existing, larger buffer?
I'm guessing the naive way would be:
var lineBuffer = new Buffer(line.replace(/\s+/g, ''), 'hex');
lineBuffer.copyTo(mainBuffer, offset);
offset += 16;
I'm wary of using line[index] and doing the simple bit shift and sum, because string[index] just resolves to another string.
To move this out of the comments,
The solution you're looking for is probably something like this:
const line = '16 03 03 00 50 40 f2 12 71 0b c0 4f 99 dc 87 6f';
const regex = / /g;
const encoding = 'hex';
const replacement = '';
function getHexString(input) {
return input.replace(regex, replacement);
}
const existingBuffer = Buffer.alloc(1024); // just as an example
const offset = existingBuffer.length; // or wherever you need them to go.
function write(buffer, str, pos) {
const hexString = getHexString(str);
buffer.write(hexString, pos, encoding);
}
write(existingBuffer, line, 0);
I can get about 1,600,000 ops/sec through benchmark.js with () => write(existingBuffer, line, 0). Cheating a bit, since I'm constantly writing to position 0 instead of appending, but it should be close enough to get you an idea. The closest I could come with other combinations I was trying was about 1,200,000 ops/sec.
Also as a side note, I would strongly suggest using Buffer.alloc() wherever you are creating your original buffer, if you aren't. You can also use Buffer.allocUnsafe() which is faster, but that may leave non-zero data in the buffer. Which may be okay, if you know you're going to fill the entire buffer with new data before using it (or are only using known-filled slices). More reading here.
Background: I'm using node.js to get the volume setting from a device via serial connection. I need to obtain this data as an integer value.
I have the data in a buffer ('buf'), and am using readInt16BE() to convert to an int, as follows:
console.log( buf )
console.log( buf.readInt16BE(0) )
Which gives me the following output as I adjust the external device:
<Buffer 00 7e>
126
<Buffer 00 7f>
127
<Buffer 01 00>
256
<Buffer 01 01>
257
<Buffer 01 02>
258
Problem: All looks well until we reach 127, then we take a jump to 256. Maybe it's something to do with signed and unsigned integers - I don't know!
Unfortunately I have very limited documentation about the external device, I'm having to reverse engineer it! Is it possible it only sends a 7-bit value? Hopefully there is a way around this?
Regarding a solution - I must also be able to convert back from int to this format!
Question: How can I create a sequential range of integers when 7F seems to be the largest value my device sends, which causes a big jump in my integer scale?
Thanks :)
127 is the maximum value of a signed 8-bit integer. If the integer is overflowing into the next byte at 128 it would be safe to assume you are not being sent a 16 bit value, but rather 2 signed 8-bit values, and reading the value as a 16-bit integer would be incorrect.
I would start by using the first byte as a multiplier of 128 and add the second byte, this will give the series you are seeking.
buf = Buffer([0,127]) //<Buffer 00 7f>
buf.readInt8(0) * 128 + buf.readInt8(1)
>127
buf = Buffer([1,0]) //<Buffer 01 00>
buf.readInt8(0) * 128 + buf.readInt8(1)
>128
buf = Buffer([1,1]) //<Buffer 01 01>
buf.readInt8(0) * 128 + buf.readInt8(1)
>129
The way to get back is to divide by 128, round it down to the nearest integer for the first byte, and the second byte contains the remainder.
i = 129
buf = Buffer([Math.floor(i / 128), i % 128])
<Buffer 01 01>
Needed to treat the data as two signed 8-bit values. As per #forrestj the solution is to do:
valueInt = buf.readInt8(0) * 128 + buf.readInt8(1)
We can also convert the int value into the original format by doing the following:
byte1 = Math.floor(valueInt / 128)
byte2 = valueInt % 128
I have a buffer in node <Buffer 42 d9 00 00> that is supposed to represent the decimal 108.5. I am using this module to try and decode the buffer: https://github.com/feross/ieee754.
ieee754.read = function (buffer, offset, isLE, mLen, nBytes)
The arguments mean the following:
buffer = the buffer
offset = offset into the buffer
value = value to set (only for write)
isLe = is little endian?
mLen = mantissa length
nBytes = number of bytes
I try to read the value: ieee754.read(buffer, 0, false, 5832704, 4) but am not getting the expected result. I think I am calling the function correctly, although I am unsure about the mLen argument.
[I discovered that] the node Buffer class has that ability built in: buffer.readFloatBE(0).
I'm attempting to display the character í from 0xed (237).
String.fromCharCode yields the correct result:
String.fromCharCode(0xed); // 'í'
However, when using a Buffer:
var buf = new Buffer(1);
buf.writeUInt8(0xed,0); // <Buffer ed>
buf.toString('utf8'); // '?', same as buf.toString()
buf.toString('binary'); // 'í'
Using 'binary' with Buffer.toString is to be deprecated so I want to avoid this.
Second, I can also expect incoming data to be multibyte (i.e. UTF-8), e.g.:
String.fromCharCode(0x0512); // Ԓ - correct
var buf = new Buffer(2);
buf.writeUInt16LE(0x0512,0); // <Buffer 12 05>, [0x0512 & 0xff, 0x0512 >> 8]
buf.toString('utf8'); // Ԓ - correct
buf.toString('binary'); // Ô
Note that both examples are inconsistent.
SO, what am I missing? What am I assuming that I shouldn't? Is String.fromCharCode magical?
Seems you might be assuming that Strings and Buffers use the same bit-length and encoding.
JavaScript Strings are 16-bit, UTF-16 sequences while Node's Buffers are 8-bit sequences.
UTF-8 is also a variable byte-length encoding, with code points consuming between 1 and 6 bytes. The UTF-8 encoding of í, for example, takes 2 bytes:
> new Buffer('í', 'utf8')
<Buffer c3 ad>
And, on its own, 0xed is not a valid byte in UTF-8 encoding, thus the ? representing an "unknown character." It is, however, a valid UTF-16 code for use with String.fromCharCode().
Also, the output you suggest for the 2nd example doesn't seem correct.
var buf = new Buffer(2);
buf.writeUInt16LE(0x0512, 0);
console.log(buf.toString('utf8')); // "\u0012\u0005"
You can detour with String.fromCharCode() to see the UTF-8 encoding.
var buf = new Buffer(String.fromCharCode(0x0512), 'utf8');
console.log(buf); // <Buffer d4 92>
I have a java-based server that allows client applications to connect via other programming languages (java, unity, obj-c). I would like to know how to add javascript to this list using node.js and socket.io. The server listens on a set port and accepts simple json data plus an int for length in bytes, it response in the same format. The format of the "packet" is like so:
first four bytes are the length
00 00 00 1c
remaining bytes are the data
7b 22 69 64 22 3a 31 2c 22 6e 61 6d 65 22 3a 22 73 6f 6d 65 77 69 64 67 65 74 22 7d
The data is sent over TCP and is encoded in little endian. The object in originating from Java is modeled like so:
public class Widget {
private int id;
private String name;
public int getId() { return id; }
public String getName() { return name; }
}
The JSON would be:
{"id":1,"name":"somewidget"}
You will need a TCP socket. Connect it to the service and listen for the data event. When the data event is fired look at the buffers length. If it is <= 4 byte, you propably should discard it*. Else read the first 4 bytes using readUInt32() specifying 'little' as the endianess. Then use this number for the length of the remainding buffer. If the buffer is shorter than the given length, only "read" the remaining length, else "read" the given length. Use the toString method for this. You will get a string that can be parsed using the JSON.parse method, which will return you the JavaScript object matching the JSON.
You can build your packets basicly the same way by instanciating a buffer and writing all the data to it.
Also see the Buffers and net documentation for details.
* I do not know when node fires it's data events but your data might be received fragmented (that is splitted up into multiple data events). It could happen and due to the streaming nature of the tcp protocol it most likely will happen if the JSON string is long enough to not fit into a single frame. In that case, you propably should not simply discard the buffer, but try to reassemble the fragmented data.
So if you want just send request to server and get response you could do this:
var net = require('net');
var soc = net.Socket();
soc.connect(8088);
soc.on('connect', function(){
var data, request, header;
data = {request : true};
data = JSON.stringify(data);
request = new Buffer(Buffer.byteLength(data));
request.write(data);
header = new Buffer(4);
header.write(request.length.toString());
// send request
soc.end(header.toString('binary') + request.toString('binary'));
});
soc.on('data', function(buffer){
// Crop length bytes
var data = JSON.parse(buffer.slice(4).toString('utf-8'));
soc.destroy();
console.log(data);
});