My code, that provides a BLE service, is running under nodejs and is using bleno package.
My service characteristic supports read and notify.
Here is my simplified implementation of onSubscribe function:
function(maxValueSize, updateValueCallback) {
var a = buildPacket();
var buf = Buffer.from(a);
console.log('Buffer: ', buf);
updateValueCallback(this.RESULT_SUCCESS, buf);
}
Function buildPacket returns a Uint8Array of 20 bytes. The console statement shows that the buffer is of 20 bytes and has the expected values.
However, call to updateValueCallback throws an error:
RangeError: Index out of range
at checkInt (buffer.js:1187:11)
at Buffer.writeUInt8 (buffer.js:1233:5)
at Gatt.<anonymous> (/home/pi/Dev/MyTest/node_modules/bleno/lib/hci socket/gatt.js:881:33)
Here is the relevant code from gatt.js:
878 if (useNotify) {
879 var notifyMessage = new Buffer(3 + dataLength);
880
881 notifyMessage.writeUInt8(ATT_OP_HANDLE_NOTIFY, 0);
882 notifyMessage.writeUInt16LE(valueHandle, 1);
Is there any step I am missing?
Most examples on bleno I read seem to send string data. However, I need to send binary data. Is there anything special required for sending binary data?
Turns out updateValueCallback takes only one parameter. I should have looked at bleno examples a bit more carefully:-(.
Just passing buf as the sole parameter fixed the problem.
Related
I'm trying to publish an ArrayBuffer to a IORedis stream.
I do so as follow:
const ab = new ArrayBuffer(1); // ArrayBuffer of length = 1 byte
const dv = new DataView(ab);
dv.setInt8(0, 7); // Write the number 7 in the buffer
const buffer = Buffer.from(ab); // Convert to Buffer since that's what `publish` expects
redisPublisher.publish('buffer-test', buffer);
It's a toy example, in practice I'll want to encode complex stuff in the ArrayBuffer, not just a number. Anyway, then I try to read with
redisSubscriber.on('message', async (channel, data) => {
logger.info(`Redis message: channel: ${channel}, data: ${data}, ${typeof data}`);
// ... do something with it
});
The problem is that data is empty, and its type is considered as string. As per the documentation I tried redisSubscriber.on('messageBuffer', ... instead, but it behaves exactly the same, so much so that I'm failing to understand the difference between the two.
Also confusing is that if I encode a Buffer, e.g.
const buffer = Buffer.from("I'm a string!", 'utf-8');
redisPublisher.publish('buffer-test', buffer);
Upon reception, data will again be a string, decoded from the Buffer, which in that toy case is ok but generally is not for me. I'd like to send an Buffer in, containing more complex data that just a string (an ArrayBuffer in my case), and get a Buffer out, that I could properly parse based on my needs and not have automatically read as a string.
Any help is welcome!
I am implementing a H264 video decoder through Direct3D 12 APIs - while I am very new to Direct3D I do have experience with other graphics APIs and H264. I have been struggling to find decent examples of D3D12 video decoding but have got as far as seemingly successfully submitting my work to the decoder.
However, I am at a loss with how to query the decode status. From piecing together the documentation and some other code I found I think it's something like this - mapping the result into the status struct - but I get an invalid argument error. Could anyone point me in the right direction, and any good examples of D3D12 video decoding would be a great resource.
// decode_commands is a ID3D12VideoDecodeCommandList
// query_heap is a ID3D12QueryHeap
// device is a ID3D12Device
// Make query for decode stats.
decode_commands->EndQuery(query_heap.Get(), D3D12_QUERY_TYPE::D3D12_QUERY_TYPE_VIDEO_DECODE_STATISTICS, 0);
// Create buffer for query result.
ComPtr<ID3D12Resource> query_result;
D3D12_RESOURCE_DESC query_result_description = CD3DX12_RESOURCE_DESC::Buffer(sizeof(D3D12_QUERY_DATA_VIDEO_DECODE_STATISTICS));
HRESULT create_query_result = device->CreateCommittedResource(
&CD3DX12_HEAP_PROPERTIES(D3D12_HEAP_TYPE::D3D12_HEAP_TYPE_DEFAULT),
D3D12_HEAP_FLAGS::D3D12_HEAP_FLAG_NONE,
&query_result_description,
D3D12_RESOURCE_STATES::D3D12_RESOURCE_STATE_COPY_DEST,
nullptr,
IID_PPV_ARGS(&query_result));
if (FAILED(create_query_result)) {
log("Failed to create query result");
return false;
}
// Resolve query.
decode_commands->ResolveQueryData(query_heap.Get(), D3D12_QUERY_TYPE::D3D12_QUERY_TYPE_VIDEO_DECODE_STATISTICS, 0, 1, query_result.Get(), 0);
// Get stats from query result.
D3D12_RANGE range;
range.Begin = 0;
range.End = sizeof(D3D12_QUERY_DATA_VIDEO_DECODE_STATISTICS);
D3D12_QUERY_DATA_VIDEO_DECODE_STATISTICS stats;
HRESULT map = query_result->Map(0, &range, reinterpret_cast<void**>(&stats));
if (FAILED(map)) {
log("Failed to map query result");
return false;
}
Perhaps you should begin with a working sample from D3D9 API :
H264Dxva2Decoder
Than, you write a D3D12 decoder, that's what i would do.
What is the nodejs (typescript) equivalent of the following Python snippet? I've put an attempt at corresponding nodejs below the Python.
Note that I want to read a chunk at a time (later that is, in this example I'm just reading the first kilobyte), synchronously.
Also, I do not want to read the entire file into virtual memory at once; some of my input files will (eventually) be too big for that.
The nodejs snippet always returns null. I want it to return a string or buffer or something along those lines. If the file is >= 1024 bytes long, I want a 1024 character long return, otherwise I want the entire file.
I googled about this for an hour or two, but all I found was things synchronously reading an entire file at a time, or reading pieces at a time asynchronously.
Thanks!
Here's the Python:
def readPrefix(filename: str) -> str:
with open(filename, 'rb') as infile:
data = infile.read(1024)
return data
Here's the nodejs attempt:
const readPrefix = (filename: string): string => {
const readStream = fs.createReadStream(filename, { highWaterMark: 1024 });
const data = readStream.read(1024);
readStream.close();
return data;
};
To read synchronously, you would use fs.openSync(), fs.readSync() and fs.closeSync().
Here's some regular Javascript code (hopefully you can translate it to TypeScript) that synchronously reads a certain number of bytes from a file and returns a buffer object containing those bytes (or throws an exception in case of error):
const fs = require('fs');
function readBytesSync(filePath, filePosition, numBytesToRead) {
const buf = Buffer.alloc(numBytesToRead, 0);
let fd;
try {
fd = fs.openSync(filePath, "r");
fs.readSync(fd, buf, 0, numBytesToRead, filePosition);
} finally {
if (fd) {
fs.closeSync(fd);
}
}
return buf;
}
For your application, you can just pass 1024 as the bytes to read and if there are less than that in the file, it will just read up until the end of the file. The returns buffer object will contain the bytes read which you can access as binary or convert to a string.
For the benefit of others reading this, I mentioned in earlier comments that synchronous I/O should never be used in a server environment (servers should always use asynchronous I/O except at startup time). Synchronous I/O can be used for stand-alone scripts that only do one thing (like build scripts, as an example) and don't need to be responsive to multiple incoming requests.
Do I need to loop on readSync() in case of EINTR or something?
Not that I'm aware of.
I have a file in a binary format:
The format is as follows:
[4 - header bytes] [8 bytes - int64 - how many bytes to read following] [variable num of bytes (size of the int64) - read the actual information]
And then it repeats, so I must first read the first 12 bytes to determine how many more bytes I need to read.
I have tried:
var readStream = fs.createReadStream('/path/to/file.bin');
readStream.on('data', function(chunk) { ... })
The problem I have is that chunk always comes back in chunks of 65536 bytes at a time whereas I need to be more specific on the number of bytes that I am reading.
I have always tried readStream.on('readable', function() { readStream.read(4) })
But it is also not very flexible, because it seems to turn asynchronous code into synchronous code because, I have to put the 'reading' in a while loop
Or maybe readStream is not appropriate in this case and I should use this instead? fs.read(fd, buffer, offset, length, position, callback)
Here's what I'd recommend as an abstract handler of a readStream to process abstract data like you're describing:
var pending = new Buffer(9999999);
var cursor = 0;
stream.on('data', function(d) {
d.copy(pending, cursor);
cursor += d.length;
var test = attemptToParse(pending.slice(0, cursor));
while (test !== false) {
// test is a valid blob of data
processTheThing(test);
var rawSize = test.raw.length; // How many bytes of data did the blob actually take up?
pending.copy(pending.copy, 0, rawSize, cursor); // Copy the data after the valid blob to the beginning of the pending buffer
cursor -= rawSize;
test = attemptToParse(pending.slice(0, cursor)); // Is there more than one valid blob of data in this chunk? Keep processing if so
}
});
For your use-case, ensure the initialized size of the pending Buffer is large enough to hold the largest possible valid blob of data you'll be parsing (you mention an int64; that max size plus the header size) plus one extra 65536 bytes in case the blob boundary happens just on the edge of a stream chunk.
My method requires a attemptToParse() method that takes a buffer and tries to parse the data out of it. It should return false if the length of the buffer is too short (data hasn't come in enough yet). If it is a valid object, it should return some parsed object that has a way to show the raw bytes it took up (.raw property in my example). Then you do any processing you need to do with the blob (processTheThing()), trim out that valid blob of data, shift the pending Buffer to just be the remainder and keep going. That way, you don't have a constantly growing pending buffer, or some array of "finished" blobs. Maybe process on the receiving end of processTheThing() is keeping an array of the blobs in memory, maybe it's writing them to a database, but in this example, that's abstracted away so this code just deals with how to handle the stream data.
Add the chunk to a Buffer, and then parse the data from there. Being aware not to go beyond the end of the buffer (if your data is large). I'm using my tablet right now so can't add any example source code. Maybe somebody else can?
Ok, mini source, very skeletal.
var chunks = [];
var bytesRead= 0;
stream.on('data', function(chunk) {
chunks.push(chunk);
bytesRead += chunk.length;
// look at bytesRead...
var buffer = Buffer.concat(chunks);
chunks = [buffer]; // trick for next event
// --> or, if memory is an issue, remove completed data from the beginning of chunks
// work with the buffer here...
}
Websocket on Client:
socket.send('helloworld');
Websocket on Node.js:
socket.ondata = function(d, start, end){
// I suppose that the start and end indicates the location of the
// actual data 'hello world' after the headers
var data = d.toString('utf8', start, end);
// Then I'm just printing it
console.log(data);
});
but I'm getting this: �����]���1���/�� on the terminal :O
I have tried to understand this doc: https://www.rfc-editor.org/rfc/rfc6455#section-5.2 but it's hard to understand because I don't know what should I work with, I mean I can't see the data even with toString?
I have tried to follow and test with this questions answer How can I send and receive WebSocket messages on the server side? but I can't get it to work, with this answer I was getting an array like this [false, true, false, false, true, true, false] etc... and I don't really know what to do with it.. :\
So I'm a bit confused, what the hell should I do after I get the data from the client side to get the real message?
I'm using the original client side and node.js API without any library.
Which node.js library are you using? Judging by the fact that you are hooking socket.ondata that looks like the HTTP server API. WebSockets is not HTTP. It has an HTTP compatible handshake so that the WebSocket and HTTP service can live on the same port, but that's where the similarity ends. After the handshake, WebSockets is a framed full-duplex, long-lived message transport more similar to regular TCP sockets than to HTTP.
If you want to implement your own WebSocket server in node.js you are going to want to use the socket library directly (or build on/borrow existing WebSocket server code).
Here is a Node.js based WebSocket server that bridges from WebSocket to TCP sockets: https://github.com/kanaka/websockify/blob/master/other/websockify.js Note that it is for the previous Hixie version of the protocol (I haven't had opportunity or motivation to update it yet). The modern HyBI version of the protocol is very different but you might be able to glean some useful information from that implementation.
You can in fact start with Node's HTTP API. That is exactly what I did when writing the WebSocket-Node module https://github.com/Worlize/WebSocket-Node
If you don't want to use an existing WebSocket Library (though you really should just use an existing library) then you need to be able to parse the binary data format defined by the RFC. It's very clear about the format and exactly how to interpret the data. From each frame you have to read in all the flags, interpret the frame size, possibly read the masking key, and unmask the contents as you read them from the wire.
That is one reason you're not seeing anything recognizable... in WebSockets, all client-to-server communications is obfuscated by applying a random mask to the contents using XOR as a security precaution against possibly poisoning the cache of older proxy servers that don't know about websockets.
There are two things -
Which node.js version are you using? I have never seen a data event with start and endpt. The emitted event is just data with buffer/string as an argument.
More importantly, if you are hooking on to the HTTP socket you should take care of the HTTP Packet. It contains hearders, body and a trailer. There might be garbage in there.
Here is a solution from this post:
https://medium.com/hackernoon/implementing-a-websocket-server-with-node-js-d9b78ec5ffa8
parseMessage(buffer) {
const firstByte = buffer.readUInt8(0);
//const isFinalFrame = Boolean((firstByte >>> 7) & 0x1);
//const [reserved1, reserved2, reserved3] = [ Boolean((firstByte >>> 6) & 0x1),
Boolean((firstByte >>> 5) & 0x1), Boolean((firstByte >>> 4) & 0x1) ];
const opCode = firstByte & 0xF;
// We can return null to signify that this is a connection termination frame
if (opCode === 0x8)
return null;
// We only care about text frames from this point onward
if (opCode !== 0x1)
return;
const secondByte = buffer.readUInt8(1);
const isMasked = Boolean((secondByte >>> 7) & 0x1);
// Keep track of our current position as we advance through the buffer
let currentOffset = 2; let payloadLength = secondByte & 0x7F;
if (payloadLength > 125) {
if (payloadLength === 126) {
payloadLength = buffer.readUInt16BE(currentOffset);
currentOffset += 2;
} else {
// 127
// If this has a value, the frame size is ridiculously huge!
//const leftPart = buffer.readUInt32BE(currentOffset);
//const rightPart = buffer.readUInt32BE(currentOffset += 4);
// Honestly, if the frame length requires 64 bits, you're probably doing it wrong.
// In Node.js you'll require the BigInt type, or a special library to handle this.
throw new Error('Large payloads not currently implemented');
}
}
const data = Buffer.alloc(payloadLength);
// Only unmask the data if the masking bit was set to 1
if (isMasked) {
let maskingKey = buffer.readUInt32BE(currentOffset);
currentOffset += 4;
// Loop through the source buffer one byte at a time, keeping track of which
// byte in the masking key to use in the next XOR calculation
for (let i = 0, j = 0; i < payloadLength; ++i, j = i % 4) {
// Extract the correct byte mask from the masking key
const shift = j == 3 ? 0 : (3 - j) << 3;
const mask = (shift == 0 ? maskingKey : (maskingKey >>> shift)) & 0xFF;
// Read a byte from the source buffer
const source = buffer.readUInt8(currentOffset++);
// XOR the source byte and write the result to the data
data.writeUInt8(mask ^ source, i);
}
} else {
// Not masked - we can just read the data as-is
buffer.copy(data, 0, currentOffset++);
}
return data
}