I am working on Change Streams introduced in MongoDB Version 3.6. Change Streams have a feature where I can specify to start streaming changes from a particular change in history. In native driver for Node.js, to resume change stream, it says (documentation here)
Specifies the logical starting point for the new change stream. This should be the _id field from a previously returned change stream document.
When I print it in console, this is what I am getting
{ _id:
{ _data:
Binary {
_bsontype: 'Binary',
sub_type: 0,
position: 49,
buffer: <Buffer 82 5a 61 a5 4f 00 00 00 01 46 64 5f 69 64 00 64 5a 61 a5 4f 08 c2 95 31 d0 48 a8 2e 00 5a 10 04 7c c9 60 de de 18 48 94 87 3f 37 63 08 da bb 78 04> } },
...
}
My problem is I do not know how to store the _id of this format in a database or a file. Is it possible to convert this binary object to string so I can use it later to resume my change stream from that particular _id. Example code would be greatly appreciated.
Convert BSON Binary to buffer and back
const Binary = require('mongodb').Binary;
const fs = require('fs');
Save _data from _id:
var wstream = fs.createWriteStream('/tmp/test');
wstream.write(lastChange._id._data.read(0));
wstream.close();
Then rebuild resumeToken:
fs.readFile('/tmp/test', void 0, function(err, data) {
const resumeToken = { _data: new Binary(data) };
});
Related
A mqtt client send a binary message to certain topic. My node.js client subscribes to the topic and recieves the binary data. As our architecture payload is Int16Array. But I cannot cast it successfully to Javascript array.
//uint16 array [255, 256, 257, 258] sent as message payload, which contents <ff 00 00 01 01 01 02 01>
When I do this:
mqttClient.on("message", (topic, payload) => {
console.log(payload.buffer);
})
The output like:
ArrayBuffer {
[Uint8Contents]: <30 11 00 07 74 65 73 74 6f 7a 69 ff 00 00 01 01 01 02 01>,
byteLength: 19
}
which cant be cast to Int16Array because of odd length
It also contains more bytes than the original message
As it seems the original bytes exist at the end of the payload, which is offset for some reason.
Buffer also contains the offset and byte length information inside. By using them casting should be successful.
let arrayBuffer = payload.buffer.slice(payload.byteOffset,payload.byteOffset + payload.byteLength)
let int16Array = new Int16Array(arrayBuffer)
let array = Array.from(arrayBuffer)
I am trying to build quick nodejs script to look at some data in my network. Using node_pcap I manage to decode almost everything but the payload data that is end through the UDP protocol. This is my code (fairly basic but gives me the headers and payloads)
const interface = 'en0';
let filter = 'udp';
const pcap = require('pcap'),
pcap_session = pcap.createSession(interface, filter),
pcap_session.on('packet', function (raw_packet) {
let packet = pcap.decode.packet(raw_packet);
let data = packet.payload.payload.payload.data;
console.log(data.toString()); // not full data
});
When I try to print data using toString() method, it gives me most of the data but the beginning. I have something like this printed :
Li?��ddn-�*ys�{"Id":13350715,... I've cut the rest of the data which is the rest of the JSON.
But I am suspecting that the bit of data that I can't read contain some useful info such has how many packet, offset packet and so on..
I manage to get a part of it from the buffer from a payload :
00 00 00 01 52 8f 0b 4a 4d 3f cb de 08 00 01 00 00 00 04 a4 00 00 26 02 00 00 26 02 00 00 00 03 00 00 00 00 00 00 09 2d 00 00 00 00 f3 03 01 00 00 2a 00 02 00 79 00 05 73 01 d2
Although I have an idea of what kind of data it can be I have no idea of its structure.
Is there a way that I could decode this bit of the buffer ? I tried to look at some of the buffer method such as readInt32LE, readInt16LE but in vain. Is there some reading somewhere that can guide me through the process of decoding it?
[Edit] The more I looked into it, the more I suspect the data to be BSON and not JSON, that would explain why I can read some bit of it but not everything. Any chance someone manage to decode BSON from a packet ?
How does the library know which packet decoder to use?
It starts at Layer 2 of the TCP/IP stack by identifying which protocol is used https://github.com/node-pcap/node_pcap/blob/master/decode/pcap_packet.js#L29-L56
switch (this.link_type) {
case "LINKTYPE_ETHERNET":
this.payload = new EthernetPacket(this.emitter).decode(buf, 0);
break;
case "LINKTYPE_NULL":
this.payload = new NullPacket(this.emitter).decode(buf, 0);
break;
case "LINKTYPE_RAW":
this.payload = new Ipv4(this.emitter).decode(buf, 0);
break;
case "LINKTYPE_IEEE802_11_RADIO":
this.payload = new RadioPacket(this.emitter).decode(buf, 0);
break;
case "LINKTYPE_LINUX_SLL":
this.payload = new SLLPacket(this.emitter).decode(buf, 0);
break;
default:
console.log("node_pcap: PcapPacket.decode - Don't yet know how to decode link type " + this.link_type);
}
Then it goes upper and tries to decode the proper protocol based on the flags it finds in the header https://github.com/node-pcap/node_pcap/blob/master/decode/ipv4.js#L12-L17 in this particular case for the IPv4 protocol
IPFlags.prototype.decode = function (raw_flags) {
this.reserved = Boolean((raw_flags & 0x80) >> 7);
this.doNotFragment = Boolean((raw_flags & 0x40) > 0);
this.moreFragments = Boolean((raw_flags & 0x20) > 0);
return this;
};
Then in your case it would match with the udp protocol https://github.com/node-pcap/node_pcap/blob/master/decode/ip_protocols.js#L15
protocols[17] = require("./udp");
Hence, If you check https://github.com/node-pcap/node_pcap/blob/master/decode/udp.js#L32 the packet is automatically decoded and it exposes a toString method
UDP.prototype.toString = function () {
var ret = "UDP " + this.sport + "->" + this.dport + " len " + this.length;
if (this.sport === 53 || this.dport === 53) {
ret += (new DNS().decode(this.data, 0, this.data.length).toString());
}
return ret;
};
What does this mean for you?
In order to decode a udp(any) packet you just call the high level api pcap.decode.packet(raw_packet) and then call toString method to display the decoded body payload
pcap_session.on('packet', function (raw_packet) {
let packet = pcap.decode.packet(raw_packet);
console.log(packet.toString());
});
I have been given a text file containing hex data which I know forms a jpeg image. Below is an example of the format:
FF D8 FF E0 00 10 4A 46 49 46 00 01 02 00 00 64 00 64 00 00 FF E1 00 B8 45 78 69 00 00 4D
This is only a snippet but you get the idea.
Does anyone know how I could convert this back into the original jpeg?
To convert from a hex string to a byte you use the Convert.ToByte with a base 16 parameter.
To convert a byte array to a Bitmap you put it in a Stream and use the Bitmap(Stream) constructor:
using System.IO;
//..
string hexstring = File.ReadAllText(yourInputFile);
byte[] bytes = new byte[hexstring.Length / 2];
for (int i = 0; i < hexstring.Length; i += 2)
bytes[i / 2] = Convert.ToByte( hexstring.Substring(i, 2), 16);
using (MemoryStream ms = new MemoryStream(bytes))
{
Bitmap bmp = new Bitmap(ms);
// now you can do this:
bmp.Save(yourOutputfile, System.Drawing.Imaging.ImageFormat.Jpeg);
bmp.Dispose(); // dispose of the Bitmap when you are done with it!
// or that:
pictureBox1.Image = bmp; // Don't dispose as long as the PictureBox needs it!
}
I guess that there are more LINQish way but as long as it works..
I have following function in node.js inside a http.request()
res.on('data', function (chunk) {
var sr="response: "+chunk;
console.log(chunk);
});
I get this in console
<Buffer 3c 3f 78 6d 6c 20 76 65 72 73 69 6f 6e 3d 22 31 2e 30 22 20 65 6e 63 6f
64 69 6e 67 3d 22 75 74 66 2d 38 22 20 3f 3e 3c 72 65 73 75 6c 74 3e 3c 73 75 63
...>
But when i use this:
res.on('data', function (chunk) {
var sr="response: "+chunk;
console.log(sr);
});
I get a proper xml response like this:
responose: .....xml responose.....
I don't understand why i need to append a string to output the proper response. And what is meant by the response generated in the first code?
chunk is a Buffer, which is Node's way of storing binary data.
Because it's binary data, you need to convert it to a string before you can properly show it (otherwise, console.log will show its object representation). One method is to append it to another string (your second example does that), another method is to call toString() on it:
console.log(chunk.toString());
However, I think this has the potential of failing when chunk contains incomplete characters (an UTF-8 character can consist of multiple bytes, but you don't get the guarantee that chunk isn't cut off right in the middle of such a byte string).
Chunk is just a buffer where the data is stored in Binary, so you could use utf8 for the character encoding as well which will output the data as String, and this you will need to do when you are creating the readStream.
var myReadStream = fs.createReadStream( __dirname + '/readme.txt', 'utf8');
myReadStream.on('data', function(chunk){
console.log('new chunk received');
console.log(chunk);
})
Can someone please explain to me how the zlib library works in Nodejs?
I'm fairly new to Nodejs, and I'm not yet sure how to use buffers and streams.
My simple scenario is a string variable, and I want to either zip or unzip (deflate or inflate, gzip or gunzip, etc') the string to another string.
I.e. (how I would expect it to work)
var zlib = require('zlib');
var str = "this is a test string to be zipped";
var zip = zlib.Deflate(str); // zip = [object Object]
var packed = zip.toString([encoding?]); // packed = "packedstringdata"
var unzipped = zlib.Inflate(packed); // unzipped = [object Object]
var newstr = unzipped.toString([again - encoding?]); // newstr = "this is a test string to be zipped";
Thanks for the helps :)
For anybody stumbling on this in 2016 (and also wondering how to serialize compressed data to a string rather than a file or a buffer) - it looks like zlib (since node 0.11) now provides synchronous versions of its functions that do not require callbacks:
var zlib = require('zlib');
var input = "Hellow world";
var deflated = zlib.deflateSync(input).toString('base64');
var inflated = zlib.inflateSync(new Buffer(deflated, 'base64')).toString();
console.log(inflated);
Syntax has changed to simply:
var inflated = zlib.inflateSync(Buffer.from(deflated, 'base64')).toString()
Update: Didn't realize there was a new built-in 'zlib' module in node 0.5. My answer below is for the 3rd party node-zlib module. Will update answer for the built-in version momentarily.
Update 2: Looks like there may be an issue with the built-in 'zlib'. The sample code in the docs doesn't work for me. The resulting file isn't gunzip'able (fails with "unexpected end of file" for me). Also, the API of that module isn't particularly well-suited for what you're trying to do. It's more for working with streams rather than buffers, whereas the node-zlib module has a simpler API that's easier to use for Buffers.
An example of deflating and inflating, using 3rd party node-zlib module:
// Load zlib and create a buffer to compress
var zlib = require('zlib');
var input = new Buffer('lorem ipsum dolor sit amet', 'utf8')
// What's 'input'?
//input
//<Buffer 6c 6f 72 65 6d 20 69 70 73 75 6d 20 64 6f 6c 6f 72 20 73 69 74 20 61 6d 65 74>
// Compress it
zlib.deflate(input)
//<SlowBuffer 78 9c cb c9 2f 4a cd 55 c8 2c 28 2e cd 55 48 c9 cf c9 2f 52 28 ce 2c 51 48 cc 4d 2d 01 00 87 15 09 e5>
// Compress it and convert to utf8 string, just for the heck of it
zlib.deflate(input).toString('utf8')
//'x???/J?U?,(.?UH???/R(?,QH?M-\u0001\u0000?\u0015\t?'
// Compress, then uncompress (get back what we started with)
zlib.inflate(zlib.deflate(input))
//<SlowBuffer 6c 6f 72 65 6d 20 69 70 73 75 6d 20 64 6f 6c 6f 72 20 73 69 74 20 61 6d 65 74>
// Again, and convert back to our initial string
zlib.inflate(zlib.deflate(input)).toString('utf8')
//'lorem ipsum dolor sit amet'
broofa's answer is great, and that's exactly how I'd like things to work. For me node insisted on callbacks. This ended up looking like:
var zlib = require('zlib');
var input = new Buffer('lorem ipsum dolor sit amet', 'utf8')
zlib.deflate(input, function(err, buf) {
console.log("in the deflate callback:", buf);
zlib.inflate(buf, function(err, buf) {
console.log("in the inflate callback:", buf);
console.log("to string:", buf.toString("utf8") );
});
});
which gives:
in the deflate callback: <Buffer 78 9c cb c9 2f 4a cd 55 c8 2c 28 2e cd 55 48 c9 cf c9 2f 52 28 ce 2c 51 48 cc 4d 2d 01 00 87 15 09 e5>
in the inflate callback: <Buffer 6c 6f 72 65 6d 20 69 70 73 75 6d 20 64 6f 6c 6f 72 20 73 69 74 20 61 6d 65 74>
to string: lorem ipsum dolor sit amet
Here is a non-callback version of the code:
var zlib = require('zlib');
var input = new Buffer.from('John Dauphine', 'utf8')
var deflated= zlib.deflateSync(input);
console.log("Deflated:",deflated.toString("utf-8"));
var inflated = zlib.inflateSync(deflated);
console.log("Inflated:",inflated.toString("utf-8"))