I'm reading a binary file from my file system using Node.
When I compare the streamed data to a hexdump that I've done, I get different results.
My node code looks like this:
this.readStream = fs.createReadStream(file, {
encoding: 'binary',
});
this.readStream.on('data', (data: string | Buffer) => {
if (!(data instanceof Buffer)) {
data = Buffer.from(data);
}
let dataString = "";
data.forEach(v => {
dataString += v.toString(2).padStart(8, '0')
});
console.log(dataString);
});
The output from above is:
000000000000000000000000000000000000000000000010001000100010001000100010001000100010001000100010001000100010001000100010001000110001100111000010101000100010001100100010001000100010001100100010001000100010001100100010001000100010001000110010001000110010001000100010001000110010001000100010001000110010001000100010001000110000011100100010001000110010001000100010001000110010001000100010001000110010001000100010001000100010001000100011001000100010001000100011001000100010001000100011001000100010001000100011000001011100001010100010001000110010001000100010001000110010001000100010001000110010001000100010001000011100001010110110001000011100001010100010001100100010000111000010101000100011001000100011001000100010001000100011000001110010001000100011001000100010001000100000011000100010001000100011001000100010001000100011110000101010001000100011001000100010001000100011001000100010001000100011001000100010001000100000010001000110001000100011001000100010001000100011001000100010001000100011001000100010001000100011001000100010001100100010001000100010000001100010001000100010001100100010001000100010001100000110001000100010001100100010001000100010001100100010001000100010001100100010001000100010001100100010001000110010001000100010001000110010001000100010001000110010001000100010001000000100010111000010101000100010001100100010001000100010001100100010001000100010001100100010001000100010001100000110001000110010001000100010001000000110001000100010001000110010001000100010001000110000011000100010001000110010001000100010001000110010001000100010001000110010001000100010001000000110001000100011001000100010001000100011001000100010001000100011001000100010001000100011000111101100001010100010001000110010001000100010001000000110001000100010001000100010001000011010001000011100001110100010001000110010001000100010001000110010001000100010001000110010001000100010001000000100011000100010001000110010001000100010001000110010001000100010001000110010001000100010001000110001101000100011001000100010001000100000011000100010001000100011001000100010001000100011000001101100001010100010001000110010001000100010001000110010001000100010001000110010001000100010001000000100010100100011001000100001001000100011001000100001001000100011001000100001001000100011000001011100001010010010001000110010001000010010001000110010001000010010001000110010001000010010001000100010000100100011001000100001001000100000110000111010001000010010001000110010001000010010001000110000010001010010001000110010001000010010001000110010001000010010001000110010001000010010001000000110000100100011001000100001001000100011001000100001001000100011001000100001001000100011000001100001001000100011001000100001001000100000110000111010001000010010001000110010001000010010001000110011000100100011001000100001001000100011001000100001001000100011001000100001001000100000110000111001110111000010100100100010000100100010001010100010001011000010101000100011001000100011001000100001001000100011110000111011111111000011101111100000000000000000110000101000000100000001110000101000001000000010110000101000001100000011110000101000010000000100110000101000010100000101110000101000011000000110110000101000011100000111110000101000100000001000110000101000100100001001110000101000101000001010110000101000101100001100000011010000111000001111000100000001000100010010000100110111111111000011101111111100001110111111110000111011111111000011101111111100001110111111110000111011111111000011101111111100001110111111110000111011111111000011101111111100001110111111110000111011111111000011101111111100001110111100
The Hexdump that I used is: xxd -b -g 0 ${file}
And its output is:
0000000000000000000000000000000000000000000000100010001000100010001000100010001000100010001000100010001000100010001000100010001100011001101000100010001100100010001000100010001100100010001000100010001100100010001000100010001000110010001000110010001000100010001000110010001000100010001000110010001000100010001000110000011100100010001000110010001000100010001000110010001000100010001000110010001000100010001000100010001000100011001000100010001000100011001000100010001000100011001000100010001000100011000001011010001000100011001000100010001000100011001000100010001000100011001000100010001000100001101101100010000110100010001100100010000110100010001100100010001100100010001000100010001100000111001000100010001100100010001000100010000001100010001000100010001100100010001000100010001110100010001000110010001000100010001000110010001000100010001000110010001000100010001000000100010001100010001000110010001000100010001000110010001000100010001000110010001000100010001000110010001000100011001000100010001000100000011000100010001000100011001000100010001000100011000001100010001000100011001000100010001000100011001000100010001000100011001000100010001000100011001000100010001100100010001000100010001100100010001000100010001100100010001000100010000001000101101000100010001100100010001000100010001100100010001000100010001100100010001000100010001100000110001000110010001000100010001000000110001000100010001000110010001000100010001000110000011000100010001000110010001000100010001000110010001000100010001000110010001000100010001000000110001000100011001000100010001000100011001000100010001000100011001000100010001000100011000111101010001000100011001000100010001000100000011000100010001000100010001000100001101000100001111000100010001100100010001000100010001100100010001000100010001100100010001000100010000001000110001000100010001100100010001000100010001100100010001000100010001100100010001000100010001100011010001000110010001000100010001000000110001000100010001000110010001000100010001000110000011010100010001000110010001000100010001000110010001000100010001000110010001000100010001000000100010100100011001000100001001000100011001000100001001000100011001000100001001000100011000001011001001000100011001000100001001000100011001000100001001000100011001000100001001000100010001000010010001100100010000100100010000011100010000100100010001100100010000100100010001100000100010100100010001100100010000100100010001100100010000100100010001100100010000100100010000001100001001000110010001000010010001000110010001000010010001000110010001000010010001000110000011000010010001000110010001000010010001000001110001000010010001000110010001000010010001000110011000100100011001000100001001000100011001000100001001000100011001000100001001000100000110111011001001000100001001000100010101000100010101000100011001000100011001000100001001000100011111111111111111000000000000000001000000100000001100000100000001010000011000000111000010000000100100001010000010110000110000001101000011100000111100010000000100010001001000010011000101000001010100010110000110000001101000011100000111100010000000100010001001000010011011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111100
I don't believe the difference is because of big or little endian.
I don't think the issue is because of my use of padStart since each unit of the buffer should be 8 bits (1 byte).
The file I'm reading is only 398 bytes
So the question is how do I get Node to read a binary file perfectly from the HDD, and get a binary string representation of the file that is identical to what I would get from a binary dump of the same file.
Related
I'm trying to compress an image with pngquant. Here is the code:
let output = '';
const quant = cp.spawn('pngquant', ['256', '--speed', '10'], {
stdio: [null, null, 'ignore'],
});
quant.stdout.on('data', data => output += data);
quant.on('close', () => {
fs.writeFileSync('image.png', output);
fs.writeFileSync('image_original.png', image);
process.exit(0);
});
quant.stdin.write(image);
image is a Buffer with pure PNG data.
The code works, however, for some reason, it generates incorrect PNG. Not only that, but also it's size is more than original's.
When I execute this from the terminal, I get excellent output file:
pngquant 256 --speed 10 < image_original.png > image.png
I have no idea of what's going on; the data in output file seems pretty PNG-ish.
EDIT: I have managed to make it work:
let output = [];
quant.stdout.on('data', data => output.push(data));
quant.stdin.write(image);
quant.on('close', () => {
const image = Buffer.concat(output);
fs.writeFileSync('image.png', image);
});
I assume that is related to how strings are represented in the NodeJS. Would be happy to get some explanation.
What are the effective ways to work Node js with a large JSON file of 600 MB and more?
My partner gives me from his REST API wery large JSON file. 600mb, 1000mb
Its structure is as follows
{ nameid1:[list id,....], nameid2:[list id,....], }
[list id,....] - An array with ID can be up to hundreds of millions of records.
Now to work with such files I use the following sequence of actions.
I save it to hard drive
With the sed command, from a single-line file, I make it multi-line
Example
exec (`sed -i 's /', '/', '\ n / g' file.json)
I work directly with the file using readline
I tried to use JSONStream but it causes FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
function getStream() {
let jsonData = __dirname + '/jsonlarge/file.json',
stream = fs.createReadStream(jsonData, {
encoding: 'utf8'
})
parser = JSONStream.parse('*');
stream.pipe(parser)
parser.on('data', (data) => {
console.log('received:', data);
});
}
Example structure json file
{"Work":"12122001","name":"Regist","world":[{"name":"000000","point":"rfg","Content":["3202b9a3fba","121323","2343454","45345543","354534534"]}, {"name":"000000","point":"rfg","Content":["3202b","121323","2343454","45345543","354534534"]}, {"name":"000000","point":"rfg","Content":["320","121323","2343454","45345543","354534534"]}]}
Maybe someone knows a faster way to work with such files.
Thanks
I'm trying to capture frames from a live video stream (h.264) and pipe the resulting JPG images to a Node JS script, instead of saving these individual frames directly to .jpg files.
As a test, I created the following Node JS script, to simply capture the incoming piped data, then dump it to a file:
// pipe.js - test pipe output
var fs = require('fs');
var data = '';
process.stdin.resume();
process.stdin.setEncoding('utf8');
var filename = process.argv[2];
process.stdin.on('data', (chunk) => {
console.log('Received data chunk via pipe.');
data += chunk;
});
process.stdin.on('end', () => {
console.log('Data ended.');
fs.writeFile(filename, data, err => {
if (err) {
console.log('Error writing file: error #', err);
}
});
console.log('Saved file.');
});
console.log('Started... Filename = ' + filename);
Here's the ffmpeg command I used:
ffmpeg -vcodec h264_mmal -i "rtsp://[stream url]" -vframes 1 -f image2pipe - | node pipe.js test.jpg
This generated the following output, and also produced a 175kB file which contains garbage (unreadable as a jpg file anyway). FYI using ffmpeg to export directly to a jpg file produced files around 25kB in size.
...
Press [q] to stop, [?] for help
[h264_mmal # 0x130d3f0] Changing output format.
Input stream #0:0 frame changed from size:1280x720 fmt:yuvj420p to size:1280x720 fmt:yuv420p
[swscaler # 0x1450ca0] deprecated pixel format used, make sure you did set range correctly
Received data chunk via pipe.
Received data chunk via pipe.
frame= 1 fps=0.0 q=7.7 Lsize= 94kB time=00:00:00.40 bitrate=1929.0kbits/s speed=1.18x
video:94kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
Received data chunk via pipe.
Data ended.
Saved file.
You can see that the Node JS script is receiving piped data (per the "Received data via pipe" messages above. However, it doesn't seem to be outputting a valid JPG file. I can't find a way to specifically request that ffmpeg output JPG format, since there is no -vcodec option for JPG. I tried using -vcodec png and outputting to a .png file, but the resulting file was about 2MB in size and also unreadable as a png file.
Is this a problem caused by using utf8 encoding, or am I doing something else wrong?
Thanks for any advice.
UPDATE: OK I got it to send a single jpg image correctly. The issue was in the way Node JS was capturing the stream data. Here's a working script:
// pipe.js - capture piped binary input and write to file
var fs = require('fs');
var filename = process.argv[2];
console.log("Opening " + filename + " for binary writing...");
var wstream = fs.createWriteStream(filename);
process.stdin.on('readable', () => {
var chunk = '';
while ((chunk = process.stdin.read()) !== null) {
wstream.write(chunk); // Write the binary data to file
console.log("Writing chunk to file...");
}
});
process.stdin.on('end', () => {
// Close the file
wstream.end();
});
However, now the problem is this: when piping the output of ffmpeg to this script, how can I tell when one JPG file ends and another one begins?
ffmpeg command:
ffmpeg -vcodec h264_mmal -i "[my rtsp stream]" -r 1 -q:v 2 -f singlejpeg - | node pipe.js test_output.jpg
The test_output.jpg file continues to grow as long as the script runs. How can I instead know when the data for one jpg is complete and another one has started?
According to this, jpeg files always start with FF D8 FF and end with FF D9, so I guess I can check for this ending signature and start a new file at that point... any other suggestions?
I have a PGP encrypted file called file.pgp which must not be in ascii-armor but binary. It looks like this:
�P��3E��Q� �i`p���
����&�9
�ֻ�<P�+�[����R0��$���q����VJ��hu���bE"2��M1r��j�K�v�#6�3E�Ҳ�A�W{Z
��FEԭ�YV��6g�V���e�,I�Zpw�r��8׆
�mc��h��n���k�p�>JH\�G�6��M1|>�G�fl�J���6��
ج��
�_��y8�..{���_⮵���F���~�vt
�8AB;z����m^��Xp���VӅCzD�ճn
����{+d�3�"��N�1p�
When I'm using GNU base64 encoder, I'm getting this string:
$ cat file.gpg | base64
hQEMA1DujfGcM0WiAQgAvcIMUfydsSDmaWBwnoWACrsapePpJpU5Co68276SK2XVBqY2YyNUgzAF
oawkpMjfcQS+7+nJVkrb7Gh1h4L9YkUiMo+dTTFyzs5qskuECNZ25UA2rzNF+NKyq0HZV3sXWg3P
AwZNZbNJIAc4xWlBNfsNoda7zhk8UJArj1sAiKPw5VIKjahGRdSt2FlWurs2Z5EXVriLG0aHZbAs
SeCjWnB3Aalyoo8414aGbWOr5WjU7rpugBLw52uAcJgcPkpIXMJjCEf4gTbc1k0xfD4YjUejZmyH
H0rYAAHw3DbjyQrYrLmHC9Vfm655HBU40xceLi5/e4n2Dxge+F/irrW9o9JGAfCf5OZ+gXZ0Ggv9
t620m704QUI7eryy0ddtXoGsWHCxu4gaVtOFQ3pEp9WzZghuC5j1/c57K2T4lzP+IvEfo07fMRFw
tg==
With the GNU base64 tool, I can successfully reconvert it to the originating pgp-file and decrypt it.
I want to implement a similar tool in NodeJS. I can successfully convert ASCII text but not binary content. My provisional code looks like this:
var stdin = process.openStdin();
var data = "";
stdin.on('data', function(chunk) {
data += chunk;
});
stdin.on('end', function() {
console.log(new Buffer(text, 'binary').toString('base64'));
});
Usage: $ cat file.gpg | node base64.js
The output looks different to what GNU base64 offers. Also I can't convert it back to the original file.gpg file - GnuPG can't find anything to decrypt.
This happens because you pass a string and not a buffer as theGleep point in its comment.
You can do it like this:
let stdin = process.openStdin();
let data = [];
stdin.on('data', chunk => {
data.push(chunk);
});
stdin.on('end', () => {
console.log(Buffer.concat(data).toString('base64'));
});
There is apparently no easy way to stream images in Raspberry Pi. While there are many hacks available, in my Raspberry Pi Zero it has some trouble keeping a decent framerate.
I suspect one of the main problems is that the 1st Google solution and most of them writes/reads to the SD for each image. I've got so far to read from the terminal an image without touching the SD:
const out = await exec(`fswebcam -r 640x480 -`);
const img = out[0];
console.log(img);
This gives me this on the terminal:
����JFIF``��>CREATOR: gd-jpeg v1.0 (using IJG JPEG v80), default quality
��
$.' ",#(7),01444'9=82<.342��C
And many more. Previously I was doing something similar with buffers:
const file = fs.readFileSync(temp);
console.log(file.toString('base64'));
ctx.socket.emit('frame', { image: true, buffer: file.toString('base64') });
Where file is a Buffer and file.toString('base64') is a string in the form of:
/9j/4AAQSkZJRgABAQEAYABgAAD//gA8Q1JFQVRPUjogZ2QtanBlZyB2MS4wICh1c2luZyBJSkcgSlBFRyB2ODApLCBxdWFsaXR5ID0gMTAwCv ...
And this worked (but through the SD card). So my question is, what is the format of the first output in terminal? And how can I convert it to a Buffer or a String similar to the latter.
I ended up just using the terminal through pipe to convert it to base64:
fswebcam -r 640x480 - | base64
So now my whole snippet is:
// Take a picture in an async way and return it as a base64 encoded string
// Props: https://scottlinux.com/2012/09/01/encode-or-decode-base64-from-the-command-line/
module.exports = async ({ resolution = '640x480', rotate = 0 } = {}) => {
const query = `fswebcam -r ${resolution} --rotate ${rotate} - | base64`;
const out = await exec(query, { maxBuffer: 1024 * 1024 });
return out[0];
};