OSX 10.12.6
node v12.2.0
gzip 1.10
I gzipped some plaintext and I'm trying to read it
fs = require('fs')
zlib = require('zlib')
fs.createReadStream(filepath, {'encoding': 'UTF-8'})
.pipe(zlib.createGunzip()) // createUnzip behaves similarly.
.pipe(somethingelse())
.on('finish', function(){
console.log("finished reading");
});
This shows
Thrown:
Error: incorrect header check
errno -3
I hadn't realized that setting the UTF-8 encoding begins parsing in a different way. Removing the {encoding: 'UTF-8'} lets the zlib step decompress correctly, and my next step can consume directly from the stream.
Related
I am trying to decompress a .gz file using zlib, one of the inbuilt library of NodeJS. But while decompressing it is throwing incorrect header check error. I am using following code to decompress.
import fs from 'fs';
import zlib from 'zlib';
const rStream = fs.createReadStream('./path-to-gz-file');
const wStream = fs.createWriteStream('./path-to-gz-file'.replace('.gz', '');
rStream
.pipe(zlib.createGunzip())
.on('error', err => { console.log(err); }
.pipe(wStream);
.on('error', err => { console.log(err); }
.pipe(wStream);
In one of the solution on internet, it was suggested to change the encoding of read and write stream to binary but that also doesn't work. I've also tried almost every solution of this issues that are available online, but nothing works.
If anyone have any further question please let me know, I will clarify as soon as possible.
PS: The same file when decompressed using gzip which is default compression library of linux it get extracted as expected by using the following command.
I'm using fs.writeFileSync(file, data[, options]) to save a file returned from http.get(options[, callback])
This works fine for text files but images, pdfs etc end up being corrupted. From the searching around that I've done, it's apparently because fs.writeFileSync(file, data[, options]) defaults to UTF-8
I've tried setting the encoding to 'binary', the mime-type and the extension to no avail. It feels like something really obvious that I'm overlooking, can anyone point me in the right direction?
Thank you in advance
Update
I'm running this through electron. I didn't think it was worth mentioning as electron is just running node, but I'm not a node or electron expert so I'm not sure
Create a Buffer from the image data and set its encoding to binary. Then pass that data into a stream.PassThrough and pipe that into a stream.Writable.
var fs = require('fs');
var stream = require('stream');
var imgStream = new stream.PassThrough();
imgStream.end(Buffer.from(data, 'binary'));
var wStream = fs.createWriteStream('./<dest>.<ext>');
imgStream.once('end', () => {
console.log('Image Written');
});
imgStream.once('error', (err) => {
console.log(err);
});
imgStream.pipe(wStream);
We have a ruby instance that sends a message to a node instance via rabbitmq (bunny and amqplib) like below:
{ :type => data, :data => msg }.to_bson.to_s
This seems to be going pretty well, but msg's are sometimes long and we are sending them across data centers. zlib would help a lot.
doing smth like this in the ruby sender:
encoded_data = Zlib::Deflate.deflate(msg).force_encoding(msg.encoding)
and then reading it inside node:
data = zlib.inflateSync(encoded_data)
returns
"\x9C" from ASCII-8BIT to UTF-8
Is what I'm trying to do possible?
I am not a Ruby dev, so I will write the Ruby part in more or less pseudo code.
Ruby code (run online at https://repl.it/BoRD/0)
require 'json'
require 'zlib'
car = {:make => "bmw", :year => "2003"}
car_str = car.to_json
puts "car_str", car_str
car_byte = Zlib::Deflate.deflate(car_str)
# If you try to `puts car_byte`, it will crash with the following error:
# "\x9C" from ASCII-8BIT to UTF-8
#(repl):14:in `puts'
#(repl):14:in `puts'
#(repl):14:in `initialize'
car_str_dec = Zlib::Inflate.inflate(car_byte)
puts "car_str_dec", car_str_dec
# You can check that the decoded message is the same as the source.
# somehow send `car_byte`, the encoded bytes to RabbitMQ.
Node code
var zlib = require('zlib');
// somehow get the message from RabbitMQ.
var data = '...';
zlib.inflate(data, function (err, buffer) {
if (err) {
// Handle the error.
} else {
// If source didn't have any encoding,
// no need to specify the encoding.
console.log(buffer.toString());
}
});
I also suggest you to stick with async functions in Node instead of their sync alternatives.
So I am working on the nodeschool.io stream-adventure tutorial track and I'm having trouble with the last problem. The instructions say:
An encrypted, gzipped tar file will be piped in on process.stdin. To beat this
challenge, for each file in the tar input, print a hex-encoded md5 hash of the
file contents followed by a single space followed by the filename, then a
newline.
You will receive the cipher name as process.argv[2] and the cipher passphrase as
process.argv[3]. You can pass these arguments directly through to
`crypto.createDecipher()`.
The built-in zlib library you get when you `require('zlib')` has a
`zlib.createGunzip()` that returns a stream for gunzipping.
The `tar` module from npm has a `tar.Parse()` function that emits `'entry'`
events for each file in the tar input. Each `entry` object is a readable stream
of the file contents from the archive and:
`entry.type` is the kind of file ('File', 'Directory', etc)
`entry.path` is the file path
Using the tar module looks like:
var tar = require('tar');
var parser = tar.Parse();
parser.on('entry', function (e) {
console.dir(e);
});
var fs = require('fs');
fs.createReadStream('file.tar').pipe(parser);
Use `crypto.createHash('md5', { encoding: 'hex' })` to generate a stream that
outputs a hex md5 hash for the content written to it.
This is my attempt so far to work on it:
var tar = require('tar');
var crypto = require('crypto');
var zlib = require('zlib');
var map = require('through2-map');
var cipherAlg = process.argv[2];
var passphrase = process.argv[3];
var cryptoStream = crypto.createDecipher(cipherAlg, passphrase);
var parser = tar.Parse(); //emits 'entry' events per file in tar input
var gunzip = zlib.createGunzip();
parser.on('entry', function(e) {
e.pipe(cryptoStream).pipe(map(function(chunk) {
console.log(chunk.toString());
}));
});
process.stdin
.pipe(gunzip)
.pipe(parser);
I know it's not complete yet, but my issue is that when I try to run this, the input never gets piped to the tar file parsing part. It seems to hang up on the piping to gunzip. This is my exact error:
events.js:72
throw er; // Unhandled 'error' event
^
Error: incorrect header check
at Zlib._binding.onerror (zlib.js:295:17)
I'm totally stumped because the node documentation for Zlib has no mention of headers except for when it has examples with the http/request modules. There are a number of other questions regarding this error with node, but most use buffers rather than streams, so I couldn't find a relevant answer to my problem. All help is greatly appreciated
I actually figured it out, I was supposed to decrypt the stream before unzipping it.
So instead of:
process.stdin
.pipe(gunzip)
.pipe(parser);
it should be:
process.stdin
.pipe(cryptoStream)
.pipe(gunzip)
.pipe(parser);
In short I'm trying to read a .zip file from my file system, deflate the zip-file and them stream it with xml-stream to do some things with the contents in the file.
I thought this would be fairly simple and started with this:
var fs = require('fs')
, XmlStream = require('xml-stream')
, zlib = require('zlib');
//- read the file and buffer it.
var path = '../path/to/some.zip';
var fileBuffer = fs.readFileSync(path, { encoding: 'utf8' });
//- use zlib to unzip it
zlib.gunzip(fileBuffer, function(err, buffer) {
if (!err) {
console.log(buffer.toString());
}
console.log(err);
});
But this results in a
{ [Error: incorrect header check] errno: -3, code: 'Z_DATA_ERROR' }
Changing the encoding or the method (.unzip, .gunzip or .inflate) isn't working either.
What am I missing here?
Gzip is not zip. They're different compression formats, just like RAR is. The error indicates that what you're trying to read is not a gzipped file.
You can use a different library, such as JSZip.
I'm using zlib.unzip instead zlib.gunzip