I'm receiving a Buffer data from a response with data in a file.
// Save it just to check if content is correct
const fd = fs.openSync('data.zip', 'w')
fs.writeSync(fd, data)
fs.closeSync(fd)
Produces file data.zip, which contains file foo.csv.
I can unzip it with UZIP in-memory with:
const unzipArray = UZIP.parse(data)['foo.csv']
However, I cannot do it with Zlib.
const unzipArray = zlib.unzipSync(data)
// Rises: incorrect header check
It looks like Zlib cannot parse the file structure.
How to unzip the above buffer in-memory with Zlib, without saving files to the filesystem?
You have a zip file, not a zlib or gzip stream. As you found, zlib doesn't process zip files. There are many solutions out there for node.js, which you can find using your friend google. Here is one.
for single file:
const fs = require('fs');
const zlib = require('zlib');
const fileContents = fs.createReadStream('./data/file1.txt.gz');
const writeStream = fs.createWriteStream('./data/file1.txt');
const unzip = zlib.createGunzip();
fileContents.pipe(unzip).pipe(writeStream);
for a group of file:
const fs = require('fs');
const zlib = require('zlib');
const directoryFiles = fs.readdirSync('./data');
directoryFiles.forEach(filename => {
const fileContents = fs.createReadStream(`./data/${filename}`);
const writeStream = fs.createWriteStream(`./data/${filename.slice(0, -3)}`);
const unzip = zlib.createGunzip();
fileContents.pipe(unzip).pipe(writeStream);
});
Related
I'm getting a ZIP archive from S3 using the aws s3 node SDK.
In this zip file there is a single .json file where I want to get the contents from. I don't want to save this file to storage, but only get the contents of this zip file.
Example:
File.zip contains a single file:
file.json with contents({"value":"abcd"})
I currently have:
const { S3Client, GetObjectCommand} = require("#aws-sdk/client-s3");
const s3Client = new S3Client({ region: 'eu-central-1'});
const file = await s3Client.send(new GetObjectCommand({Bucket:'MyBucket', Key:'file.zip'}));
file.body now contains a Readable stream with the contents of the zip file. I now want to transfer this Readable stream into {"value":"abcd"}
Is there a library or piece of code that can help me do this and produce the result without having to save the file to disk?
You could use the package archiver or zlib (zlib is integrated in nodejs)
a snippet from part of my project looks like this:
import { unzipSync } from 'zlib';
// Fetch data and get buffer
const res = await fetch('url')
const zipBuffer = await res.arrayBuffer()
// Unzip data and convert to utf8
const unzipedBuffer = await unzipSync(zipBuffer)
const fileData = unzipedBuffer.toString('utf8')
Now fileData is the content of your zipped file as a string, you can use JSON.parse(fileData) to get the content as a json and work with it
How can I convert a string to a buffer?
I tried: Uint8Array.from('hello world') but it isn't working
The equivalent of Buffer.from('Hello World') is:
const encoder = new TextEncoder()
const buffer = encoder.encode('Hello World');
If you want to decode it back, you'll need to use TextDecoder.
const decoder = new TextDecoder()
console.log(decoder.decode(buffer))
Deno tries to implement Web APIs when possible, reason why it works the same way on the browser.
const decoder = new TextDecoder();
const encoder = new TextEncoder();
const buffer = encoder.encode('Hello World');
console.log(buffer);
console.log(decoder.decode(buffer))
Have in mind that Node.js' Buffer supports multiple encodings, such as base64 or hex, which won't work with TextDecoder
So if you have a base64 string and want to convert it to utf8 instead of doing:
const base64String = Buffer.from('Hello World').toString('base64'); // Hello World
const utf8String = Buffer.from(base64String, 'base64').toString();
You would need to use atob (Same as Web API) instead:
const base64String = btoa('Hello World');
const utf8String = atob(base64String);
console.log('Base64:', base64String);
console.log('utf8string:', utf8String);
I simply want to download a .gz file from a URL and save it in a folder. I would like to do this without any third party libraries if possible. Here's what I have so far, but it only downloads an empty file:
const fs = require('fs')
const https = require('https')
let file = fs.createWriteStream('./folder/filename.gz')
let request = https.get('https://someurl/somefile.gz', function(res) {
res.pipe(file)
})
you can try this, using HTTP module for nodesJS,it looks similar to downloading any other file, just remember to mention the extension of the downloaded file when calling instead....Here is an example:
NOTE: IF you are trying to download from an HTTPS link, use the HTTPS
module instead, its exactly the same, but just replace all the
HTTP in the following code with HTTPS
const http = require('http');
const fs = require('fs');
//I added './' assuming that you want to download it where the server
//file is located, just change it to your desired path, followed by the
//filename and the EXTENSION
const file = fs.createWriteStream("./result.tar.gz");
const request = http.get("http://alpha.gnu.org/gnu/gzip/gzip-1.3.6.tar.gz", (response) => {
response.pipe(file);
});
multiple zip File read and display in angularjs and those files convert base64 in nodejs and push into gitlab. please suggest me if it possible in nodejs. is there any blug available for reference.
use fs module of nodejs to read the files from directory
const testFolder = './tests/';
const fs = require('fs');
fs.readdirSync(testFolder).forEach(file => {
console.log(file);
});
once you get the files you can covert to base64
function base64_encode(file) {
// read binary data
var bitmap = fs.readFileSync(file);
// convert binary data to base64 encoded string
return new Buffer(bitmap).toString('base64');
}
Im wondering if is posible to know what is the file name of an incomming binary request.
This is my situation I have this code that handles the file upload
router.route('/:filename')
.put(function(req,res){
var uuid = guid();
var fileExtension = req.params.filename.substring(req.params.filename.lastIndexOf("."));
if(!fs.existsSync('../files')){
fs.mkdirSync('../files')
}
var newFile = fs.createWriteStream('../files/'+uuid+fileExtension);
req.pipe(newFile);
req.on('end',function(end){
console.log("Finished")
res.send(uuid+fileExtension)
})
})
as you can see now ,I need the file name specified in the URL('/:filename'). My question is: If it is possible to take that attribute from the resquest stream, instead the url or a form key?
If you use multer middleware you can access the uploaded filename like so
var multer = require('multer')
var upload = multer()
router.route('/:filename')
.put(upload.single('fileField'),function(req,res){
var fileName = req.file.originalname
var uuid = guid();
var fileExtension = req.params.filename.substring(req.params.filename.lastIndexOf("."));
if(!fs.existsSync('../files')){
fs.mkdirSync('../files')
}
var newFile = fs.createWriteStream('../files/'+uuid+fileExtension);
req.pipe(newFile);
req.on('end',function(end){
console.log("Finished")
res.send(uuid+fileExtension)
})
})
You'll need to inspect the Content-Disposition header of the request and parse the file name information out if processing the HTTP request manually.
However, I'd recommend you look at some of the existing file upload middlewares, no point in reinventing the wheel
busboy
multer
formidable
multiparty
pez