I simply want to download a .gz file from a URL and save it in a folder. I would like to do this without any third party libraries if possible. Here's what I have so far, but it only downloads an empty file:
const fs = require('fs')
const https = require('https')
let file = fs.createWriteStream('./folder/filename.gz')
let request = https.get('https://someurl/somefile.gz', function(res) {
res.pipe(file)
})
you can try this, using HTTP module for nodesJS,it looks similar to downloading any other file, just remember to mention the extension of the downloaded file when calling instead....Here is an example:
NOTE: IF you are trying to download from an HTTPS link, use the HTTPS
module instead, its exactly the same, but just replace all the
HTTP in the following code with HTTPS
const http = require('http');
const fs = require('fs');
//I added './' assuming that you want to download it where the server
//file is located, just change it to your desired path, followed by the
//filename and the EXTENSION
const file = fs.createWriteStream("./result.tar.gz");
const request = http.get("http://alpha.gnu.org/gnu/gzip/gzip-1.3.6.tar.gz", (response) => {
response.pipe(file);
});
Related
At my endpoint in my NodeJS server, after retrieving an audio file stored as a Buffer in MongoDB, I want to represent it with a URL (much like how you do with URL.createObjectURL(blob) in the frontend on the browser). I then plan to res.render() the URL in HTML through Handlebars on the client, so that the user can click on it to download it:
<a href={{url}}>Click me to download the file!</a>
In the NodeJs server, I have converted the MongoDB Buffer into a JavaScript ArrayBuffer through:
var buffer = Buffer.from(recordingFiles[0].blobFile);
var arrayBuffer = Uint8Array.from(buffer).buffer;
I am unsure where to proceed from here. I seen solutions using fs or res.download(), but they don't seem applicable to my situation. Thanks in advance for any help!
Hopefully this can help.
var blob = new Blob(BUFFER, {type: "audio mime type"});
var link = document.createElement('a');
link.href = window.URL.createObjectURL(blob);
var fileName = reportName;
link.download = fileName;
link.click();
Do you always need to preload the audio file onto the page?
If not, then I would advise you to add a separate endpoint to download the file on demand. The frontend link can send a get request to the endpoint and download the file only if the user clicked it.
Otherwise you'd always be downloading the buffer behind the scenes, even if the user didn't intend to download it. This is especially problematic on slow connections.
Frontend:
<a href={{`${baseUrl}/download/${audioId}`}}>Click me to download the file!</a>
Backend:
const stream = require('stream');
app.get('/download/:audioId', function (request, response) {
// Retrieve the tag from our URL path
const audioId = request.params.audioId;
const fileData; // TODO: Get file buffer from mongo.
const fileContents = Buffer.from(fileData, 'base64');
const readStream = new stream.PassThrough();
readStream.end(fileContents);
response.set('Content-disposition', 'attachment; filename=' + fileName);
response.set('Content-Type', '<your MIME type here>');
readStream.pipe(response);
});
A list of relevant MIME types can be found here.
I've set up a small node js BE app, built with express and fastCsv module on top of it. The desired outcome would be to be able to download a csv file to the client side, without storing it anywhere inside the server, since the data is generated depending on user criteria.
So far I've been able to get somewhere it it, Im using streams, since that csv file could be pretty large depending on the user selection. Im pretty sure something is missing inside the code bellow:
const fs = require('fs');
const fastCsv = require('fast-csv');
.....
(inside api request)
.....
router.get('/', async(req, res) => {
const gatheredData ...
const filename = 'sometest.csv'
res.writeHead(200, {
'Content-Type': 'text/csv',
'Content-Disposition': 'attachment; filename=' + filename
})
const csvDataStream = fastCsv.write(data, {headers: true}).pipe(res)
})
The above code 'works' in some way as it does deliver back the response, but not the actual file, but the contents of the csv file, which I can view in the preview tab as a response. To sum up, Im trying to stream in that data, into a csv and push it to download file to client, and not store it on the server. Any tips or pointers are very much appreciated.
Here's what worked for me after created a CSV file on the server using the fast-csv package. You need to specify the full, absolute directory path where the output CSV file was created:
const csv = require("fast-csv");
const csvDir = "abs/path/to/csv/dir";
const filename = "my-data.csv";
const csvOutput = `${csvDir}/${filename}`;
console.log(`csvOutput: ${csvOutput}`); // full path
/*
CREATE YOUR csvOutput FILE USING 'fast-csv' HERE
*/
res.type("text/csv");
res.header("Content-Disposition", `attachment; filename="${filename}"`);
res.header("Content-Type", "text/csv");
res.sendFile(filename, { root: csvDir });
You need to make sure to change the response content-type and headers to "text/csv", and try enclosing the filename=... part in double-quotes, like in the above example.
I have been trying to access a url with a CSV file to download it in a specific directory, using the Selenium Webdriver for Firefox(geckodriver), in a NodeJS enviroment on Linux-Mint.
This is my code:
const {Builder} = require('selenium-webdriver');
const firefox = require('selenium-webdriver/firefox');
const path = require('path');
const options = new firefox.Options();
options.setPreference('browser.download.dir', path.resolve(__dirname));
options.setPreference('browser.download.folderList', 2);
options.setPreference('browser.helperApps.neverAsk.saveToDisk', 'application/x-csv');
function example(){
let driver = new Builder().forBrowser('firefox').setFirefoxOptions(options).build();
driver.get('http://insight.dev.schoolwires.com/HelpAssets/C2Assets/C2Files/C2ImportCalEventSample.csv');
}
example();
As you can see, I am correctly setting the browser option to browser.helperApps.neverAsk.saveToDisk, so as to be able to bypass the dialog. However, I am still getting the dialog no matter what I do. I haven't tried this code on Windows, but for my purposes it needs to work on Linux.
Am I missing something? Some preference that needs to be added or changed? Or does this not work on my current enviroment?
Thank you in advance for any help provided.
If you are just downloading a file from link why do you need selenium?
A much simple approach will be just to get the file by http and save to file.
const http = require('http');
const fs = require('fs');
const file = fs.createWriteStream("C2ImportCalEventSample.csv");
const request = http.get("http://insight.dev.schoolwires.com/HelpAssets/C2Assets/C2Files/C2ImportCalEventSample.csv", function(response) {
response.pipe(file);
});
If you have to use selenium let me know in the comments and i will try to find a solution for your problem using selenium.
I'm using Unity's WebGL and I'm getting this message on the console "You can reduce your startup time if you configure your web server to host .unityweb files using gzip compression." So according to Unity's documentation, I need to add the correct response Headers https://docs.unity3d.com/Manual/webgl-deploying.html.
I found the "express-static-gzip" module, and I tried to do just that, but the warning is still there. Below is the server.
const express = require('express');
const ip = require("ip");
const expressStaticGzip = require('express-static-gzip');
const http = require('http');
const app = express();
const server = http.Server(app);
app.use('/public/Builds/Build/', expressStaticGzip('public/Builds/Build/', {
customCompressions: [{
encodingName: "gzip",
fileExtension: "unityweb"
}]
}));
// app.use(compression());
app.use(express.static('public'));
server.listen(3000, function(){
console.log( ":: http://" + ip.address() + "/ ::" );
});
Any ideas?
Nick
Many thanks to #d_shiv for his help. I changed the code to the following, and the warning went away.
(you can change gzip with br if you're using brotli)
const express = require('express');
const ip = require("ip");
const http = require('http');
const app = express();
const server = http.Server(app);
app.use(express.static('public', {
setHeaders: function(res, path) {
if(path.endsWith(".unityweb")){
res.set("Content-Encoding", "gzip");
}
}
}));
server.listen(3000, function(){
console.log( ":: http://" + ip.address() + ":3000/ ::" );
});
express-static-gzip does not gzip the files on the fly before serving it. It assumes that you have the normal as well as gzipped versions of the file available on the specified directory. Check the Examples section of documentation here.
In this scenario, if the public/Builds/Build/Builds.wasm.framework.unityweb had to be transferred with gzip compression, you'd need to create a gzipped version by name of public/Builds/Build/Builds.wasm.framework.unityweb.gz. The middleware will automatically scan the folder for all such file pairs where original as well as gzipped versions are available. It will serve the gzipped version when request comes for original file, if the browser supports it.
The customCompressions array should also be skipped since that's enabled by default. The middleware would be registered, something like this:
app.use('/Builds/Build/', expressStaticGzip('public/Builds/Build/'));
Also note that public/ is removed from the middleware path (should be present in the expressStaticGzip path though). This is because your assets are being loaded from path https://{hostname}/Builds/Build/....
If you intend to compress the files on the fly and server it, take a look at compression module. The can be very costly operation for your server though, if possible do the gzipping during build time to create the equivalent .gz files, and continue to use express-static-gzip.
Im wondering if is posible to know what is the file name of an incomming binary request.
This is my situation I have this code that handles the file upload
router.route('/:filename')
.put(function(req,res){
var uuid = guid();
var fileExtension = req.params.filename.substring(req.params.filename.lastIndexOf("."));
if(!fs.existsSync('../files')){
fs.mkdirSync('../files')
}
var newFile = fs.createWriteStream('../files/'+uuid+fileExtension);
req.pipe(newFile);
req.on('end',function(end){
console.log("Finished")
res.send(uuid+fileExtension)
})
})
as you can see now ,I need the file name specified in the URL('/:filename'). My question is: If it is possible to take that attribute from the resquest stream, instead the url or a form key?
If you use multer middleware you can access the uploaded filename like so
var multer = require('multer')
var upload = multer()
router.route('/:filename')
.put(upload.single('fileField'),function(req,res){
var fileName = req.file.originalname
var uuid = guid();
var fileExtension = req.params.filename.substring(req.params.filename.lastIndexOf("."));
if(!fs.existsSync('../files')){
fs.mkdirSync('../files')
}
var newFile = fs.createWriteStream('../files/'+uuid+fileExtension);
req.pipe(newFile);
req.on('end',function(end){
console.log("Finished")
res.send(uuid+fileExtension)
})
})
You'll need to inspect the Content-Disposition header of the request and parse the file name information out if processing the HTTP request manually.
However, I'd recommend you look at some of the existing file upload middlewares, no point in reinventing the wheel
busboy
multer
formidable
multiparty
pez