I'm trying to programmatically download a video file from a web-server through a link.
If you'd click that link through a web-browser it would just prompt you to download the video and to provide a name for the file and then download the video properly.
I have some nodejs code that just makes an HTTP request to that link and successfully gets raw data from it and saves it to a default file video.mp4
const https = require('https');
const fs = require('fs');
https.get('https://url.tocdn.com/myvideoid', (resp) => {
let data = '';
// A chunk of data has been recieved.
resp.on('data', (chunk) => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
fs.writeFile('./video.mp4', data, (err) => console.log(err))
});
}).on("error", (err) => {
console.log("Error: " + err.message);
});
The problem is that when I try to play that file through Windows Media Player, for example, it just shows an error regarding the file format. Am I missing something obvious?
Don't use string if you're pretending to store bytes,
or application/octet-stream in my case (which is the same)
From the given snippet change:
resp.on('data', (chunk) => {
data = Buffer.concat([data, Buffer.from(chunk)]);
});
And make sure to initialize data as follows:
let data = Buffer.from([]);
Related
I have an endpoint in node/express which does something to this effect
https.get(EXTERNAL_ENDPOINT, response =>
{
const data : ArrayBuffer[] = [];
response.on("data", chunk =>
{
data.push(chunk);
});
response.on("end", () =>
{
res.setHeader("Content-type", "application/octet-stream");
return res.status(200).send(Buffer.from(data.toString(), "base64"));
});
});
I am doing this to redirect a binary data stream I receive from an external endpoint to my client.
How do I receive this binary data on the client side as a downloadable file? It can be a pdf that can be downloaded or something like an image which has to be displayed in the browser on the website (as part of the website).
I'm making a GET request to an API that responds with Transfer-Encoding: chunked.
It's an https request, I've attached on('data') listener to it which then adds the response to an array.
There are around 5 chunks incoming each of them being a Buffer depending on the response. When I concat the array and try converting it to a String I get the weirdly decoded reply which looks to be encoded. I think it might be due to the request being HTTPS but I'm not sure, below's my code for decoding it
response.on('data', (data) => {
// Data is sent in binary chunks, let's add them up to an array
binaryDataArray.push(data)
}).on('end', () => {
const buffer = Buffer.concat(binaryDataArray);
try{
const formattedData = JSON.parse(buffer.toString('utf8'));
resolveRequest(formattedData);
}catch(error){
setTimeout(makeRequest, 5000);
}
});
Any help would be appreciated.
Check your res.headers['content-encoding'], the response may be encoded by gzip.
So you need to unzip the response buffer like this:
response.on('data', (data) => {
// Data is sent in binary chunks, let's add them up to an array
binaryDataArray.push(data)
}).on('end', () => {
const buffer = Buffer.concat(binaryDataArray);
zlib.gunzip(Buffer.concat(respData), function (err, decoded) {
if (err) throw err;
const formattedData = decoded.toString()
resolveRequest(formattedData)
});
});
I want to get binary from image to rotate then, using sharp.rotate();
I try to do this content += chunk; but dosent work.
let Client = require('ftp');
let fs = require('fs');
let sharp = require('sharp');
let path = 'users/'+userId+'/headerImage/header';
let Ftp = new Client();//create new istance of Ftp
//Start. Here we get image from server
await Ftp.on('ready', function(){
Ftp.get(path, async function(err, stream){
if(err){
res.status(400).send(err);
};
var content = '';
await stream.on('data', async (chunk) => {
content += chunk;
});
await stream.on('end', async function(){
console.log(content);
let image = await sharp(content);
await image
.rotate(90)
.toBuffer()
.then(async data => {
console.log(data);
})
.catch(error => {
console.log(error);
});
Ftp.end();
});
});
});
await Ftp.connect({
host: fileTransferProtocol.host,
port: fileTransferProtocol.port,
user: fileTransferProtocol.user,
password: fileTransferProtocol.pass
});
console: Error: [Error: Input file is missing]
I believe the problem you are having is that you are not handling the incoming data as a buffer. The stream variable inside the Ftp.get callback is of type ReadableStream. By default, stream data will be returned as Buffer objects unless you specify an encoding for the data, using the readable.setEncoding() method.
For your specific purpose, you want to handle the data as a Buffer object, since that is what the sharp function is expecting. To store the incoming data into a Buffer modify what happens on the data event.
var content = new Buffer(0);
stream.on("data", async chunk => {
content = Buffer.concat([content, chunk]);
});
Also, I don't think you are using async/await duly. The ftp module runs with callbacks and events, not promises. Appending those functions with await won't make them run synchronously.
Please check the following link to find more information about this feature:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
If you want to us async/await to handle your ftp requests try this module:
https://www.npmjs.com/package/promise-ftp
It provides an asynchronous interface for communicating with an FTP server.
On my node express server, I am receiving a pdf file. I am using the below code to get the pdf contents from the request
var data = new Buffer('');
request.on('data', function (chunk) {
data = Buffer.concat([data, chunk]);
});
request.on('end', function() {
console.log('PDF data is '+JSON.stringify(data));
});
Now that PDF content is available on node, I need to send it as it is to a J2EE server. In order to do that, I am first saving the PDF file in the node server, reading it from the node server and then piping it to request.post (https://github.com/request/request)
var req = require('request');
fs.writeFile('abc.pdf', data, 'binary', function(err) {
if (err) {
console.log('Error ' + JSON.stringify(err) );
throw err;
}
var source = fs.createReadStream('abc.pdf');
//send our data via POST request
source.pipe(req.post('http://'+j2ee_host+':'+j2ee_port+'/myjavaapp/Upload')
});
This works fine. However, I feel the part of saving the PDF file on the node server and then reading it is (before posting to the J2EE server using request module) is completely unnecessary, as I am not making any changes to the file.
Once I have the PDF contents in 'data' variable, I would like to directly post them to the J2EE server. However, I have not been able to find a way to use the request module to directly post file contents. I have seen some examples related to POST using request module but they refer to formData. In my case, I don't have formData but instead reading the file from request and directly posting it to the J2EE server.
Is there a way to achieve this and avoid the file write and read?
EDIT
Below is my complete code
function upload(request, response) {
var data = new Buffer('');
request.on('data', function (chunk) {
data = Buffer.concat([data, chunk]);
});
request.on('end', function () {
fs.writeFile('abc.pdf', data, 'binary', function(err){
if (err) {
console.log('Error ' + JSON.stringify(err) );
throw err;
}
var source = fs.createReadStream('abc.pdf');
source.pipe(req.post('http://'+j2ee_host+':'+j2ee_port+'/myj2eeapp/Upload'));
})
})
}
You can pipe directly from the data request to the servlet
var req = require('request');
function upload(request, response) {
var target = req.post('http://'+j2ee_host+':'+j2ee_port+'/myjavaapp/Upload');
request.pipe(target);
target.on('finish', function () {
console.log('All done!');
//send the response or make a completed callback here...
});
}
I am still a beginner in Node.js and I am trying to explore as much as I can.
I know that Express.js is a framework used by many people for creating websites in Node.js.
But without using Express.js, I know that it is it possible to read .html files using 'fs.readFile', and then "display" this .html file in the browser.
Is there a way to get user input (say a button click, or fill in a box) from this web page into Node.js? So far, I have not found any examples of this.
Yes, this is possible. Study how the connect bodyParser's urlencoded function works.
When a request comes in from the browser, node is going to represent this as a readable data stream. For web forms, the pattern will be:
Use the request's data and end events to buffer the chunks of data from the stream into a single string.
Parse that string appropriately given its data format. In the case of a web form, this will normally urlencoded (application/x-www-form-urlencoded) MIME type
.
var qs = require('qs'); //https://github.com/visionmedia/node-querystring
function handle(req, res) {
var buf = '';
req.setEncoding('utf8');
req.on('data', function(chunk){
//assemble the request from distinct chunks into a single string
buf += chunk
});
req.on('end', function(){
//OK, you have a usable string request body, parse it and handle it
try {
var formData = qs.parse(buf);
//Yay, it parsed. Now you have your form data
//depending on your form's html, you might have formData.email, for example
} catch (err){
//oops, respond with an error
}
});
}
Tutorial
Long story short:
http.createServer(function (req, res) {
var data = '';
req.on('data', function(chunk) {
console.log("Received body data:");
console.log(chunk);
data += chunk.toString();
});
req.on('end', function() {
console.log('Received Data: ', data);
res.end();
});
}