so my question is on Node js piping. So my backend looks like this -- there is a simple route, the route calls function and passes to it a file path for an executable type file. This file is then run with the childProcess.spawn and there is a data output that I can console.log
const express = require("express");
const app = express();
etc...
const runExecutable = (executableFile) => {
const runFile = childProcess.spawn(executableFile);
runFile.stdout.on('data', function(data){
console.log("DATA", data);
})
runFile.on('exit', function(code, signal){
[some code here]
})
}
app.get('/example', (req, res) => {
var file = "./testFile.exe";
runExecutable(file);
})
The question I have is how can I pipe this output of data/a.k.a chunks in real time to the client, it's important for them to get the data as it comes out and not for me to write it to a file and send them the whole thing. One more thing to note, the client is accessing my route through a curl curl 123.45.678.901/example in their terminal and I want to pipe the data to their terminal.
On reading around, I know that for example the request module does a request.get(url).pipe(res) /[Express res] and so I'm wondering if this is similar to what I might need to be doing.
Thanks all!
Found the answer: Any stream can be piped - readable.pipe(destination[, options]) - childProcess.spawn(executableFile) is not a stream, but once the file starts being executed it does emit a "data" event which is another way of saying there is a stream being emitted from the running of the file. So if you are looking at these chunks of "data" coming out - like I am - like this:
runFile.stdout.on('data', function(data){
console.log("DATA", data);
})
then that's the stream that you use and that's the where you attach the pipe
Node documentation basically says - to the stream attach .pipe and then just send it to it's destination. Since I wanted to send these chunks of data to my client I also had to pass res around, so my code now looks like this:
const express = require("express");
const app = express();
etc...
const runExecutable = (executableFile, res) => {
const runFile = childProcess.spawn(executableFile);
runFile.stdout.on('data', function(data){
console.log("DATA", data);
}).pipe(res)
runFile.on('exit', function(code, signal){
[some code here]
})
}
app.get('/example', (req, res) => {
var file = "./testFile.exe";
runExecutable(file, res);
})
and it works! I hope this is helpful to others - Thanks for the help Lee!
Related
So essentially, what my api call does is it 1) takes in video data using parse multipart, 2) converts that video data to a real mp4 file using ffmpeg, and then 3) is supposed to send back the video data to the client in the response body.
Steps 1 and 2 work perfectly - it's that third step that I am stuck on.
The api call creates the Out.mp4 file, but when I try and read its info using createReadStream, the chunks array doesn't populate, and a null context.res body is returned.
Please let me know what I am doing wrong and how I can pass back the video info properly so as to be able to convert the video info back to a playable mp4 file on the client's side.
Also, lmk if you have any questions or things I can clarify.
Here is the api call index.js file
const fs = require("fs");
module.exports=async function(context, req){
try{
//Get the input file setup
context.log("Javascript HTTP trigger function processed a request.");
var bodyBuffer=Buffer.from(req.body);
var boundary=multipart.getBoundary(req.headers['content-type']);
var parts=multipart.Parse(bodyBuffer, boundary);
var temp = "C:/home/site/wwwroot/In.mp4";
fs.writeFileSync(temp, Buffer(parts[0].data));
//Actually execute the ffmpeg script
var execLineBuilder= "C:/home/site/wwwroot/ffmpeg-5.1.2-essentials_build/bin/ffmpeg.exe -i C:/home/site/wwwroot/In.mp4 C:/home/site/wwwroot/Out.mp4"
var execSync = require('child_process').execSync;
//Executing the script
execSync(execLineBuilder)
//EVERYTHING WORKS UP UNTIL HERE (chunks array seems to be empty, even though outputting chunk to a file populates
//That file with data)
//Storing the chunks of the output mp4 into chunks array
execSync.on('exit', ()=>{
chunks = [];
const myPromise = new Promise((resolve, reject) => {
var readStream = fs.createReadStream("C:/home/site/wwwroot/Out.mp4");
readStream.on('data', (chunk)=> {
chunks.push(chunk);
resolve("foo");
});
})
})
myPromise.then(()=>{
context.res={
status:200,
body:chunks
}
})
}catch (e){
context.res={
status:500,
body:e
}
}
}```
you can use an npm package called azure-function-express this package will basically convert your azure function to an express
This way you can directly read the mp3 file you saved and send it directly.
const createHandler = require("azure-function-express").createHandler;
const express = require("express");
const fs = require('fs');
const app = express();
app.get("/api/HttpTrigger1", (req, res) => {
res.writeHead(200, {'Content-Type': 'video/mp4'});
let open = fs.createReadStream('./test.mp3');
res.send(open);
});
This way you will be able to share the video also running the ffmpeg might also be simple
I need to serve local files from a different server using node. The api endpoint are being handled by express.
The goal is not to contain the entire file in memory instead stream the data so it shows the output to the enduser progressively.
By reading the stream api documentation i came up with this solution with a combination with expressjs response. Here is the example:
const open = (req, res) => {
const formattedUrl = new url.URL(
"https://dl.bdebooks.com/Old%20Bangla%20Books/Harano%20Graher%20Jantra%20Manob%20-%20Shaktimoy%20Biswas.pdf"
);
const src = fs.createReadStream(formattedUrl);
return src.pipe(res);
};
But when i hit this express endpoint http://localhost:3000/open it throws following error:
TypeError [ERR_INVALID_URL_SCHEME]: The URL must be of scheme file
I would like to display the file content inline! What I am doing wrong? Any suggestions will be greatly appreciated. :)
fs.createReadStream() operates on the file system. It does not accept an http or https URL. Instead, you need to use something like http.get() to make an http request and then return a readable stream that you can then pipe from.
const open = (req, res) => {
const formattedUrl = new url.URL("https://dl.bdebooks.com/Old%20Bangla%20Books/Harano%20Graher%20Jantra%20Manob%20-%20Shaktimoy%20Biswas.pdf");
http.get(formattedUrl, (stream) => {
stream.pipe(res);
}).on('error', (err) => {
// send some sort of error response here
});
};
I am working on NodeJs/Express project and need to show in the browesr pdf file that is stored in
/public/images
Here is relevant router code:
router.post('/show_file', async (req,res)=>{
try {
let path = './public/images/1.pdf'
var data =fs.readFileSync(path);
res.contentType("application/pdf");
res.send(data);
} catch (err) {
res.status(500)
console.log(err)
res.send(err.message)
}
})
I don't get any errors but nothing is happening ie.browser is not opening etc.
Thanks in advance for any guidance.
The first change that I would do is to remove the async. It will just mess out the code with unneeded Promises.
Second, I removed the need to catch the exception, verifying the existence of the file with fs.existsSync(path). Try to not to rise exceptions as often as possible. If you know something can rise an exception, test it.
Last, and most important, I created a reading stream of the file and piped the result to the response with fs.createReadStream(path).pipe(res). This way, the client recieves the file as it is read and your memory is spared. Great for large files.
Reading a file can be memory intensive, so loading it all in memory is a bad practice. You just need a handfull of request to overload your machine.
You can read more on the pipe method here.
In this example, any GET call to /router/show_file will return the pdf.
const express = require('express')
const app = express()
const fs = require('fs')
const router = express.Router()
router.get('/show_file', (req, res) => {
const path = './public/images/1.pdf'
if (fs.existsSync(path)) {
res.contentType("application/pdf");
fs.createReadStream(path).pipe(res)
} else {
res.status(500)
console.log('File not found')
res.send('File not found')
}
})
app.use('/router', router) // Here we pass the router to the app with a path
app.listen(9999, () => console.log('Listening to port 9999'))
I am currently working on parsing a remote csv product feed from a Node app and would like to use Papa Parse to do that (as I have had success with it in the browser in the past).
Papa Parse Github: https://github.com/mholt/PapaParse
My initial attempts and web searching haven't turned up exactly how this would be done. The Papa readme says that Papa Parse is now compatible with Node and as such Baby Parse (which used to serve some of the Node parsing functionality) has been depreciated.
Here's a link to the Node section of the docs for anyone stumbling on this issue in the future: https://github.com/mholt/PapaParse#papa-parse-for-node
From that doc paragraph it looks like Papa Parse in Node can parse a readable stream instead of a File. My question is;
Is there any way to utilize Readable Streams functionality to use Papa to download / parse a remote CSV in Node some what similar to how Papa in the browser uses XMLHttpRequest to accomplish that same goal?
For Future Visibility
For those searching on the topic (and to avoid repeating a similar question) attempting to utilize the remote file parsing functionality described here: http://papaparse.com/docs#remote-files will result in the following error in your console:
"Unhandled rejection ReferenceError: XMLHttpRequest is not defined"
I have opened an issue on the official repository and will update this Question as I learn more about the problems that need to be solved.
After lots of tinkering I finally got a working example of this using asynchronous streams and with no additional libraries (except fs/request). It works for remote and local files.
I needed to create a data stream, as well as a PapaParse stream (using papa.NODE_STREAM_INPUT as the first argument to papa.parse()), then pipe the data into the PapaParse stream. Event listeners need to be implemented for the data and finish events on the PapaParse stream. You can then use the parsed data inside your handler for the finish event.
See the example below:
const papa = require("papaparse");
const request = require("request");
const options = {/* options */};
const dataStream = request.get("https://example.com/myfile.csv");
const parseStream = papa.parse(papa.NODE_STREAM_INPUT, options);
dataStream.pipe(parseStream);
let data = [];
parseStream.on("data", chunk => {
data.push(chunk);
});
parseStream.on("finish", () => {
console.log(data);
console.log(data.length);
});
The data event for the parseStream happens to run once for each row in the CSV (though I'm not sure this behaviour is guaranteed). Hope this helps someone!
To use a local file instead of a remote file, you can do the same thing except the dataStream would be created using fs:
const dataStream = fs.createReadStream("./myfile.csv");
(You may want to use path.join and __dirname to specify a path relative to where the file is located rather than relative to where it was run)
OK, so I think I have an answer to this. But I guess only time will tell. Note that my file is .txt with tab delimiters.
var fs = require('fs');
var Papa = require('papaparse');
var file = './rawData/myfile.txt';
// When the file is a local file when need to convert to a file Obj.
// This step may not be necissary when uploading via UI
var content = fs.readFileSync(file, "utf8");
var rows;
Papa.parse(content, {
header: false,
delimiter: "\t",
complete: function(results) {
//console.log("Finished:", results.data);
rows = results.data;
}
});
Actually you could use a lightweight stream transformation library called scramjet - parsing CSV straight from http stream is one of my main examples. It also uses PapaParse to parse CSVs.
All you wrote above, with any transforms in between, can be done in just couple lines:
const {StringStream} = require("scramjet");
const request = require("request");
request.get("https://srv.example.com/main.csv") // fetch csv
.pipe(new StringStream()) // pass to stream
.CSVParse() // parse into objects
.consume(object => console.log("Row:", object)) // do whatever you like with the objects
.then(() => console.log("all done"))
In your own example you're saving the file to disk, which is not necessary even with PapaParse.
I am adding this answer (and will update it as I progress) in case anyone else is still looking into this.
It seems like previous users have ended up downloading the file first and then processing it. This SHOULD NOT be necessary since Papa Parse should be able to process a read stream and it should be possible to pipe 'http' GET to that stream.
Here is one instance of someone discussing what I am trying to do and falling back to downloading the file and then parsing it: https://forums.meteor.com/t/processing-large-csvs-in-meteor-js-with-papaparse/32705/4
Note: in the above Baby Parse is discussed, now that Papa Parse works with Node Baby Parse has been depreciated.
Download File Workaround
While downloading and then Parsing with Papa Parse is not an answer to my question, it is the only workaround I have as of now and someone else may want to use this methodology.
My code to download and then parse currently looks something like this:
// Papa Parse for parsing CSV Files
var Papa = require('papaparse');
// HTTP and FS to enable Papa parse to download remote CSVs via node streams.
var http = require('http');
var fs = require('fs');
var destinationFile = "yourdestination.csv";
var download = function(url, dest, cb) {
var file = fs.createWriteStream(dest);
var request = http.get(url, function(response) {
response.pipe(file);
file.on('finish', function() {
file.close(cb); // close() is async, call cb after close completes.
});
}).on('error', function(err) { // Handle errors
fs.unlink(dest); // Delete the file async. (But we don't check the result)
if (cb) cb(err.message);
});
};
download(feedURL, destinationFile, parseMe);
var parseMe = Papa.parse(destinationFile, {
header: true,
dynamicTyping: true,
step: function(row) {
console.log("Row:", row.data);
},
complete: function() {
console.log("All done!");
}
});
Http(s) actually has a readable stream as parameter in the callback, so here is a simple solution
try {
var streamHttp = await new Promise((resolve, reject) =>
https.get("https://example.com/yourcsv.csv", (res) => {
resolve(res);
})
);
} catch (e) {
console.log(e);
}
Papa.parse(streamHttp, config);
const Papa = require("papaparse");
const { StringStream } = require("scramjet");
const request = require("request");
const req = request
.get("https://example.com/yourcsv.csv")
.pipe(new StringStream());
Papa.parse(req, {
header: true,
complete: (result) => {
console.log(result);
},
});
David Liao's solution worked for me, I did tweak it a little bit since I am using local file. He did not include the example how to solve the file access in node if you did get Error: ENOENT: no such file or directory message in your console.
To test your actual working directory and to understand where you must point your path to console log the following, this gave me better understanding of the file location: console.log(process.cwd()).
const fs = require('fs');
const papa = require('papaparse');
const request = require('request');
const path = require('path');
const options = {
/* options */
};
const fileName = path.resolve(__dirname, 'ADD YOUR ABSOLUTE FILE LOCATION HERE');
const dataStream = fs.createReadStream(fileName);
const parseStream = papa.parse(papa.NODE_STREAM_INPUT, options);
dataStream.pipe(parseStream);
let data = [];
parseStream.on('data', chunk => {
data.push(chunk);
});
parseStream.on('finish', () => {
console.log(data);
console.log(data.length);
});
I'm trying to POST a raw body with restify. I have the receive side correct, when using POSTman I can send a raw zip file, and the file is correctly created on the server's file system. However, I'm struggling to write my test in mocha. Here is the code I have, any help would be greatly appreciated.
I've tried this approach.
const should = require('should');
const restify = require('restify');
const fs = require('fs');
const port = 8080;
const url = 'http://localhost:' + port;
const client = restify.createJsonClient({
url: url,
version: '~1.0'
});
const testPath = 'test/assets/test.zip';
fs.existsSync(testPath).should.equal(true);
const readStream = fs.createReadStream(testPath);
client.post('/v1/deploy', readStream, function(err, req, res, data) {
if (err) {
throw new Error(err);
}
should(res).not.null();
should(res.statusCode).not.null();
should(res.statusCode).not.undefined();
res.statusCode.should.equal(200);
should(data).not.null();
should(data.endpoint).not.undefined();
data.endpoint.should.equal('http://endpointyouhit:8080');
done();
});
Yet the file size on the file system is always 0. I'm not using my readStream correctly, but I'm not sure how to correct it. Any help would be greatly appreciated.
Note that I want to stream the file, not load it in memory on transmit and receive, the file can potentially be too large for an in memory operation.
Thanks,
Todd
One thing is that you would need to specify a content-type of multi-part/form-data. However, it looks like restify doesn't support that content type, so you're probably out of luck using the restify client to post a file.
To answer my own question, it doesn't appear to be possible to do this with the restify client. I also tried the request module, which claims to have this capability. However, when using their streaming examples, I always had a file size of 0 on the server. Below is a functional mocha integration test.
const testPath = 'test/assets/test.zip';
fs.existsSync(testPath).should.equal(true);
const readStream = fs.createReadStream(testPath);
var options = {
host: 'localhost'
, port: port
, path: '/v1/deploy/testvalue'
, method: 'PUT'
};
var req = http.request(options, function (res) {
//this feels a bit backwards, but these are evaluated AFTER the read stream has closed
var buffer = '';
//pipe body to a buffer
res.on('data', function(data){
buffer+= data;
});
res.on('end', function () {
should(res).not.null();
should(res.statusCode).not.null();
should(res.statusCode).not.undefined();
res.statusCode.should.equal(200);
const json = JSON.parse(buffer);
should(json).not.null();
should(json.endpoint).not.undefined();
json.endpoint.should.equal('http://endpointyouhit:8080');
done();
});
});
req.on('error', function (err) {
if (err) {
throw new Error(err);
}
});
//pipe the readstream into the request
readStream.pipe(req);
/**
* Close the request on the close of the read stream
*/
readStream.on('close', function () {
req.end();
console.log('I finished.');
});
//note that if we end up with larger files, we may want to support the continue, much as S3 does
//https://nodejs.org/api/http.html#http_event_continue