I'm trying to read an http/2 push stream, but the code below fails with an error. When I use the same URL in Chrome browser, I receive the 'heartbeat' being generated by the stream server and can see data coming in as well. Any pointers? My goal is simply to read all the JSON push responses being generated by the server.
Error:
{ Error [ERR_HTTP2_ERROR]: Protocol error
at Http2Session.onSessionInternalError [as error] (internal/http2/core.js:637:26)
code: 'ERR_HTTP2_ERROR',
name: 'Error [ERR_HTTP2_ERROR]',
errno: -505 }
Node.JS Code:
const http2 = require('http2');
const client = http2.connect('http://api.service.com/absolute/path/subscribe?api_key={key}');
const req = client.request();
req.setEncoding('utf8');
req.on('response', (headers, flags) => {
console.log(headers);
});
let data = '';
req.on('data', (d) => data += d);
req.on('end', () => {
console.log('end');
console.log(data);
client.destroy()
});
req.end();
Side Note: I'm new to Node.js, and http/2 push stream is a new topic for me as well, so consider me a beginner, struggling with this.
Resolved with some help, using 'request' library:
https://www.npmjs.com/package/request
const request = require("request");
const getPushData = () => {
let completeResponse = "";
request
.get('http://api.service.com/absolute/path/subscribe?api_key={key}')
.on('response', (response)=>{
console.log("Response received successfully");
})
.on('data', (chunk)=>{
console.log("Receiving the chunk...");
console.log(chunk);
let dataReceived = new Buffer(chunk).toString("utf8");
console.log(dataReceived);
completeResponse += chunk;
})
.on('error', (err)=>{
console.log("Something went wrong:");
console.log(err.message);
})
.on("end", ()=>{
console.log("The complete data received is:");
console.log(completeResponse);
let jsonObj = JSON.parse(completeResponse);
console.log(jsonObj);
});
}
getPushData();
Side Note: The problem with this script is that if connection drops, the stream doesn't resume.
Related
I am trying to use StreamingPull from Pub/Sub. There are messages published but I don't see any response. for following code
const {v1 } = require('#google-cloud/pubsub');
const request = {
subscription: 'projects/<projectId>/subscriptions/Temp'',
stream_ack_deadline_seconds: 600
};
console.log('Pulling Messages...');
const stream = await subClient.streamingPull({});
stream.on('data', response => {
console.log(response);
});
stream.on('error', err => {
console.log(err);
});
stream.on('end', () => {
console.log("end");
});
stream.write(request);
stream.end();
I see the code silently finishing without the response being logged. Is there any attribute I am missing in my request. As per doc of StreamingPullRequest nothing else is mandatory. The only usage example is here in the test files.
I am looking at a few, currently widely used request libraries and how I could use them to automate file downloads + make them reliable enough.
I stumbled over download (npm), but since its based on got (npm) I thought I would try got first directly.
Problem
One problem I could encounter while downloading a file, is that the source file (on the server) could be overwritten during download. When I try reproduce this behaviour with got, got just stops the download process without rising any errors.
What I have so far
The only solution I could come up with, was to use got.stream - piping the request into a FileWriter, and compare total with transferred after the request has ended.
const app = require('express')();
const fs = require('fs');
const stream = require('stream');
const { promisify } = require('util');
const got = require('got');
const pipeline = promisify(stream.pipeline);
app.use('/files', require('express').static('files'));
app.listen(8080);
(async () => {
try {
let progress = null;
// Setup Got Request + EventHandlers
const request = got.stream('http://localhost:8080/files/1gb.test')
.on('downloadProgress', (p) => { progress = p; })
.on('end', () => {
console.log("GOT END");
console.log(progress && progress.transferred === progress.total ? "COMPLETE" : "NOTCOMPLETE");
})
// this does not get fired when source file is overwritten
.on('error', (e) => {
console.log("GOT ERROR");
console.log(e.message);
});
// WriteStream + EventHandlers
const writer = fs.createWriteStream('./downloaded/1gb_downloaded.test')
.on('finish', () => {
console.log("WRITER FINISH");
})
.on('error', (error) => {
console.log("WRITER ERROR", error.message);
})
.on('end', () => {
console.log("WRITER END");
})
.on('close', () => {
console.log("WRITER CLOSE");
});
await pipeline(request, writer);
} catch (e) {
console.error(e.name, e.message);
}
})();
Where do the files come from
In the real world the files i am trying to download are coming from a server which I do not have access to, I don't own it. I don't have any information how this server is setup. However I added a simple local express server to the example code above to try things out.
const app = require('express')();
app.use('/files', require('express').static('files'));
app.listen(8080);
Question
Is this solution reliable enough to detect a "none-finished" download ( so for the case the source file gets overwritten during download ) ? Or are there any othere events I could listen to which I missed ?
The got Request stream emits a error event whenever something goes wrong.
const request = got('http://localhost:8080/files/1gb.test')
.on('downloadProgress', (p) => { progress = p; })
.on('end', (e) => {
console.log("GOT END");
})
.on('error', (err) => {
// Handle error here
});
Various properties in the error object are available here
progress.total will not be available unless the server explicity sets the Content-Length (Most servers do, but you might want to have a look out for that)
It seems there is no inbuilt way to safely check if a download has been completed 100% using got. I came to the conclusion that my best option for now would be to use the NodeJS Module http, which includes, on the ClientRequest Object, an aborted property. When the ReadStream emits an end event I can check whether aborted is true or false.
I tested this method in the case when the source file gets overwritten during download, it works !
const http = require('http');
const app = require('express')();
const fs = require('fs');
app.use('/files', require('express').static('files'));
app.listen(8080);
http.get('http://localhost:8080/files/1gb.test', function (response) {
// WriteStream + EventHandlers
const writer = fs.createWriteStream('./downloaded/1gb_downloaded.test')
.on('finish', () => {
console.log("WRITER FINISH");
})
.on('error', (error) => {
console.log("WRITER ERROR", error.message);
})
.on('end', () => {
console.log("WRITER END");
})
.on('close', () => {
console.log("WRITER CLOSE");
});
// ReadStream + EventHandlers
response
.on('error', (e) => {
console.log("READER ERROR", e.message)
})
.on('end', () => {
console.log("READER END")
console.log(response.aborted ? "DOWNLOAD NOT COMPLETE" : "DOWNLOAD COMPLETE")
})
.on('close', () => {
console.log("READER CLOSE")
})
response.pipe(writer);
});
On the upside, this gives me -1 on dependencies :) , since I don't need got.
On the downside, this just assures me, that a running download was not aborted due to the source file being overwritten. When using http module I need to include more error handling when for example the file was not found to begin with, which had been more convenient using a request library like axios or got.
UPDATE
Realizing that the ReadableStream from http has something like the aborted property made me wonder why none of the request wrapper libraries like got does offer something similar. So I tried axios again, with :
axios({
method: 'get',
url: 'http://localhost:8080/files/1gb.test',
responseType: 'stream'
}).then( function ( response ) {
});
Here the ReadableStream comes in response.data and it has the same aborted property ! 🎉 .
I picked up some old stream code recently (written when 8.x was LTS) and attempted to update it to 12.x. This led to an interesting break in the way I dealt with ENOENT file errors.
Here's a simplification:
const { createServer } = require('http')
const { createReadStream } = require('fs')
const PORT = 3000
const server = createServer((req, res) => {
res.writeHead(200, {
'Content-Type': 'application/json'
})
const stream = createReadStream(`not-here.json`, {encoding: 'utf8'})
stream.on('error', err => {
stream.push(JSON.stringify({data: [1,2,3,4,5]}))
stream.push(null)
})
stream.pipe(res)
})
server.listen(PORT)
server.on('listening', () => {
console.log(`Server running at http://localhost:${PORT}/`)
})
In Node 8, the above code works fine. I'm able to intercept the error, write something to the stream and let it close normally.
In Node 10+ (tested 10, 12, and 13) the stream is already destroyed when my error callback is called. I can't push new things on the stream and handle the error gracefully for the client side.
Was this an intentional change and can I still handle this error in a nice way for the clint side?
One possibility. Open the file yourself and only create the stream with that already successfully opened file. That will allow you to handle ENOENT (or any other errors upon opening the file) before you get into the messy stream error handling mechanics. The stream architecture seems most aligned with aborting upon error, not recovering with some alternate behavior.
const { createServer } = require('http');
const fs = require('fs');
const PORT = 3000;
const server = createServer((req, res) => {
res.writeHead(200, {'Content-Type': 'application/json'});
fs.open('not-here.json', {encoding: 'utf8'}, (err, fd) => {
if (err) {
// send alternative response here
res.end(JSON.stringify({data: [1,2,3,4,5]}));
} else {
const stream = fs.createReadStream(null, {fd, encoding: 'utf8'});
stream.pipe(res);
}
});
});
server.listen(PORT);
server.on('listening', () => {
console.log(`Server running at http://localhost:${PORT}/`)
});
You could also try experimenting with the autoDestroy or autoClose options on your stream to see if any of those flags will allow the stream to still be open for you to push data into it, even if the file created an error opening or reading. The doc on those flags is not very complete so some combination of programming experiements and studying the code would be required to see if they could be manipulated to still add data to the stream after your stream got an error.
The answer by jfriend00 pointed me in the right direction.
Here are two different ways I solved this. I wanted a function that returned a stream rather than handle the error in the req handler function. This is more like what I'm actually doing in real code.
Handling error from stream:
Just like above except I took care to manually destroy the stream. Does this correctly take care of the internal file descriptor? I think it does.
const server = createServer((req, res) => {
res.writeHead(200, {
'Content-Type': 'application/json'
})
getStream().pipe(res)
})
function getStream() {
const stream = createReadStream(`not-here.json`, {
autoClose: false,
encoding: 'utf8'
})
stream.on('error', err => {
// handling "no such file" errors
if (err.code === 'ENOENT') {
// push JSON data to stream
stream.push(JSON.stringify({data: [1,2,3,4,5]}))
// signal the end of stream
stream.push(null)
}
// destory/close the stream regardless of error
stream.destroy()
console.error(err)
})
return stream
}
Handling the error during file open:
Like jfriend00 suggests.
const { promisify } = require('util')
const { Readable } = require('stream')
const { open, createReadStream } = require('fs')
const openAsync = promisify(open)
const server = createServer(async (req, res) => {
res.writeHead(200, {
'Content-Type': 'application/json'
})
const stream = await getStream()
stream.pipe(res)
})
async function getStream() {
try {
const fd = await openAsync(`not-here.json`)
return createReadStream(null, {fd, encoding: 'utf8'})
} catch (error) {
console.log(error)
// setup new stream
const stream = new Readable()
// push JSON data to stream
stream.push(JSON.stringify({data: [1,2,3,4,5]}))
// signal the end of stream
stream.push(null)
return stream
}
}
I still like handling in the stream better but would love to hear reasons why you might do it one way or the other.
I want to get binary from image to rotate then, using sharp.rotate();
I try to do this content += chunk; but dosent work.
let Client = require('ftp');
let fs = require('fs');
let sharp = require('sharp');
let path = 'users/'+userId+'/headerImage/header';
let Ftp = new Client();//create new istance of Ftp
//Start. Here we get image from server
await Ftp.on('ready', function(){
Ftp.get(path, async function(err, stream){
if(err){
res.status(400).send(err);
};
var content = '';
await stream.on('data', async (chunk) => {
content += chunk;
});
await stream.on('end', async function(){
console.log(content);
let image = await sharp(content);
await image
.rotate(90)
.toBuffer()
.then(async data => {
console.log(data);
})
.catch(error => {
console.log(error);
});
Ftp.end();
});
});
});
await Ftp.connect({
host: fileTransferProtocol.host,
port: fileTransferProtocol.port,
user: fileTransferProtocol.user,
password: fileTransferProtocol.pass
});
console: Error: [Error: Input file is missing]
I believe the problem you are having is that you are not handling the incoming data as a buffer. The stream variable inside the Ftp.get callback is of type ReadableStream. By default, stream data will be returned as Buffer objects unless you specify an encoding for the data, using the readable.setEncoding() method.
For your specific purpose, you want to handle the data as a Buffer object, since that is what the sharp function is expecting. To store the incoming data into a Buffer modify what happens on the data event.
var content = new Buffer(0);
stream.on("data", async chunk => {
content = Buffer.concat([content, chunk]);
});
Also, I don't think you are using async/await duly. The ftp module runs with callbacks and events, not promises. Appending those functions with await won't make them run synchronously.
Please check the following link to find more information about this feature:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
If you want to us async/await to handle your ftp requests try this module:
https://www.npmjs.com/package/promise-ftp
It provides an asynchronous interface for communicating with an FTP server.
I make https request with Node's https.request() and listen to response 'data' event:
const req = https.request('https://stackoverflow.com/', res => {
res.on('data', (chunk) => {
console.log('data');
})
});
req.end();
I works fine. I get 'data' outputs in console.
However, when I add hook to reqest's emit (for debug purposes) as this:
const req = https.request('https://stackoverflow.com/', res => {
res.on('data', (chunk) => {
console.log('data');
})
});
const original_emit = req.emit;
req.emit = function(event_name) {
console.log('reqest event: ' + event_name);
original_emit.apply(req, arguments);
};
req.end();
I don't get 'data' output anymore.
Cannot find reason for that.
Node version: 8.11.3