Pipe data from Writable to Readable - node.js

Question

I am working on getting files from an SFTP server and piping the data to Box.com using their sdk. The Box sdk takes a readable stream as as parameter for uploading a file. The code that I have written to fetch the files from the sftp server uses the npm module ssh2-sftp-client.
The issue I am having is that a writable stream is "the end of the line" with streams unless you are using something like a Transform which is a Duplex and implements both read and write. Below is the code that I am using. Because I am working on this for a client I am intentionally leaving out some stuff that is not necessary.
Below is the method on the sftp class
async getFile(filepath: string): Promise<Readable> {
logger.info(`Fetching file: ${filepath}`);
const writable = new Writable();
const stream = new PassThrough();
await this.client.get(filepath, writable);
return writable.pipe(stream);
}
The implementation of getting a file and attempting to pipe to box which is an instance of an authorized BoxSDK client.
try {
for (const filename of filenames) {
const stream: Readable = await tmsClient.getFile(
'redacted' + filename,
);
logger.info(`Piping ${filename} to Box...`);
await box.createFile(filename, 'redacted', stream);
logger.info(`${filename} successfully downloaded`);
}
} catch (error) {
logger.error(`Failed to move files: ${error}`);
}
I am not super well versed in streams but based on my research I think this should work in theory.
I have also tried this implementation where the ssh client returns a buffer and then I try and pipe that buffer as a readable stream. With this implementation though I keep getting errors from the Box sdk that the stream ended unexpectedly.
async getFile(filepath: string): Promise<Readable> {
logger.info(`Fetching file: ${filepath}`);
const stream = new Readable();
const buffer = (await this.client.get(filepath)) as Buffer;
stream._read = (): void => {
stream.push(buffer);
stream.push(null);
};
return stream;
}
And the error message: 2020-02-06 15:24:57 error: Failed to move files: Error: Unexpected API Response [400 Bad Request] bad_request - Stream ended unexpectedly.
Any insight is greatly appreciated!

Answer 1

So after doing some more research into this it turns out that the issue is actually with the Box sdk for Node. The sdk is terminating the body of the stream before it is actually done. This is because under the hood they are using the request library which requires a content-length header to send large payloads. Without that in place it will continue to terminate the stream before the payload is sent.
On the Box community forum they suggest adding properties to the stream prototype to pass stuff to the underlying request library. I STRONGLY disagree with this because it is not the correct way to go about it. The Box sdk needs to provide a way to pass in the length of the content in Bytes. As the user of their API I should not have to manipulate their underlying dependencies. I am going to open an issue with their sdk and hopefully get this fixed.
Hope this is useful to someone else!