Pipe the first part of the request result to another host - node.js

I want to download the first megabyte of a link, and then upload that to another server without putting any data on the disk.
What I have tried is this:
axios({
method: 'get',
url: "#DIRECT_DOWNLOAD_LINK#",
responseType: 'stream',
}).then((response) => {
let sizeCounter = 0
response.data.on('data', (newChunk) => {
sizeCounter += newChunk.length
if(sizeCounter > 1024000){
response.data.emit('end')
}
})
let formData = new FormData()
form.append('file', response.data)
axios({method: 'POST', data: formData, url: '#UPLOAD_URL#' headers: {...formData.getHeaders()}}).then((response) => {
console.log(response.data)
})
})
What happens here is that the first request looks like it has stopped, since the data event doesn't get called anymore. However, the next request only gets sent after the first request has finished downloading the whole file!
I have tried calling response.data.destroy() as well, but that doesn't work either.

Related

Extract data from axios body array buffer response

I am calling an API that returns a gzip file in it's body. The problem I am having is I don't know how to extract specific data from that csv file as I need to use it further in my code. This is my code:
const response = await axios.request({
method: 'GET',
url: api,
decompress: true,
responseType: 'arraybuffer',
headers: {
Authorization: `Bearer ${token}`
}
}).then(resp => {
zlib.gunzip(resp.data,(err,output) => {
console.log(output);
})
});
The console log in the zlib callback displays the file as I want it but i don't know how to assign that value to response once the gunzip is finished.
Do you want the variable response to be set to output? If so you can just return output using an intermediary variable. I just called it out, but of course you can rename it to whatever seems fitting. Of course, you also need to handle the error cases, which I did not do below, but that should be a relatively easy task.
const response = await axios.request({
method: 'GET',
url: api,
decompress: true,
responseType: 'arraybuffer',
headers: {
Authorization: `Bearer ${token}`
}
}).then(resp => {
let out;
zlib.gunzip(resp.data,(err,output) => {
out = output;
})
return out;
});
Above the const response is set to the variable output in the gunzip func.

send formData POST request with nodejs

I want to send a post request that contain a form data, i want to do that from nodejs to another external api, i don't have the front-end to send the formData, so all i have a javascript object that has keys and values, so how can i do that?, when i tried sending a normal object, i didn't get the right api response, but when i sent it with a POSTMAN client, i got the correct response.
If you have a correct result in Postman, it's interesting to use the code generator in the same tools to have the desired code :). The button "</>" is on the right bar of the screen.
Here is the code generated from the tool :
var axios = require('axios');
var FormData = require('form-data');
var data = new FormData();
data.append('data', 'asldkfjalsdkjf');
var config = {
method: 'post',
url: 'https://some-domain.com/formdata',
headers: {
...data.getHeaders()
},
data : data
};
axios(config)
.then(function (response) {
console.log(JSON.stringify(response.data));
})
.catch(function (error) {
console.log(error);
});
It's cool, isn't it?
One more thing, you have many options from NodeJS to C#, PHP.. :)
So you want to make a post request with nodejs? In order to do so, you can use the axios library and send the data in the following way.
const axios = require('axios');
let formData = new FormData();
formData.append('x': 'some test data');
axios({
method: 'post',
url: 'https://stackoverflow.com/posts/67709177',
data: formData,
headers: { 'Content-Type': 'multipart/form-data' },
})
.then(res => {
console.log(`statusCode: ${res.statusCode}`)
console.log(res)
})
.catch(error => {
console.error(error)
})
You can install axios using this command.
npm i axios

Get image from Axios and send as Form Data to Wordpress API in a Cloud Function

What I'm trying to accomplish is using a Firebase Cloud Function (Node.js) to:
First download an image from an url (f.eg. from unsplash.com) using an axios.get() request
Secondly take that image and upload it to a Wordpress site using the Wordpress Rest API
The problem seems (to me) to be that the formData doesnt actually append any data, but the axios.get() request actually does indeed retrieve a buffered image it seems. Maybe its something wrong I'm doing with the Node.js library form-data or maybe I get the image in the wrong encoding? This is my best (but unsuccessfull) attempt:
async function uploadMediaToWordpress() {
var FormData = require("form-data");
var formData = new FormData();
var response = await axios.get(
"https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80"
{ responseType: "arraybuffer" }
);
formData.append("file", response.data);
try {
var uploadedMedia = await axios.post("https://wordpresssite.com/wp-json/wp/v2/media",
formData, {
headers: {
"Content-Disposition": 'form-data; filename="example.jpeg"',
"Content-Type": "image/jpeg",
Authorization: "Bearer <jwt_token>",
},
});
} catch (error) {
console.log(error);
throw new functions.https.HttpsError("failed-precondition", "WP media upload failed");
}
return uploadedMedia.data;
}
I have previously successfully uploaded an image to Wordpress with Javascript in a browser like this:
async function uploadMediaToWordpress() {
let formData = new FormData();
const response = await fetch("https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80");
const blob = await response.blob();
const file = new File([blob], "image.jpeg", { type: blob.type });
formData.append("file", file);
var uploadedMedia = await axios.post("https://wordpresssite.com/wp-json/wp/v2/media",
formData, {
headers: {
"Content-Disposition": 'form-data; filename="example.jpeg"',
"Content-Type": "image/jpeg",
Authorization: "Bearer <jwt_token>",
},
});
return uploadedMedia.data;
},
I have tried the last couple of days to get this to work but cannot for the life of me seem to get it right. Any pointer in the right direction would be greatly appreciated!
The "regular" JavaScript code (used in a browser) works because the image is sent as a file (see the new File in your code), but your Node.js code is not really doing that, e.g. the Content-Type value is wrong which should be multipart/form-data; boundary=----...... Nonetheless, instead of trying (hard) with the arraybuffer response, I suggest you to use stream just as in the axios documentation and form-data documentation.
So in your case, you'd want to:
Set stream as the responseType:
axios.get(
'https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80',
{ responseType: 'stream' }
)
Use formData.getHeaders() in the headers of your file upload request (to the /wp/v2/media endpoint):
axios.post( 'https://wordpresssite.com/wp-json/wp/v2/media', formData, {
headers: {
...formData.getHeaders(),
Authorization: 'Bearer ...'
},
} )
And because the remote image from Unsplash.com does not use a static name (e.g. image-name.jpg), then you'll need to set the name when you call formData.append():
formData.append( 'file', response.data, 'your-custom-image-name.jpeg' );
I hope that helps, which worked fine for me (using the node command for Node.js version 14.15.4, the latest release as of writing).

Fetch image and send it using koa2

I am trying to fetch an image from external URL and then send it as a response in koa2. For fetching the image I am using Axios library.
I am trying to do that the following way:
router.get('/get-image', async (ctx, next) => {
const {authToken} = ctx.query
const response = await axiosInstance.get(
'https://www.someurl.com/image/992',
{
headers: {
Authorization: `Bearer ${authToken}`,
},
}
)
ctx.type = 'image/jpeg'
ctx.body = response.data
})
But the image I get from that request is not valid. It only shows empty rectangle).
Can someone point me in the right direction on how to resend the received image?
Set responseType: 'stream'. 'arraybuffer' works too, but 'stream' is even better since you're just passing through the bytes.
By default, I believe axios decodes into a utf-8 string which of course is nonsensical for image binary data.
const response = await axios.get(url, {
responseType: 'stream',
...
})

Node JS upload file streams over HTTP

I'm switching one of my projects from request over to something a bit more light-weight (such as got, axios, or fetch). Everything is going smoothly, however, I'm having an issue when attempting to upload a file stream (PUT and POST). It works fine with the request package, but any of the other three return a 500 from the server.
I know that a 500 generally means an issue on the server's end, but it is consistent only with the HTTP packages that I'm testing out. When I revert my code to use request, it works fine.
Here is my current Request code:
Request.put(`http://endpoint.com`, {
headers: {
Authorization: `Bearer ${account.token.access_token}`
},
formData: {
content: fs.createReadStream(localPath)
}
}, (err, response, body) => {
if (err) {
return callback(err);
}
return callback(null, body);
});
And here is one of the attempts using another package (in this case, got):
got.put(`http://endpoint.com`, {
headers: {
'Content-Type': 'multipart/form-data',
Authorization: `Bearer ${account.token.access_token}`,
},
body: {
content: fs.createReadStream(localPath)
}
})
.then(response => {
return callback(null, response.body);
})
.catch(err => {
return callback(err);
});
Per the got documentation, I've also tried using the form-data package in conjunction with it according to its example and I still get the same issue.
The only difference between these 2 I can gather is with got I do have to manually specify the Content-Type header otherwise the endpoint does give me a proper error on that. Otherwise, I'm not sure how the 2 packages are constructing the body with the stream, but as I said, fetch and axios are also producing the exact same error as got.
If you want any of the snippets using fetch or axios I'd be happy to post them as well.
I know this question was asked a while ago, but I too am missing the simple pipe support from the request package
const request = require('request');
request
.get("https://res.cloudinary.com/demo/image/upload/sample.jpg")
.pipe(request.post("http://127.0.0.1:8000/api/upload/stream"))
// Or any readable stream
fs.createReadStream('/Users/file/path/localFile.jpeg')
.pipe(request.post("http://127.0.0.1:8000/api/upload/stream"))
and had to do some experimenting to find similar features from current libraries.
Unfortunately, I haven't worked with "got" but I hope the following 2 examples help someone else that are interested in working with the Native http/https libraries or the popular axios library
HTTP/HTTPS
Supports piping!
const http = require('http');
const https = require('https');
console.log("[i] Test pass-through: http/https");
// Note: http/https must match URL protocol
https.get(
"https://res.cloudinary.com/demo/image/upload/sample.jpg",
(imageStream) => {
console.log(" [i] Received stream");
imageStream.pipe(
http.request("http://localhost:8000/api/upload/stream/", {
method: "POST",
headers: {
"Content-Type": imageStream.headers["content-type"],
},
})
);
}
);
// Or any readable stream
fs.createReadStream('/Users/file/path/localFile.jpeg')
.pipe(
http.request("http://localhost:8000/api/upload/stream/", {
method: "POST",
headers: {
"Content-Type": imageStream.headers["content-type"],
},
})
)
Axios
Note the usage of imageStream.data and that it's being attached to data in the Axios config.
const axios = require('axios');
(async function selfInvokingFunction() {
console.log("[i] Test pass-through: axios");
const imageStream = await axios.get(
"https://res.cloudinary.com/demo/image/upload/sample.jpg",
{
responseType: "stream", // Important to ensure axios provides stream
}
);
console.log(" [i] Received stream");
const upload = await axios({
method: "post",
url: "http://127.0.0.1:8000/api/upload/stream/",
data: imageStream.data,
headers: {
"Content-Type": imageStream.headers["content-type"],
},
});
console.log("Upload response", upload.data);
})();
Looks like this was a headers issue. If I use the headers directly from FormData (i.e., headers: form.getHeaders()) and just add in my additional headers afterwards (Authorization), then this ends up working just fine.
For me just works when I added other parameters on FormData.
before
const form = new FormData();
form.append('file', fileStream);
after
const form = new FormData();
form.append('file', fileStream, 'my-whatever-file-name.mp4');
So that way I can send stream from my backend to another backend in node, waiting a file in multipart/form-data called 'file'

Resources