I have two microservices: 1) in which I am generating pdf using Puppeteer, which is essentially a Buffer object. From this service I want to send the pdf to another microservice, 2) which receives the pdf in request and attaches it in email using mailgun (once I am able to send pdf from one service to another, attaching as an email wont be difficult).
The way I m sending the pdf in requestpromise is this:
import requestPromise from "request-promise";
import {Readable} from "stream";
//pdfBuffer is result of 'await page.pdf({format: "a4"});' (Puppeteer method).
const stream = Readable.from(pdfBuffer);
/*also tried DUPLEX and Readable.from(pdfBuffer.toString()) and this code too.
const readable = new Readable();
readable._read = () => {}
readable.push(pdf);
readable.push(null);
*/
requestPromise({
method: "POST",
url: `${anotherServiceUrl}`,
body: {data},
formData: {
media: {
value: stream,
options: {
filename: "file.pdf",
knownLength: pdfBuffer.length,
contentType: "application/pdf"
}
}
},
json: true
}
});
But doing so, I get "ERR_STREAM_WRITE_AFTER_END" error. How can I send this pdf from one service to another, since the other service sends the email to the user?
I've done it from the frontend with:
fetch(url, {
method: 'POST',
headers: {
'Content-Type': 'application/pdf'
},
body: pdfData
}
In this case pdfData is a blob so you would need a polyfill for that plus node-fetch
const buffer = Buffer.from(pdfBuffer).toString("base64").toString();
sending the buffer in the body.
On Receiving service:
const pdf = Buffer.from(body.buffer, "base64");
fs.write("file.pdf", pdf, ()=> {});
Related
What I'm trying to accomplish is using a Firebase Cloud Function (Node.js) to:
First download an image from an url (f.eg. from unsplash.com) using an axios.get() request
Secondly take that image and upload it to a Wordpress site using the Wordpress Rest API
The problem seems (to me) to be that the formData doesnt actually append any data, but the axios.get() request actually does indeed retrieve a buffered image it seems. Maybe its something wrong I'm doing with the Node.js library form-data or maybe I get the image in the wrong encoding? This is my best (but unsuccessfull) attempt:
async function uploadMediaToWordpress() {
var FormData = require("form-data");
var formData = new FormData();
var response = await axios.get(
"https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80"
{ responseType: "arraybuffer" }
);
formData.append("file", response.data);
try {
var uploadedMedia = await axios.post("https://wordpresssite.com/wp-json/wp/v2/media",
formData, {
headers: {
"Content-Disposition": 'form-data; filename="example.jpeg"',
"Content-Type": "image/jpeg",
Authorization: "Bearer <jwt_token>",
},
});
} catch (error) {
console.log(error);
throw new functions.https.HttpsError("failed-precondition", "WP media upload failed");
}
return uploadedMedia.data;
}
I have previously successfully uploaded an image to Wordpress with Javascript in a browser like this:
async function uploadMediaToWordpress() {
let formData = new FormData();
const response = await fetch("https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80");
const blob = await response.blob();
const file = new File([blob], "image.jpeg", { type: blob.type });
formData.append("file", file);
var uploadedMedia = await axios.post("https://wordpresssite.com/wp-json/wp/v2/media",
formData, {
headers: {
"Content-Disposition": 'form-data; filename="example.jpeg"',
"Content-Type": "image/jpeg",
Authorization: "Bearer <jwt_token>",
},
});
return uploadedMedia.data;
},
I have tried the last couple of days to get this to work but cannot for the life of me seem to get it right. Any pointer in the right direction would be greatly appreciated!
The "regular" JavaScript code (used in a browser) works because the image is sent as a file (see the new File in your code), but your Node.js code is not really doing that, e.g. the Content-Type value is wrong which should be multipart/form-data; boundary=----...... Nonetheless, instead of trying (hard) with the arraybuffer response, I suggest you to use stream just as in the axios documentation and form-data documentation.
So in your case, you'd want to:
Set stream as the responseType:
axios.get(
'https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80',
{ responseType: 'stream' }
)
Use formData.getHeaders() in the headers of your file upload request (to the /wp/v2/media endpoint):
axios.post( 'https://wordpresssite.com/wp-json/wp/v2/media', formData, {
headers: {
...formData.getHeaders(),
Authorization: 'Bearer ...'
},
} )
And because the remote image from Unsplash.com does not use a static name (e.g. image-name.jpg), then you'll need to set the name when you call formData.append():
formData.append( 'file', response.data, 'your-custom-image-name.jpeg' );
I hope that helps, which worked fine for me (using the node command for Node.js version 14.15.4, the latest release as of writing).
I'm trying to upload a remote file via Slack API https://slack.com/api/files.upload using Node with axios library.
https://api.slack.com/methods/files.upload
async filesUpload(token, channel, content, filename) {
const form = new FormData()
form.append('token', token)
form.append('channels', channel)
form.append('content', content)
form.append('filename', filename)
form.append('filetype', 'auto')
const { data } = await axios.post(
'https://slack.com/api/files.upload',
form,
{
headers: form.getHeaders(),
}
)
}
// url is a publicly available remote jpg image
const { data } = await axios.get(url, {
responseType: 'blob',
})
filesUpload('XXXX', 'XXXXX', data, 'foo.jpg')
Slack API says all good, and post its content (some gibberish) to the channel and in the response I get plain text filetype:
...
mimetype: 'text/plain',
filetype: 'text',
...
I'm pretty sure is about encoding I'm sending, but I'm out of options. I was trying to downloading the file with responseType: 'blob', responseType: 'arraybuffer' but no luck.
Please help.
I didn't manage to fix it using axios, but I switched the request to https://github.com/request/request-promise and everything works fine.
I have an API that takes a binary file stream. I am able to hit the API using postman.
Now on the server-side, the content of XML is in string object, So I have created stream first then posted it using axios lib (to call third party API) with form data. This is how I am doing
const Readable = require("stream").Readable;
const stream = new Readable();
stream.push(myXmlContent);
stream.push(null); // the end of the stream
const formData = new FormData();
formData.append("file", stream);
const response = await axios({
method: "post",
url: `${this.BASE_URL}/myurl`,
data: formData
});
return response.data;
but this is not sending data properly as third party API throws Bad Request: 400.
How can I send XML string content to API as a stream?
Used Buffer.from method to send the stream. This worked for me
const response = await axios({
method: "post",
url: `${this.BASE_URL}/myUrl`,
data: Buffer.from(myXmlContent),
headers: { "Content-Type": `application/xml`, }
});
return response.data;
I'm trying to send an image to my server using axios with react-native.
For doing this, I'm passing the image data (the base 64 encoded image data) directly to an uploadPicture function which uses axios this way:
const uploadPicture = async (data): Promise<AxiosResponse<string>> => {
const response = publicApi.post(
`${API_URL}/upload`,
{
image: data,
},
{
headers: { 'Content-Type': 'multipart/form-data' },
transformRequest: [transformToFormData],
}
);
return response;
};
const transformToFormData: AxiosTransformer = data => {
const formData = new FormData();
for (const key in data) {
formData.append(key, data[key]);
}
return formData;
};
The issue I face :
I get an internal error, like if my image was not correctly transmitted through my request.
If I'm doing the exact same request using Postman, it works fine, setting the body like this :
Which make me think that the issue doesn't come from my server but from my axios request.
Any idea of what I could be doing wrong ? Am I missing any axios option somewhere ?
I managed to find a solution:
I used the fetch function from javascript instead of axios
I send a file object instead of the data directly
I had to disable the react-native network inspect, otherwise the upload won't work
My working solution below, image being the response of react native image picker:
const file = {
uri: image.uri,
name: image.fileName,
type: image.type,
size: image.fileSize,
slice: () => new Blob(),
};
const body = new FormData();
body.append('image', file);
const response = await fetch(`${API_URL}/upload`, {
method: 'POST',
body,
});
I have an API endpoint that lets the client post their csv to our server then post it to someone else server. I have done our server part which save uploaded file to our server, but I can't get the other part done. I keep getting error { message: 'File not found', code: 400 } which may mean the file never reach the server. I'm using axios as an agent, does anyone know how to get this done? Thanks.
// file = uploaded file
const form_data = new FormData();
form_data.append("file", fs.createReadStream(file.path));
const request_config = {
method: "post",
url: url,
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios(request_config);
Update
As axios doc states as below and the API I'm trying to call requires a file
// data is the data to be sent as the request body
// Only applicable for request methods 'PUT', 'POST', and 'PATCH'
// When no transformRequest is set, must be of one of the following types:
// - string, plain object, ArrayBuffer, ArrayBufferView, URLSearchParams
// - Browser only: FormData, File, Blob
// - Node only: Stream, Buffer
Is there any way to make axios send a file as a whole? Thanks.
The 2 oldest answers did not work for me. This, however, did the trick:
const FormData = require('form-data'); // npm install --save form-data
const form = new FormData();
form.append('file', fs.createReadStream(file.path));
const request_config = {
headers: {
'Authorization': `Bearer ${access_token}`,
...form.getHeaders()
}
};
return axios.post(url, form, request_config);
form.getHeaders() returns an Object with the content-type as well as the boundary.
For example:
{ "content-type": "multipart/form-data; boundary=-------------------0123456789" }
I'm thinking the createReadStream is your issue because its async. try this.
Since createReadStream extends the event emitter, we can "listen" for when it finishes/ends.
var newFile = fs.createReadStream(file.path);
// personally I'd function out the inner body here and just call
// to the function and pass in the newFile
newFile.on('end', function() {
const form_data = new FormData();
form_data.append("file", newFile, "filename.ext");
const request_config = {
method: "post",
url: url,
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios(request_config);
});
This is what you really need:
const form_data = new FormData();
form_data.append("file", fs.createReadStream(file.path));
const request_config = {
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios
.post(url, form_data, request_config);
In my case, fs.createReadStream(file.path) did not work.
I had to use buffer instead.
const form = new FormData();
form.append('file', fs.readFileSync(filePath), fileName);
const config = {
headers: {
Authorization: `Bearer ${auth.access_token}`,
...form.getHeaders(),
},
};
axios.post(api, form.getBuffer(), config);
I have made an interceptor you can connect to axios to handle this case in node: axios-form-data. Any feedback would be welcome.
npm i axios-form-data
example:
import axiosFormData from 'axios-form-data';
import axios from 'axios';
// connect axiosFormData interceptor to axios
axios.interceptors.request.use(axiosFormData);
// send request with a file in it, it automatically becomes form-data
const response = await axios.request({
method: 'POST',
url: 'http://httpbin.org/post',
data: {
nonfile: 'Non-file value',
// if there is at least one streamable value, the interceptor wraps the data into FormData
file: createReadStream('somefile'),
},
});
// response should show "files" with file content, "form" with other values
// and multipart/form-data with random boundary as request header
console.log(response.data);
I had a same issue, I had a "pdf-creator-service" for generate PDF document from html.
I use mustache template engine for create HTML document - https://www.npmjs.com/package/mustache
Mustache.render function returns html as a string what do I need to do to pass it to the pdf-generator-service ? So lets see my suggestion bellow
//...
async function getPdfDoc(props: {foo: string, bar: string}): Promise<Buffer> {
const temlateFile = readFileSync(joinPath(process.cwd(), 'file.html'))
mustache.render(temlateFile, props)
const readableStream = this.getReadableStreamFromString(htmlString)
const formData = new FormData() // from 'form-data'
formData.append('file', options.file, { filename: options.fileName })
const formHeaders = formData.getHeaders()
return await axios.send<Buffer>(
{
method: 'POST',
url: 'https://pdf-generator-service-url/pdf',
data: formData,
headers: {
...formHeaders,
},
responseType: 'arraybuffer', // ! important
},
)
}
getReadableStreamFromString(str: string): Readable {
const bufferHtmlString = Buffer.from(str)
const readableStream = new Readable() // from 'stream'
readableStream._read = () => null // workaround error
readableStream.push(bufferHtmlString)
readableStream.push(null) // mark end of stream
return readableStream
}
For anyone who wants to upload files from their local filesystem (actually from anywhere with the right streams architecture) with axios and doesn't want to use any external packages (like form-data).
Just create a readable stream and plug it right into axios request function like so:
await axios.put(
url,
fs.createReadStream(path_to_file)
)
Axios accepts data argument of type Stream in node context.
Works fine for me at least in Node v.16.13.1 and with axios v.0.27.2