Node + Angular : download a file with HTTP request - node.js

I tried 400 combinations of syntaxes and headers, I can't figure out how to make a HTTP call from Angular to retrieve a file from my NodeJS server.
Found on Stackoverflow and tried, to no avail :
Download file from http post request - Angular 6
How download a file from HttpClient
Download a file from NodeJS Server using Express
How do I download a file with Angular2
It can't be a simple <a download> tag, or a public express.static() folder, because access to the file is restricted and I need to pass a JWT token along (in Node, I have an Express authentication middleware that will reject the request if no token is provided in the headers or if it is invalid).
The file is a GZIP : ./dumps/dump.gz and weighs 812 Kb.
I do manage to download the file, but whatever I try, it weighs 1.4 MB or 94 bytes (wrong size) and can't be opened (7zip can't open file downloads/dump.gz as archive).
What I have tried Angular-side (multiple attempts) :
import { saveAs } from 'file-saver';
let headers = new Headers({
"Authorization": "Bearer " + user.jwt, // I need this in the headers
"Content-Type" : "application/octet-stream", // Tried with and without, "application/gzip", "application/json", no difference
"responseType": "blob" as "json", // Tried with and without, "text", "json", no difference
"Access-Control-Expose-Headers" : "Content-Disposition" // Tried with and without, no difference
})
this.http
.get("/download/dump", { headers })
.toPromise()
.then(res => {
const blob = new Blob([res["_body"]] , { type: "application/octet-stream;"} ); // Error : body is not a blob or an array buffer
// const blob = new Blob([res["_body"]]); // Same result
// const blob = new Blob([res.blob()]); // Error : body is not a blob or an array buffer
saveAs(blob, "dump.gz"); // Saves a big corrupted file
// window.URL.createObjectURL(new Blob(blob, {type: 'blob'})); Saves a 94 byte corrupted file. Tried {type: 'gzip'}, same thing
})
.catch(err => console.error("download error = ", err))
What I have tried Node-side (multiple attempts) :
EDIT
Node has been innocented as I could retrieve the file directly from Chrome after disabling authentication. So, the back-end works and the issue is in Angular.
app.get( "/download/dump", authenticate, (req:Request, res:Response) => {
const file = path.resolve(__dirname, `./dumps/dump.gz`);
res
.set({ // Tried with and without headers, doesn't seem to do anything
"Content-Disposition" : "attachment", // Tried with and without
"filename" : "dump.gz", // Tried with and without
"filename*" : "dump.gz", // Tried with and without
"Content-Encoding" : "gzip", // Tried with and without
"Content-Type" : "application/gzip" // Tried with and without, "application/text", "application/json", no difference
})
.sendFile(file); // getting a big corrupted file
// .download(file); // Same result (big corrupted file)
})

Assuming that you are using the new HttpClient from angular (available since angular 4), this should work
front
import { saveAs } from 'file-saver';
import {HttpHeaders} from "#angular/common/http";
let headers = new HttpHeaders({
"Authorization": "Bearer " + user.jwt, // Auth header
//No other headers needed
});
this.http
.get("/download/dump", { headers, responseType: "blob" }) //set response Type properly (it is not part of headers)
.toPromise()
.then(blob => {
saveAs(blob, "dump.gz");
})
.catch(err => console.error("download error = ", err))
backend
app.get( "/download/dump", authenticate, (req:Request, res:Response) => {
const file = path.resolve(__dirname, `./dumps/dump.gz`);
//No need for special headers
res.download(file);
})

Related

NodeJS server side - file Expected UploadFile, received: <class 'str'>

having issues uploading file from NodeJs server side, found 100 posts and reasearches but nothing works, would appreciate any help.
Structure of the App
Front App - React Admin framework receving file and i encode in base64 the content of the image to send to API
Backend - NestJS App - receving base64 image in API
From my backend API need to send file to an external backend (Python API) to upload - here is the problem
Please see below my code, something wrong with the file from JS
i have tried several methods and all of them ends in same error
1 solution
converting base64 image in buffer and send to external backend to upload the file
have tried to pass as well cleanImageBuffer but no changes
import axios from 'axios';
import FormData from 'form-data';
export async function upload(
fileBase64: string,
filename: string
): Promise<any> {
const buffer = Buffer.from(fileBase64, 'base64')
const extension = fileBase64.substring(fileBase64.indexOf('/') + 1, fileBase64.indexOf(";base64"))
const cleanBase64 = fileBase64.replace(/^data:image\/png;base64,/, '')
const cleanImageBuffer = Buffer.from(cleanBase64, 'base64')
const formData = new FormData();
// have tried to pass as well cleanImageBuffer but no changes
formData.append('file', buffer);
formData.append('fileName', filename + '.' + extension);
formData.append('namespace', 'test');
return await axios
.post('external_api_url', JSON.stringify(formData), {
headers: {
Authorization: `Bearer token`,
ContentType: 'multipart/form-data'
}
})
.then((response) => {
console.log('response = ' + JSON.stringify(response))
})
result 1 solution
{
"status": "error",
"error": {
"code": "bad_request",
"message": "file Expected UploadFile, received: <class 'str'>"
}
}
2 solution
from base64 image received saving on my disk
after creating a stream and sending the image
export async function upload (
fileBase64: string,
filename: string
): Promise<any> {
const extension = fileBase64.substring(fileBase64.indexOf('/') + 1, fileBase64.indexOf(";base64"))
const cleanBase64 = fileBase64.replace(/^data:image\/png;base64,/, '')
const TMP_UPLOAD_PATH = '/tmp'
if (!fs.existsSync(TMP_UPLOAD_PATH)) {
fs.mkdirSync(TMP_UPLOAD_PATH);
}
fs.writeFile(TMP_UPLOAD_PATH + '/' + filename + '.' + extension, cleanBase64, 'base64', function(err) {
console.log(err);
})
const fileStream = fs.createReadStream(TMP_UPLOAD_PATH + '/' + filename + '.' + extension)
const formData = new FormData();
formData.append('file', fileStream, filename + '.' + extension);
formData.append('fileName', filename + '.' + extension);
formData.append('namespace', 'test');
return await axios
.post('external_api_url', formData, {
headers: {
Authorization: `Bearer token`,
ContentType: 'multipart/form-data'
}
})
.then((response) => {
console.log('response = ' + JSON.stringify(response))
})
}
result 2 solution
{
"status": "error",
"error": {
"code": "bad_request",
"message": "file Expected UploadFile, received: <class 'str'>"
}
}
other solution that ended in same result
tried to use fetch from node-fetch - same result
found out that some people had an outdated version of axios and having this issues, i have installed latest axios version 1.1.3 but same result
best scenario that i need
from base64 image received
convert in buffer and send file to external Python API so to avoid saving the file on local disk
would appreciate any help
below is a python example that works but not JS (JS nothing works)
import requests
url = "http://127.0.0.1:8000/external_api"
payload={'namespace': 'test'}
files=[
('file',('lbl-pic.png',open('/local/path/lbl-pic.png','rb'),'image/png'))
]
headers = {
'Authorization': 'Bearer token'
}
response = requests.request("POST", url, headers=headers, data=payload, files=files)
print(response.text)
Just a suggestion:
Here's a line which returns mentioned error https://github.com/tiangolo/fastapi/blob/41735d2de9afbb2c01541d0f3052c718cb9f4f30/fastapi/datastructures.py#L20, you might find it useful.
First see if you can make it work with regular HTML file input (don't complicate with Base64 yet), as described here https://stackoverflow.com/a/70824288/2347084
If (1) works, then try converting base64 into a File object as suggested here https://stackoverflow.com/a/47497249/2347084
Combine (2) and (1)
I want to post my solution that worked, because as i can see in internet everybody have issues with FormData on nodejs
i was using axios to send the buffer for uploading file
issue is with axios and specially with FormData, it does not add Content-Lenght in headers, any version of axios does not do this
python API had required Content-Lenght
So kindly was asking to make this header optionally in python API and the code started to work
The solution is if anybody goes in similar issues
axios does not add Content-Lenght when working with FormData (could not find any version of axios that works)
if you work with a buffer without having the file on local disk than will be issue because of Content-Lenght
if u have the file locally than using the module fs u are able to read the file and add all hedears and Content-Lenght
on axios GitHub issue is saying that this bug is fixed in latest axios, but it was still not working in my case
below is a code by using buffer and Content-Lenght is not required in 3rd API
function upload (image: {imageBase64: string, fileName: string}) {
const { imageBase64, fileName } = image;
const cleanBase64 = imageBase64.substr(imageBase64.indexOf(',') + 1);
// buffer should be clean base64
const buffer = Buffer.from(cleanBase64, 'base64');
const formData = new FormData();
// filename as option is required, otherwise will not work, will say that received file is string and UploadFile
formData.append('file', buffer, { filename: fileName });
return client
.post('url', formData, {
headers: {
...formData.getHeaders(),
},
})
.then((response) => response.data)
.catch((error) => {
return {
status: 'error',
error,
};
});
}

how do you set a custom name for a file being uploaded to google drive via API call?

I am trying to upload a text file to google drive via API call. I am working low level, so I don't use any external libraries. I send HTTPS Request, the file gets uploaded on Google Drive along with the content, but the document name is "Untitled". I am able to change the name in a separate HTTPS request using PATCH method, but I want to know if it is possible to set the name before sending the initial HTTPS request. Below is my the header file and HTTPS code I am sending, I would appreciate it if you can help me set the name anything other than "Untitled" .
const options = {
method: "POST",
headers: {
"Authorization": `${token_credential.token_type} ${token_credential.access_token}`,
"Content-Type":'text/plain' ,
},
body: {
name:"MyTextFile.txt"
}
}
let makeFileRequest=https.request(
sendDriveTask,
options,
(res, err) => {
if(err) {
console.log(err)
} else {
console.log("success", res.statusCode)
}
}
)
makeFileRequest.end(filecontent);
}
Below is the respond I get back from google drive:
{"kind":"drive#file","id":"1XBcb9q8Q__b6gIP0sH5oOhZOKQMezlIK","name":"Untitled","mimeType":"text/plain"}
From your showing script, I understood that you want to achieve your goal using https.request. In your situation, how about the following modification?
Modification points:
About I am trying to upload a text file to google drive via API call., when I saw your script, the text data is not included. If you want to create a text file with empty content, please modify the endpoint.
If you want to create a text file with the text content, please include the text data.
Modified script 1:
In this modification, a text file with empty content is created.
const https = require("https");
const token_credential = {
token_type: "Bearer",
access_token: "###", // Please set your access token.
};
const sendDriveTask = "https://www.googleapis.com/drive/v3/files";
const body = JSON.stringify({ name: "MyTextFile.txt" });
const options = {
method: "POST",
headers: {
Authorization: `${token_credential.token_type} ${token_credential.access_token}`,
"Content-Type": "application/json",
},
};
let makeFileRequest = https.request(sendDriveTask, options, (res) => {
res.on("data", (r) => {
console.log(r.toString());
});
res.on("end", () => {
console.log("Done.");
});
});
makeFileRequest.on("error", (e) => {
console.log(e.message);
});
makeFileRequest.write(body);
makeFileRequest.end();
When this script is run, a text file of "MyTextFile.txt" with empty content is created in the root folder.
Modified script 2:
In this modification, a text file with sample text content is created. In this case, the request body is sent as multipart/related.
const https = require("https");
const metadata = JSON.stringify({ name: "MyTextFile.txt" }); // Please set the file metadata.
const textData = "sample text data"; // Please set the sample text content.
const token_credential = {
token_type: "Bearer",
access_token: "###", // Please set your access token.
};
const sendDriveTask = "https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart";
const boundary = "sample123";
const body = [`--${boundary}\r\n`, "Content-Type: application/json; charset=utf-8\r\n\r\n", metadata + "\r\n", `--${boundary}\r\n`, "Content-Type: text/plain\r\n\r\n", textData + "\r\n", `--${boundary}--`].join("");
const options = {
method: "POST",
headers: {
Authorization: `${token_credential.token_type} ${token_credential.access_token}`,
"Content-Type": "multipart/related; boundary=" + boundary,
},
};
let makeFileRequest = https.request(sendDriveTask, options, (res) => {
res.on("data", (r) => {
console.log(r.toString());
});
res.on("end", () => {
console.log("Done.");
});
});
makeFileRequest.on("error", (e) => {
console.log(e.message);
});
makeFileRequest.write(body);
makeFileRequest.end();
When this script is run, a text file of "MyTextFile.txt" with the sample text of sample text data is created in the root folder.
Note:
This sample script supposes that your access token can be used for creating a file on Google Drive. Please be careful about this.
About the 2nd pattern, the maximum file size is 5 MB. When you want to upload more sizes, please use the resumable upload. Ref
References:
Files: create
Upload file data

How to PUT file to Tika-server in NodeJs

The Scenario
I am running a VueJs client, a NodeJs Restify API Server, and a Tika-server out of the official Docker Image. A user makes a POST call with formData containing a PDF file to be parsed. The API server receives the POST call and I save the PDF on the server. The API server should PUT the file to the unpack/all endpoint on the Tika-server and receive a zip containing a text file, a metadata file, and the set of images in the PDF. I would then process the zip and pass some data back to the client.
The Problem
I create a buffer containing the file to be parsed using let parsingData = fs.createReadStream(requestFilename); or let parsingData = fs.readFileSync(requestFilename);, set the axios data field to parsingData, then make my request. When I get the response from the Tika-server, it seems the Tika-server has treated the request as empty; within the zip, there are no images, the TEXT file is empty, the METADATA.
When I make the following request to the Tika-server via CURL curl -T pdf_w_images_and_text.pdf http://localhost:9998/unpack/all -H "X-Tika-PDFExtractInlineImages: true" -H "X-Tika-PDFExtractUniqueInlineImagesOnly: true"> tika-response.zip, I get a response zip file containing accurate text, metadata, stripped images.
The Code
let parsingData = fs.createReadStream('pdf_w_images_and_text.pdf');
axios({
method: 'PUT',
url: 'http://localhost:9998/unpack/all',
data: parsingData,
responseType: 'arraybuffer',
headers: {
'X-Tika-PDFExtractInlineImages': 'true',
'X-Tika-PDFExtractUniqueInlineImagesOnly': 'true'
},
})
.then((response) => {
console.log('Tika-server response recieved');
const outputFilename = __dirname+'\\output.zip';
console.log('Attempting to convert Tika-server response data to ' + outputFilename);
fs.writeFileSync(outputFilename, response.data);
if (fs.existsSync(outputFilename)) {
console.log('Tika-server response data saved at ' + outputFilename);
}
})
.catch(function (error) {
console.error(error);
});
The Question
How do I encode and attach my file to my PUT request in NodeJs such that the Tika-server treats it as it does when I make the request through CURL?
Axios is sending the request with a content type of application/x-www-form-urlencoded and therefore the file content isn't being detected and parsed.
You can change this by passing either the known content type of the file, or a content type of application/octet-stream to allow Apache Tika Server to auto-detect.
Below is a sample based on your question's code that illustrates this:
#!/usr/bin/env node
const fs = require('fs')
const axios = require('axios')
let parsingData = fs.createReadStream('test.pdf');
axios({
method: 'PUT',
url: 'http://localhost:9998/unpack/all',
data: parsingData,
responseType: 'arraybuffer',
headers: {
'X-Tika-PDFExtractInlineImages': 'true',
'X-Tika-PDFExtractUniqueInlineImagesOnly': 'true',
'Content-Type': 'application/octet-stream'
},
})
.then((response) => {
console.log('Tika-server response recieved');
const outputFilename = __dirname+'/output.zip';
console.log('Attempting to convert Tika-server response data to ' + outputFilename);
fs.writeFileSync(outputFilename, response.data);
if (fs.existsSync(outputFilename)) {
console.log('Tika-server response data saved at ' + outputFilename);
}
})
.catch(function (error) {
console.error(error);
});

Get image from Axios and send as Form Data to Wordpress API in a Cloud Function

What I'm trying to accomplish is using a Firebase Cloud Function (Node.js) to:
First download an image from an url (f.eg. from unsplash.com) using an axios.get() request
Secondly take that image and upload it to a Wordpress site using the Wordpress Rest API
The problem seems (to me) to be that the formData doesnt actually append any data, but the axios.get() request actually does indeed retrieve a buffered image it seems. Maybe its something wrong I'm doing with the Node.js library form-data or maybe I get the image in the wrong encoding? This is my best (but unsuccessfull) attempt:
async function uploadMediaToWordpress() {
var FormData = require("form-data");
var formData = new FormData();
var response = await axios.get(
"https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80"
{ responseType: "arraybuffer" }
);
formData.append("file", response.data);
try {
var uploadedMedia = await axios.post("https://wordpresssite.com/wp-json/wp/v2/media",
formData, {
headers: {
"Content-Disposition": 'form-data; filename="example.jpeg"',
"Content-Type": "image/jpeg",
Authorization: "Bearer <jwt_token>",
},
});
} catch (error) {
console.log(error);
throw new functions.https.HttpsError("failed-precondition", "WP media upload failed");
}
return uploadedMedia.data;
}
I have previously successfully uploaded an image to Wordpress with Javascript in a browser like this:
async function uploadMediaToWordpress() {
let formData = new FormData();
const response = await fetch("https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80");
const blob = await response.blob();
const file = new File([blob], "image.jpeg", { type: blob.type });
formData.append("file", file);
var uploadedMedia = await axios.post("https://wordpresssite.com/wp-json/wp/v2/media",
formData, {
headers: {
"Content-Disposition": 'form-data; filename="example.jpeg"',
"Content-Type": "image/jpeg",
Authorization: "Bearer <jwt_token>",
},
});
return uploadedMedia.data;
},
I have tried the last couple of days to get this to work but cannot for the life of me seem to get it right. Any pointer in the right direction would be greatly appreciated!
The "regular" JavaScript code (used in a browser) works because the image is sent as a file (see the new File in your code), but your Node.js code is not really doing that, e.g. the Content-Type value is wrong which should be multipart/form-data; boundary=----...... Nonetheless, instead of trying (hard) with the arraybuffer response, I suggest you to use stream just as in the axios documentation and form-data documentation.
So in your case, you'd want to:
Set stream as the responseType:
axios.get(
'https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80',
{ responseType: 'stream' }
)
Use formData.getHeaders() in the headers of your file upload request (to the /wp/v2/media endpoint):
axios.post( 'https://wordpresssite.com/wp-json/wp/v2/media', formData, {
headers: {
...formData.getHeaders(),
Authorization: 'Bearer ...'
},
} )
And because the remote image from Unsplash.com does not use a static name (e.g. image-name.jpg), then you'll need to set the name when you call formData.append():
formData.append( 'file', response.data, 'your-custom-image-name.jpeg' );
I hope that helps, which worked fine for me (using the node command for Node.js version 14.15.4, the latest release as of writing).

Node, Multer & Axios : Sending a media to server

I am struggling with a simple media (mp3/mp4) upload to a server using axios.
I have an angular application that creates a formData and send this formData to node server via :
return this.http.post(this.apiURL + '/uploadFile', formData);
My server method looks like this :
app.post('/api/uploadFile', upload.single('file'), (req, res) => {
inputFile = req.file;
let fd = new FormData();
fd.append('file',inputFile.buffer, inputFile.originalname);
axios.post(uploadFileURL , fd, { headers: { 'Content-Type': 'multipart/form-data' } })
.then((response) => {
console.log(response);
})
.catch((error) => {
console.error(error)
})
})
The inputFile contains the original files. The error I get now is that the request is not a multipart request...
I tried as well to define the formData differently :
formData = {
file: {
value: inputFile.buffer,
options: {
filename: inputFile.originalname,
contentType: inputFile.mimetype
}
}
};
Which brought me to a different error : 'Failed to parse multipart servlet request; nested exception is java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadException: the request was rejected because no multipart boundary was found'
Am I doing something wrong ?
I am wondering if this could be link to the fact that I use const bodyParser = require('body-parser'); for some of my other requests.
Any help would be appreciated!
Thanks
EDIT :
Here is my need and what I've done so far :
I have a web application that allow users to upload media files.
I have to send those files to a server, but I can not use the browser to send the request directly.
I created a nodejs application to realize the proxy task of getting the files from the browser and sending it to my remote server.

Resources