i am struggling to get this stream downloading file works
if i try the same request using curl the stream works fine and streaming the data.
in my angular app the file is completely download before the client see the download file tab it seems like the subscribe only happened after all stream body fully downloaded
what i am trying to achive is after first chunk of data send to angular app
i want the file to start downloading.
instead after observing the network only after all file downloaded from the backend
the downLoadFile methood is called and the ui expirence is stuck
this is a minimal example of what i am trying todo
at the backend i have a genrator that genreate a huge file
and pipe the request
Node JS
const FILE = './HUGE_FILE.csv'
const lines = await fs.readFileSync(FILE, 'utf8').split('\n')
function* generator() {
for (const i of files) {
console.log(i)
yield i
}
}
app.get('/', async function (req, res) {
res.setHeader('Content-Type', 'text/csv');
res.setHeader('Content-Disposition', 'attachment; filename=\"' + 'download-' + Date.now() + '.csv\"');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Pragma', 'no-cache');
const readable = Readable.from(generator());
readable.pipe(res);
});
at the client side calling the endpoint and waiting for the resonse
Angular code
#Component({
selector: 'is-optimizer-csv',
templateUrl: './optimizer-csv.component.html',
styleUrls: ['./optimizer-csv.component.scss']
})
export class OptimizerCsvComponent implements OnInit {
private onDownloadButtonClicked() {
const data = {...this.form.get('download').value, ...(this.advertiserId ? {advertiserId: this.advertiserId} : null)};
this.loading$.next(true);
this.optimizerService
.downloadOptimizerCsvData(data)
.pipe(
take(1),
tap(_ => this.loading$.next(false))
)
.subscribe();
}
}
#Injectable({
providedIn: 'root'
})
export class OptimizerService {
constructor(private readonly http: Ht, private datePipe: DatePipe) {}
downloadOptimizerCsvData(data: any) {
this.http.get(`${environment.apiUrl}`,{
responseType: 'arraybuffer',headers:headers}
).subscribe(response => this.downLoadFile(response, "text/csv"));
}
downLoadFile(data: any, type: string) {
let blob = new Blob([data], { type: type});
let url = window.URL.createObjectURL(blob);
let pwa = window.open(url);
if (!pwa || pwa.closed || typeof pwa.closed == 'undefined') {
alert( 'Please disable your Pop-up blocker and try again.');
}
}
}
it look like angular common http client does not support streaming out of the box .
see this github issue by #prabh-62
https://github.com/angular/angular/issues/44143
from this thread the why to solve your issue is by implementing the fetch stream logic using native fetch
Related
I'm trying to send a blob image, but I'm getting Error: Unexpected end of form using multer with Serverless Framework.
From console.log
My understanding is I have to append it to FormData before sending it in the body, but I haven't been able to get backend to accept file without crashing
uploadImage(imageData: File) {
console.log('IMAGE DATA', imageData);
let formData = new FormData();
formData.append('file', imageData, 'file.png');
let headers = new HttpHeaders();
headers.append('Content-Type', 'multipart/form-data');
headers.append('Accept', 'application/json');
let options = { headers: headers };
const api = environment.slsLocal + '/add-image';
const req = new HttpRequest('PUT', api, formData, options);
return this.http.request(req);
}
backend
const multerMemoryStorage = multer.memoryStorage();
const multerUploadInMemory = multer({
storage: multerMemoryStorage
});
router.put(
'/add-image',
multerUploadInMemory.single('file'),
async (req, res: Response) => {
try {
if (!req.file || !req.file.buffer) {
throw new Error('File or buffer not found');
}
console.log(`Upload Successful!`);
res.send({
message: 'file uploaded'
});
} catch (e) {
console.error(`ERROR: ${e.message}`);
res.status(500).send({
message: e.message
});
}
console.log(`Upload Successful!`);
return res.status(200).json({ test: 'success' });
}
);
app.ts
import cors from 'cors';
import express from 'express';
import routers from './routes';
const app = express();
import bodyParser from 'body-parser';
app.use(cors({ maxAge: 43200 }));
app.use(
express.json({
verify: (req: any, res: express.Response, buf: Buffer) => {
req.rawBody = buf;
}
})
);
app.use('/appRoutes', routers.appRouter);
app.use(
bodyParser.urlencoded({
extended: true // also tried extended:false
})
);
export default app;
From my understanding with serverless framework I have to install
npm i serverless-apigw-binary
and add
apigwBinary:
types: #list of mime-types
- 'image/png'
to the custom section of the serverless template yaml file.
The end goal is not to save to storage like S3, but to send the image to discord.
What am I missing? I appreciate any help!
I recently encountered something similar in a react native app. I was trying to send a local file to an api but it wasn't working. turns out you need to convert the blob file into a base64 string before sending it. What I had in my app, took in a local file path, converted that into a blob, went through a blobToBase64 function, and then I called the api with that string. That ended up working for me.
I have this code snippet to help you but this is tsx so I don't know if it'll work for angular.
function blobToBase64(blob: Blob) {
return new Promise((resolve, reject) => {
const reader = new FileReader();
reader.onerror = reject;
reader.onload = () => {
resolve(reader.result as string);
};
reader.readAsDataURL(blob);
});
}
Hope this helps!
You can convert your Blob to a File using
new File([blob], "filename")
and then you should be able pass that file to your existing uploadImage method.
Looks like you are passing Blob instead of File based on your console.log(). So you should convert Blob to a File before calling the server. You can change your frontend code like this:
uploadImage(imageData: File) {
// Convert Blob to File
const file = new File([imageData], "file_name", { type: imageData.type });
let formData = new FormData();
formData.append('file', file, 'file.png');
const api = environment.slsLocal + '/add-image';
return this.http.put(api, formData);
}
Note: For more info about converting Blob to File, you can check this StackOverflow question.
The thing that got it working for me was this article.
There might be something different about using Express through Serverless Framework so things like mutler and express-fileupload might not work. Or could be because it's an AWS Lambda function. I don't know this for sure though. I just know I never got it working. This article was the only thing that worked for Serverless Framework + Express.
I also had to install version 0.0.3 of busboy ie npm i busboy#0.0.3. The newer version didn't work for busboy. Newer version was saying Busboy is not a constructor
Since I'm sending the file to discord and not S3 like this article does, I had to tweak the parser.event part in this part of the article for the handler.ts
export const uploadImageRoute = async (
event: any,
context: Context
): Promise<ProxyResult> => {
const parsedEvent: any = await parser(event);
await sendImageToDiscord(parsedEvent.body.file);
const response = {
statusCode: 200,
body: JSON.stringify('file sent successfully')
};
return response;
};
comes in as a Buffer which I was able to send as a file like this
const fs = require('fs-extra');
const cwd = process.cwd();
const { Webhook } = require('discord-webhook-node');
const webhook = new Webhook('<discord-webhook-url>');
export async function sendImageToDiscord(arrayBuffer) {
var buffer = Buffer.from(arrayBuffer, 'base64');
const newFileName = 'nodejs.png';
await fs.writeFile(`./${newFileName}`, buffer, 'utf-8').then(() => {
webhook.sendFile(`${cwd}/${newFileName}`);
});
}
});
I hope this helps someone!
I have MERN application and I want to download a CSV file on button click. I implemented everything but when I click the button there is no download in browser.
Axios call
const handleDownloadCsv = async () => {
try {
await axios.get('http://localhost:2000/users/admin-panel/csv');
} catch (error) {
console.log(error);
}
};
NodeJs controller
export const admin_panel_csv = async (req,res) => {
try {
let __dirname = res.locals.__dirname;
const myReadStream = fs.createReadStream(__dirname + '/documents/admin_csv.csv');
//myReadStream.pipe(res);
res.download(__dirname + '/documents/admin_csv.csv')
} catch (error) {
console.log('Error csv: ',error.message);
res.status(400).json({msg:error.message});
}
}
I've tried both createReadStream(with pipe) and res.download(path to file) but non of them seams to work. I am not getting any errors when making this api call through axios. Is there some way to accomplish this without using React libraries.
There is no download prompt in the browser since you are initiating the download via axios and not through the browser (e.g., through a <form> POST or an <a> click).
Change the back-end code back to res.download and on the front-end, initiate the download through an <a> click:
const handleDownloadCsv = () => {
const tempLink = document.createElement('a')
tempLink.href = 'http://localhost:2000/users/admin-panel/csv'
tempLink.click()
}
I think that you should use js-file-download in React and just :
const FileDownload = require('js-file-download');
Axios.get(API_URL + "download")
.then((response) => {
FileDownload(response.data, 'file.txt');
});
Could someone show me an example of a user basic file download using Node and Angular please. I understand it like this, but this is not working:
Nodejs:
// Does it matter that it is a post and not a get?
app.post('/some-api', someData, (request, response) => {
response.download('file/path/mytext.txt');
});
Angular 2+:
this.httpclient.post<any>('.../some-api', {...data...}).subscribe(response => {
console.log(response);
// This throws an error, but even if it doesn't,
// how do I download the Nodejs `response.download(...) ?`
});
Here are possible answers, but they are so complex, could someone just give me a super basic example (basically what I have here, but a working version). The easiest solution please.
How do I download a file with Angular2
Angular download node.js response.sendFile
There you go..
Node.js Server:
const express = require("express");
const router = express.Router();
router.post("/experiment/resultML/downloadReport",downloadReport);
const downloadReport = function(req, res) {
res.sendFile(req.body.filename);
};
Component Angular:
import { saveAs } from "file-saver"
download() {
let filename = "/Path/to/your/report.pdf";
this.api.downloadReport(filename).subscribe(
data => {
saveAs(data, filename);
},
err => {
alert("Problem while downloading the file.");
console.error(err);
}
);
}
Service Angular:
public downloadReport(file): Observable<any> {
// Create url
let url = `${baseUrl}${"/experiment/resultML/downloadReport"}`;
var body = { filename: file };
return this.http.post(url, body, {
responseType: "blob",
headers: new HttpHeaders().append("Content-Type", "application/json")
});
}
My application using fetch-jsonp is failing to read JSON from a site I serve from a simple http module. I get an error: "Uncaught SyntaxError: Unexpected token :" and then a time-out error.
I'm trying out ReactJS and so put together this react component
import React, { Component } from 'react';
import fetchJsonp from 'fetch-jsonp';
var data = { 'remote':{}, 'local':{} };
var fetched= false;
class FastTable extends Component {
loadData(url,element) {
return fetchJsonp(url)
.then(function(response) {
return response.json(); })
.then((responseJson) => {
data[element] = responseJson;
this.setState(data);
return responseJson;
})
.catch((error) => {
console.error(error);
});
}
render() {
if (!fetched) {
this.loadData('https://jsonplaceholder.typicode.com/posts/1','remote');
this.loadData('http://localhost','local');
}
fetched=true;
return (
<div><pre>{JSON.stringify(data, null, 2) }</pre></div>
);
}
}
export default FastTable;
It uses the fetchJsonp to grab a JSON dataset from a test website, which works - and then from my http test site, which doesn't.
The test server has the following code:
var http = require('http');
http.createServer(function (req, res) {
var result = {
'Bob':'Likes cheese'
};
res.writeHead(200, {'Content-Type': 'application/json'});
res.write(JSON.stringify(result));
res.end();
}).listen(80);
I've also had mixed results reading from other JSON servers within our project.
Why is fetch-jsonp not reading from my test site? Should I be reading this data in another way?
More than likely, you are calling a JSON API, which does not support
JSONP. The difference is that JSON API responds with an object like
{"data": 123} and will throw the error above when being executed as a
function. On the other hand, JSONP will respond with a function
wrapped object like jsonp_123132({data: 123}).
If you want to work with JSON API try axios or supergent npm.
I have an angular2 client and I'm trying to send a file to my NodeJS server via http using a FormData object.
upload.html
...
<input #fileinput type="file" [attr.multiple]="multiple ? true : null" (change)="uploadFile()" >
...
upload.component.ts
uploadFile() {
let files = this.fileinput.nativeElement.files;
if (files.length == 0) return;
this.formData = new FormData();
for (var i = 0; i < files.length; i++) {
console.log("Appended file: " + files[i].name + " File object:" + files[i]); // Output: Appended file: simple.txt File object:[object File]
this.formData.append(files[i].name, files[i]);
}
console.log("uploadFile formData: " + this.formData); // Output: uploadFile formData: [object FormData]
// this.formData.keys() (or any other function) results in TypeError: dataForm.keys is not a function
this.userService
.formDataUpload(this.formData)
.then((response: any) => {
console.log(response);
this.router.navigateByUrl('/');
});
}
user.service.ts
formDataUpload(formData: FormData): Promise<any> {
return this.http.post('/api/v0/formdataupload', formData)
.toPromise()
.then((response: any) => {
return response.json()
})
.catch(this.handleError);
}
server.js
app.post('/api/v0/formdataupload', function(req, res) {
console.log(req.body); // Output: {}
});
How can I access the FormData object in server.js and getthe file object that was uploaded? Also, why am I unable to use any FormData functions in upload.component.ts? All the functions throw TypeError: x is not a function.
Your question is really two questions:
How to POST files from an Angular form to a server?
How to handle POSTed files on a (Node.js) server?
I'm no Node.js expert but a quick search turned up a nice tutorial using the Node.js formidable module to handle file uploads from Node.js.
As for the Angular part, the framework itself isn't of much help as far as file uploads so it all comes down to how would we do that in plain JS?
Your code seems fine. Here is a slightly simplified, working version of it (and a Plunkr):
#Component({
selector: 'my-app',
template: `
<form [formGroup]="myForm">
<p><input type="file" formControlName="file1" (change)="uploadFile($event)"></p>
</form>
`
})
export class AppComponent {
myForm: ControlGroup;
constructor(fb: FormBuilder, private http: Http) {
this.myForm = fb.group({
file1: []
});
}
uploadFile(evt) {
const files = evt.target.files;
if (files.length > 0) {
let file;
let formData = new FormData();
for (let i = 0; i < files.length; i++) {
file = files[i];
formData.append('userfile', file, file.name);
}
this.http.post('https://httpbin.org/post', formData)
.map(resp => resp.json())
.subscribe(data => console.log('response', data));
}
}
}
Only notable difference with your code is the way I access the files property of the input file element. In your code, writing <input #fileinput> in your template doesn't automagically create a fileinput property in your component, maybe that's the problem.
Notes:
I'm using https://httpbin.org/post as the backend to debug the POST request (it returns all the fields it has received in the request).
My file field doesn't have the multiple option but I think the code would work just the same if it did.
In terms of handling files posted to the Node.js server, your issue could well be one of Content-Type. If you are using something like body-parser to handle POSTs on the nodejs server, then it will not understand forms with the content type set to be multipart/form-data, which is what FormData deals in.
Note the comment here:
[body-parser] does not handle multipart bodies, due to their complex and typically large nature. For multipart bodies, you may be interested in the following modules: busboy and connect-busboy; multiparty and connect-multiparty; formidable; multer.
So in other words, you have to user a different module to handle the multipart body that FormData sends. I can recommend formidable, in which case you're server code would look something like:
const formidable = require('formidable')
exports.createPost = (req, res, next) => {
var form = new formidable.IncomingForm();
form.parse(req, (err, fields, files) => {
console.log(fields)
res.send('NOT IMPLEMENTED: pollsController createPost');
}
}