i try to readout an image, saved in mongodb, via gridfs (without temporary file)
it should be directly sent to ajax, which injects it into html
when i use my actual functions a large bit string is formed and sent to client (is saved in ajax response var)
but as it reaches the client, the bits arent correct anymore
so i look for a way to encode the picture before it is sent (into base64)
(or is there any other way?)
Serverside - javascript, gridfs
exports.readFileFromDB = function(req, res, profile, filename, callback){
console.log('Find data from Profile ' + JSON.stringify(profile));
var GridReader = new GridStore(db, filename,"r");
GridReader.open(function(err, gs) {
var streamFile = gs.stream(true);
streamFile.on("end", function(){
});
// Pipe out the data
streamFile.pipe(res);
GridReader.close(function(err, result) {
});
Clientside - javascript ajax call:
function imgUpload(){
var thumb = $("#previewPic");
$('#uploadForm').ajaxSubmit({
beforeSend:function(){
//launchpreloader();
},
error: function(xhr) {
//status('Error: ' + xhr.status);
},
success: function(response) {
console.log(response);
var imageData = $.base64Encode(response);
console.log(imageData);
thumb.attr("src","data:image/png;base64"+imageData);
$("#spanFileName").html("File Uploaded")
}
});
}
I'm doing something similar for a current project, but when the upload is complete, I return a JSON object containing the URL for the uploaded image:
{ success : true, url : '/uploads/GRIDFSID/filename.ext' }
I have a route in Express that handles the /uploads route to retrieve the file from GridFS and stream it back to the client, so I can use the above URL in an IMG SRC. This is effectively what appears in the DOM:
<img src="/uploads/GRIDFSID/filename.ext">
The route handler looks something like this (it uses node-mime and gridfs-stream):
app.get(/^\/uploads\/([a-f0-9]+)\/(.*)$/, function(req, res) {
var id = req.params[0];
var filename = req.params[1];
// Set correct content type.
res.set('Content-type', mime.lookup(filename));
// Find GridFS file by id and pipe it to the response stream.
gridfs
.createReadStream({ _id : id })
.on('error', function(err) {
res.send(404); // or 500
})
.pipe(res);
});
It obviously depends on your exact setup if my solution works for you.
Related
Looking for help on Uploading and Retrieving Images from MongoDb using multer.
My front end is ReactNative.(Not sure if this is needed but just to be sure.)
Multer
Problem: After looking and following tutorials i'm able to encode my path to base64 and upload it to my DB but now i'm confused how to retrieve the file from my DB. I saw some tutorials about decoding it from base64 but I don't quite understand how do I go about retrieving an image and displaying it in postman. (I tried looking but haven't found anything that gives me an answer. I'm sorry if this is a duplicated question. If you could point me in a direction or give me some advice I would be really greatful.)
**POST**
route.post("/sad", upload.single("image"), (req, res, next) => {
console.log(req.file);
const img = fs.readFileSync(req.file.path);
const img_enc = img.toString('base64');
const obj = {
usrImage: {
data: new Buffer.from(img_enc, 'base64'),
contentType: "image/jpg",
},
};
console.log(obj);
const newAccout = new account(obj);
newAccout.save();
});
**RETRIEVE**
route.get('/sad',(req,res)=>{
img.find({}).then((img)=>{
res.json(img)
//How do decode my buffer to show an image in Postman?
})
}
)
I am trying to create a userprofile where a username,password and image is saved. If you can help save an Image and then retrieve it from my accounts collection.
Hey I would advise that you start using a 3rd party for file upload like cloudinary very good way of managing files i.e images or video...
I am not that well of with multer but I can give a quick code example using Formidable does the same work as multer
Before you can start you'd need to make an account on cloudinary.com(don't worry its free)
Code below is how you could handle file upload
const Formidable = require("formidable"); //Meant for body parsing
const cloudinary = require("cloudinary").v2; // file uploader
//This below is your connection/configuration to get access to your cloudinary account so cloud_name, api_key and api_secret you'll get in your home dashboard(Cloudinary)
cloudinary.config({
cloud_name: process.env.CLOUD_NAME,
api_key: process.env.API_KEY,
api_secret: process.env.API_SECRET,
});
router.post('/api/file-upload', (req, res)=>{
const form = new Formidable.InconmingForm();
form.parse(req, (error, fields, files)=>{
const {file} = files
cloudinary.uploader.upload(file.path, {folder:"/"}, (err, res)=>{
const file_url = res.secure_url //This would be the url for your file given back by cloudinary
})
})
})
This script should upload your file and the file_url will be having the url of the file that you upload having ssl then after that you can now continue saving to mongoDB
Cloudinary docs for NodeJS
https://cloudinary.com/documentation/node_integration
Nice clear and understandable docs
Shameless plug
If you get lost you can check this video out on YouTube that I made handling file upload with cloudinary then save url given back to mongoDB
https://youtu.be/mlu-tbr2uUk
First call api find one
you will need fs module to complete following query
const fs = require('fs');
let data = await db.user.findOne({
where: {
id = req.body.id
}
})
// _________________ base 64 string data from findone query data
// |
let buff = new Buffer(data.image, 'base64');
let name = name.jpeg
let path = `tmp/${name}`; // <--- destination and file name you want to give to your file
fs.writeFileSync(path, buff);// < --this will write file to given path
fs.readFile(path, function (err, content) {// <------to send file in postman response
if (err) {
res.writeHead(400)
console.log(err);
res.end("No such image");
} else {
//specify the content type in the response will be an image
res.writeHead(200);
res.end(content);
}
});
fs.unlink(path, (err) => { // <-----to delete file from tmp directory
if (err) {
console.log(err)
}
})
Try this and switch to preview tab in postman.
I haven't tried it but maybe it helps.
route.get('/sad',(req,res)=>{
img.find({}).then((img)=>{
res.setHeader('contentType','image/jpg').send(img)
})
})
How do I create a file in express and node on my server and then download it to my client. I am using NextJS for my frontend and backend. I am confused on how I would download the file on the front end after the file is created on the root of the server folder. Since I am using React for my frontend whenever I try to visit that filepath it tries to take me to a page instead of the file
Here is what I have in my express route in node
var xls = json2xls(json, {
fields
});
// If there isn't a folder called /temp in the
// root folder it creates one
if (!fs.existsSync('./temp')) {
fs.mkdirSync('./temp');
}
const fileName = `temp/${req.user.first_name}${req.body._id + Date.now()}.xlsx`
// fs.writeFileSync(fileName, xls, 'binary');
fs.writeFile(fileName, xls, 'binary', function (err, result) {
if (err) {
return console.log(err);
}
console.log(result, 'this is result')
});
Here is what I have on my frontend
axios.post('api/download',payload)
.then(res => {
const link = document.createElement('a');
link.href = res.data.url;
link.download
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
})
.catch(err => {
throw err
})
Can you make request with GET on api, and.
Make request with GET.
Make temp directory to be static resources directory:
app.use(express.static('temp')); // app is your express instance.
// Maybe you have to correct temp's path
Response the post request with file url data
fs.writeFile(fileName, xls, 'binary', function (err, result) {
if (err) {
return console.log(err);
res.status(500).json({err});
}
console.log(result, 'this is result');
res.json({url: 'http://localhost:8080/temp/' + fileName}); // res is response object of you router handler.
// Maybe you have correct the server address
});
On other way, you can send the xls binary direct to client, in the client you create a BLOB object from the response, then create download link for the blob object.
I'm trying to download a pdf file generated at nodejs server. When I send the data of pdf file then it is coming to angularjs frontend but I've tried blob object to convert into pdf but pdf is showing no data.
Already tried the {responseType: 'arraybuffer'} as mentioned in other answers but still my pdf is not showing any data.
Here is nodejs server which will serve the pdf data:
var pdff = require('html-pdf');
pdff.create(html, options).toFile('./report.pdf', function(err) {
if (err) return console.log(err);
res.contentType("application/pdf");
res.download("./report.pdf");
});
In the above code - report.pdf is generated at the server which is completely fine.
Here is the angularjs code:
app.controller("ExampleCtrl",['FileSaver', 'Blob','$http', function(FileSaver, Blob,$http) {
var vm = this;
vm.consolidate=function(){
$http.post('/mngmnt/consolidate_grv',{responseType: 'arraybuffer'}).then(function(response){
//console.log(response.data);
console.log(response);
//var byteArray = new Uint8Array(response.data);
var blob = new Blob([response.data], { type: 'application/pdf'});
console.log(blob);
var fileURL = URL.createObjectURL(blob);
//FileSaver.saveAs(blob, 'report.pdf');
window.open(fileURL)
});
}
}]);
Everything is working properly but the generated pdf is not showing the same data as shown on the server side.
I'm expecting to generate the same pdf as generated at the server side
app.controller("ExampleCtrl", function(FileSaver, Blob,$http) {
var vm = this;
vm.consolidate=function(){
$http.post('/mngmnt/consolidate_grv', {}, {responseType: 'blob'})
.then(function(response){
//console.log(response.data);
console.log(response);
//var byteArray = new Uint8Array(response.data);
var blob = response.data;
console.log(blob);
var fileURL = URL.createObjectURL(blob);
//FileSaver.saveAs(blob, 'report.pdf');
window.open(fileURL)
});
};
});
If you insist on using a POST request, the second argument to the $http.post method needs to be the data for the body of the request. The options object needs to be the third argument.
I'm new to node.js. What I'm trying to do is to stream the upload of a file from web browser to a cloud storage through my node.js server.
I'm using 'express', 'request' and 'busboy' modules.
var express = require("express");
var request = require("request");
var BusBoy = require("busboy");
var router = express.Router();
router.post("/upload", function(req, res, next) {
var busboy = new BusBoy({ headers: req.headers });
var json = {};
busboy.on("file", function (fieldname, file, filename, encoding, mimetype) {
file.on("data", function(data) {
console.log(`streamed ${data.length}`);
});
file.on("end", function() {
console.log(`finished streaming ${filename}`);
});
var r = request({
url: "http://<my_cloud_storage_api_url>",
method: "POST",
headers: {
"CUSTOM-HEADER": "Hello",
},
formData: {
"upload": file
}
}, function(err, httpResponse, body) {
console.log("uploaded");
json.response = body;
});
});
busboy.on("field", function(name, val) {
console.log(`name: ${name}, value: ${value}`);
});
busboy.on("finish", function() {
res.send(json);
});
req.pipe(busboy);
});
module.exports = router;
But I keep getting the following error on the server. What am I doing wrong here? Any help is appreciated.
Error: Part terminated early due to unexpected end of multipart data
at node_modules\busboy\node_modules\dicer\lib\Dicer.js:65:36
at nextTickCallbackWith0Args (node.js:420:9)
at process._tickCallback (node.js:349:13)
I realize this question is some 7 months old, but I shall answer it here in an attempt help anyone else currently banging their head against this.
You have two options, really: Add the file size, or use something other than Request.
Note: I edited this shortly after first posting it to hopefully provide a bit more context.
Using Something Else
There are some alternatives you can use instead of Request if you don't need all the baked in features it has.
form-data can be used by itself in simple cases, or it can be used with, say, got. request uses this internally.
bhttp advertises Streams2+ support, although in my experience Streams2+ support has not been an issue for me. No built in https support, you have to specify a custom agent
got another slimmed down one. Doesn't have any special handling of form data like request does, but is trivially used with form-data or form-data2. I had trouble getting it working over a corporate proxy, though, but that's likely because I'm a networking newb.
needle seems pretty light weight, but I haven't actually tried it.
Using Request: Add the File Size
Request does not (as of writing) have any support for using transfer-encoding: chunked so to upload files with it, you need to add the file's size along with the file, which if you're uploading from a web client means that client needs to send that file size to your server in addition to the file itself.
The way I came up with to do this is to send the file metadata in its own field before the file field.
I modified your example with comments describing what I did. Note that I did not include any validation of the data received, but I recommend you do add that.
var express = require("express");
var request = require("request");
var BusBoy = require("busboy");
var router = express.Router();
router.post("/upload", function(req, res, next) {
var busboy = new BusBoy({ headers: req.headers });
var json = {};
// Use this to cache any fields which are file metadata.
var fileMetas = {};
busboy.on("file", function (fieldname, file, filename, encoding, mimetype) {
// Be sure to match this prop name here with the pattern you use to detect meta fields.
var meta = fileMetas[fieldname + '.meta'];
if (!meta) {
// Make sure to dump the file.
file.resume();
// Then, do some sort of error handling here, because you cannot upload a file
// without knowing it's length.
return;
}
file.on("data", function(data) {
console.log(`streamed ${data.length}`);
});
file.on("end", function() {
console.log(`finished streaming ${filename}`);
});
var r = request({
url: "http://<my_cloud_storage_api_url>",
method: "POST",
headers: {
"CUSTOM-HEADER": "Hello",
},
formData: {
// value + options form of a formData field.
"upload": {
value: file,
options: {
filename: meta.name,
knownLength: meta.size
}
}
}
}, function(err, httpResponse, body) {
console.log("uploaded");
json.response = body;
});
});
busboy.on("field", function(name, val) {
// Use whatever pattern you want. I used (fileFieldName + ".meta").
// Another good one might be ("meta:" + fileFieldName).
if (/\.meta$/.test(name)) {
// I send an object with { name, size, type, lastModified },
// which are just the public props pulled off a File object.
// Note: Should probably add error handling if val is somehow not parsable.
fileMetas[name] = JSON.parse(val);
console.log(`file metadata: name: ${name}, value: ${value}`);
return;
}
// Otherwise, process field as normal.
console.log(`name: ${name}, value: ${value}`);
});
busboy.on("finish", function() {
res.send(json);
});
req.pipe(busboy);
});
module.exports = router;
On the client, you need to then send the metadata on the so-named field before the file itself. This can be done by ordering an <input type="hidden"> control before the file and updating its value onchange. The order of values sent is guaranteed to follow the order of inputs in appearance. If you're building the request body yourself using FormData, you can do this by appending the appropriate metadata before appending the File.
Example with <form>
<script>
function extractFileMeta(file) {
return JSON.stringify({
size: file.size,
name: file.name,
type: file.type,
lastUpdated: file.lastUpdated
});
}
function onFileUploadChange(event) {
// change this to use arrays if using the multiple attribute on the file input.
var file = event.target.files[0];
var fileMetaInput = document.querySelector('input[name=fileUpload.meta]');
if (fileMetaInput) {
fileMetaInput.value = extractFileMeta(file);
}
}
</script>
<form action="/upload-to-cloud">
<input type="hidden" name="fileUpload.meta">
<input type="file" name="fileUpload" onchange="onFileUploadChange(event)">
</form>
Example with FormData:
function onSubmit(event) {
event.preventDefault();
var form = document.getElementById('my-upload-form');
var formData = new FormData();
var fileUpload = form.elements['fileUpload'];
var fileUploadMeta = JSON.stringify({
size: fileUpload.size,
name: fileUpload.name,
type: fileUpload.type,
lastUpdated: fileUpload.lastUpdated
});
// Append fileUploadMeta BEFORE fileUpload.
formData.append('fileUpload.meta', fileUploadMeta);
formData.append('fileUpload', fileUpload);
// Do whatever you do to POST here.
}
On my node express server, I am receiving a pdf file. I am using the below code to get the pdf contents from the request
var data = new Buffer('');
request.on('data', function (chunk) {
data = Buffer.concat([data, chunk]);
});
request.on('end', function() {
console.log('PDF data is '+JSON.stringify(data));
});
Now that PDF content is available on node, I need to send it as it is to a J2EE server. In order to do that, I am first saving the PDF file in the node server, reading it from the node server and then piping it to request.post (https://github.com/request/request)
var req = require('request');
fs.writeFile('abc.pdf', data, 'binary', function(err) {
if (err) {
console.log('Error ' + JSON.stringify(err) );
throw err;
}
var source = fs.createReadStream('abc.pdf');
//send our data via POST request
source.pipe(req.post('http://'+j2ee_host+':'+j2ee_port+'/myjavaapp/Upload')
});
This works fine. However, I feel the part of saving the PDF file on the node server and then reading it is (before posting to the J2EE server using request module) is completely unnecessary, as I am not making any changes to the file.
Once I have the PDF contents in 'data' variable, I would like to directly post them to the J2EE server. However, I have not been able to find a way to use the request module to directly post file contents. I have seen some examples related to POST using request module but they refer to formData. In my case, I don't have formData but instead reading the file from request and directly posting it to the J2EE server.
Is there a way to achieve this and avoid the file write and read?
EDIT
Below is my complete code
function upload(request, response) {
var data = new Buffer('');
request.on('data', function (chunk) {
data = Buffer.concat([data, chunk]);
});
request.on('end', function () {
fs.writeFile('abc.pdf', data, 'binary', function(err){
if (err) {
console.log('Error ' + JSON.stringify(err) );
throw err;
}
var source = fs.createReadStream('abc.pdf');
source.pipe(req.post('http://'+j2ee_host+':'+j2ee_port+'/myj2eeapp/Upload'));
})
})
}
You can pipe directly from the data request to the servlet
var req = require('request');
function upload(request, response) {
var target = req.post('http://'+j2ee_host+':'+j2ee_port+'/myjavaapp/Upload');
request.pipe(target);
target.on('finish', function () {
console.log('All done!');
//send the response or make a completed callback here...
});
}