I'm trying to store images in mongo after downloading it with request
here is my code which causes a corrupted image to be stored in db.
request('http://test.jpg', function (error, response, image) {
db.images.insert(
{
file_name: 'test.jpg',
image: new Buffer(image)
},
function(err){
//mongojs callback
}
);
});
Please note I am using mongojs module and storing the images in regular document as BinData type.
Also if I write the image to a file, read it then save the image to the database then there is no corruption. But I don't want to do this as is my intention to avoid the file-system altogether.
I'm pretty this has something to do with encoding or buffers but I don't know enough about these to solve my problem.
If you don't want to use a proper image storing solution like gridfs you could base64 encode your images.
How can you encode a string to Base64 in JavaScript?
That gives you a string that you can store in mongo.
Related
I'm working with Etsy api uploading images like this example, and it requires the images be in binary format. Here is how I'm getting the image binary data:
async function getImageBinary(url) {
const imageUrlData = await fetch(url);
const buffer = await imageUrlData.buffer();
return buffer.toString("binary");
}
However Etsy says it is not a valid image file. How can I get the image in the correct format, or make it in a valid binary format?
Read this for a working example of Etsy API
https://github.com/etsy/open-api/issues/233#issuecomment-927191647
Etsy API is buggy and has an inconsistent guide. You might think of using 'binary' encoding for the buffer because the docs saying that the data type is string but you actually don't need to. Just put the default encoding.
Also currently there is a bug for image upload, try to remove the Content-type header. Better read the link above
I wanna upload small files of size lesser than 16MB to MongoDB via BinData type that i came to know is the only default option for smaller files whereas GRIDFS is ideally used for files exceeding 16MB in size.
Unfortunately I didn't easily get proper documentation and examples of uploading files without GridFS on MongoDB docs. The information I found about BinData type is either quite limited or I failed to understand. Going through several similar questions here (that are mostly python based) and elsewhere, I got some idea about usage of BinData but still I'm unable to successfully upload the files in this way.
I need more information about uploading files via BinData and especially the right way to initialise as I usually get BinData not a function or BinData is not defined errors. Here's my current code where I'm testing the functionality:
import { Meteor } from "meteor/meteor";
import { Mongo } from "meteor/mongo";
export const Attachment = new Mongo.Collection("attachment");
let BinData = Mongo.BinData; //wrong initialisation
function createAttachment(fileData) {
const data = new Buffer(fileData, "base64");
Attachment.insert({file: new BinData(0, data)});
}
Some helpful links:
BSON Types in Mongo
BSON spec
There are several Meteor Packages that you can use for file uploading.
I have used this one myself https://atmospherejs.com/vsivsi/file-collection
It can store your files in gridfs, and provides urls for retrieving images etc
Also:
https://atmospherejs.com/jalik/ufs
https://atmospherejs.com/ostrio/files
I'm trying to save a remote image file into a database, but I'm having some issues with it since I've never done it before.
I need to download the image and pass it along (with node-request) with a few other properties to another node api that saves it into a mysql database (using sequelize). I've managed to get some data to save, but when I download it manually and try to open it, it's not really usable and no image shows up.
I've tried a few things: getting the image with node-request, converting it to a base64 string (read about that somewhere) and passing it along in a json payload, but that didn't work. Tried sending it as a multipart, but that didn't work either. Haven't worked with streams/buffers/multipart all that much before and never in node. I've tried looking into node-request pipes, but I couldn't really figure out how possibly apply them to this context.
Here's what I currently have (it's a part es6 class so there's no 'function' keywords; also, request is promisified):
function getImageData(imageUrl) {
return request({
url: imageUrl,
encoding: null,
json: false
});
}
function createEntry(entry) {
return getImageData(entry.image)
.then((imageData) => {
entry.image_src = imageData.toString('base64');
var requestObject = {
url: 'http://localhost:3000/api/entry',
method: 'post',
json: false,
formData: entry
};
return request(requestObject);
});
}
I'm almost 100% certain the problem is in this part because the api just takes what it gets and gives it to sequelize to put into the table, but I could be wrong. Image field is set as longblob.
I'm sure it's something simple once I figure it out, but so far I'm stumped.
This is not a direct answer to your question but it is rarely needed to actually store an image in the database. What is usually done is storing an image on storage like S3, a CDN like CloudFront or even just in a file system of a static file server, and then storing only the file name or some ID of the image in the actual database.
If there is any chance that you are going to serve those images to some clients then serving them from the database instead of a CDN or file system will be very inefficient. If you're not going to serve those images then there is still very little reason to actually put them in the database. It's not like you're going to query the database for specific contents of the image or sort the results on the particular serialization of an image format that you use.
The simplest thing you can do is save the images with a unique filename (either a random string, UUID or a key from your database) and keep the ID or filename in the database with other data that you need. If you need to serve it efficiently then consider using S3 or some CDN for that.
I have a Node server where, instead of storing cropped images, I want to crop them in response to an AJAX call, and send them to the client that way. I'm storing the information of what to crop and how to crop it in cookies and the body. On the server I crop it, encode it in base64, and send it back the user. Here is what my code looks like
res.set('Content-Type', 'image/jpeg');
gm(request(body.URL))
.crop(req.cookies[name+"Width"],req.cookies[name+"Height"],req.cookies[name+"X"],req.cookies[name+"Y"])
.stream(function streamOut (err, stdout, stderr) {
if (err) return next(err);
stdout.pipe(base64encode()).pipe(res);
stdout.on('error', next);
});
This works, but I don't like it. I was only able to get this to work by encoding it in base64, but on the client side this is seems slow to decode this to an image. I would rather just send an image directly, but I was unable to get this to work. Pipping the image without decoding it resulted in a gibberish response from the server. Is there a better way to do this? Or does the unsaved image have to be encoded in order to send?
I have no idea how you write node.js - it all looks like a bunch of dots and parentheses to me, but using what I know about the GraphicsMagick command line, I tried this and it does what I think you want - which is to write a JPEG encoded result on stdout:
// Send header "Content-type: image/jpeg"...
var gm = require('gm');
var input = 'input.jpg';
gm(input).resize(350).toBuffer('JPG',function (err, buffer) {
if (err) return handle(err);
process.stdout.write(buffer);
})
Update
Have you considered ruling out the AJAX aspects and just using a static src for your image that refers to the node script? As I said, I do not know node and Javascript but if I generate a thumbnail via a PHP script, I would add this into the HTML
<img src="/php/thumb.php"/>
So that just invokes a PHP script to generate an image. If you remove the /php/thumb.php and replace that with however you have named the node script I suggested above, it should tell you whether the problem is the AJAX or the GraphicsMagick aspects...
I am saving files as Buffers in my mongo database (using mongoose, nodejs, electron). For now, I'm keeping it simple with text-only files. I read a file in using
fs.readFile(filePath, function(err, data) {
if (err) {console.log(err);}
typeof callback == "function" && callback(data);
});
Then I create a new file in my database using the data variable. And, now I have something that looks like BinData(0,"SGVsbG8gV29ybGQK") stored in my mongodb. All is fine so far.
Now, what if I wanted to display that file in the UI? In this case, in Electron? I think there are two steps.
Step 1 The first is bringing this variable out of the DB and into the front-end. FYI: The model is called File and the variable that stores the file contents is called content.
So, I've tried File.content.toString() which gives me Object {type: "Buffer", data: Array[7]} which is not the string I'm expecting. What is happening here? I read here that this should work.
Step 2 Display this file. Now, since I'm only using text files right now, I can just display the string I get once Step 1 is working. But, is there a good way to do this for more complex files? Images and GIFs and such?
You should save the file mime.
And then set response.header MIME
response.setHeader("Content-Type", "image/jpeg");
response.write(binary)
response.end()