I'm trying to save a remote image file into a database, but I'm having some issues with it since I've never done it before.
I need to download the image and pass it along (with node-request) with a few other properties to another node api that saves it into a mysql database (using sequelize). I've managed to get some data to save, but when I download it manually and try to open it, it's not really usable and no image shows up.
I've tried a few things: getting the image with node-request, converting it to a base64 string (read about that somewhere) and passing it along in a json payload, but that didn't work. Tried sending it as a multipart, but that didn't work either. Haven't worked with streams/buffers/multipart all that much before and never in node. I've tried looking into node-request pipes, but I couldn't really figure out how possibly apply them to this context.
Here's what I currently have (it's a part es6 class so there's no 'function' keywords; also, request is promisified):
function getImageData(imageUrl) {
return request({
url: imageUrl,
encoding: null,
json: false
});
}
function createEntry(entry) {
return getImageData(entry.image)
.then((imageData) => {
entry.image_src = imageData.toString('base64');
var requestObject = {
url: 'http://localhost:3000/api/entry',
method: 'post',
json: false,
formData: entry
};
return request(requestObject);
});
}
I'm almost 100% certain the problem is in this part because the api just takes what it gets and gives it to sequelize to put into the table, but I could be wrong. Image field is set as longblob.
I'm sure it's something simple once I figure it out, but so far I'm stumped.
This is not a direct answer to your question but it is rarely needed to actually store an image in the database. What is usually done is storing an image on storage like S3, a CDN like CloudFront or even just in a file system of a static file server, and then storing only the file name or some ID of the image in the actual database.
If there is any chance that you are going to serve those images to some clients then serving them from the database instead of a CDN or file system will be very inefficient. If you're not going to serve those images then there is still very little reason to actually put them in the database. It's not like you're going to query the database for specific contents of the image or sort the results on the particular serialization of an image format that you use.
The simplest thing you can do is save the images with a unique filename (either a random string, UUID or a key from your database) and keep the ID or filename in the database with other data that you need. If you need to serve it efficiently then consider using S3 or some CDN for that.
Related
I am trying to send a number of data including an image to a node js server so that I can create a new document and store the image in the file system. I know that I should send a form data after appending the image to the backend and in the backend I should use multer package to read the form data and store the image. This I have implemented and works just fine.
The problem I am facing is that I have not been able to find a good way of sending form data alongside additional data, that are required for creating the new document in a single request. For that I first tried
something like this {...data, ...formData} but it didn't work. Then I tried appending all the additional data inside the form data itself.
formData.append("basicDetails", data);
formData.append("picture", picture);
const res = await axios.post(BASE_URL, formData, {
headers: {
'Authorization': `Bearer ${token}`
}
});
But now, since all the data is in form data, reading the data like req.body.basicDetails and trying the create a new document seems to not work. If I am not mistaken I need to use packages like multer to read form data. But is there a way to just use form data for picture and not for other data without making multiple requests?
I was researching a bit and I found this StackOverflow thread talking about formdata.
They were mentioning the headers, so add:
'Content-Type': 'multipart/form-data'
to your headers and try that.
Hope it works then :)
I am using the One Drive API to grab a file with a node application using the axios library.
I am simply trying to save the file to the local machine (node is running locally).
I use the One Drive API to get the download document link, which does not require authentication (with https://graph.microsoft.com/v1.0/me/drives/[location]/items/[id]).
Then I make this call with the download document link:
response = await axios.get(url);
I receive a JSON response, which includes, among other things, the content-type, content-length, content-disposition and a data element which is the contents of the file.
When I display the JSON response to the console, the data portion looks like this:
data: 'PK\u0003\u0004\u0014\u0000\u0006\u0000\b\u0000\u0000\u0000!\u...'
If the document is simply text, I can save it easily using:
fs.writeFileSync([path], response.data);
But if the file is binary, like a docx file, I cannot figure out how to write it properly. Every time I try it seems to have the wrong encoding. I tried different encodings.
How do I save the file properly based on the type of file retrieved.
Have you tried using an encoding option of fs.writeFileSync of explicitly null, signifying the data is binary?
fs.writeFileSync([path], response.data, {
encoding: null
});
I want to use server-to-server cloudkit js. to save record with Asset field.
the Asset field is a m4a audio. after saved, the audio file is corrupt to play
The Apple's Doc is not clear about the Asset field.
In a record that is being saved to the database, the value of an Asset field must be a window.Blob type. In the code fragment above, the type of the assetFile variable is window.File.
Docs:
https://developer.apple.com/documentation/cloudkitjs/cloudkit/database/1628735-saverecords
but in nodejs ,there is no Blob or .File, I filled it with a buffer like this code:
var dstFile = path.join(__dirname,"../test.m4a");
var data = fs.readFileSync(dstFile);
let buffer = Buffer.from(data);
var rec = {
recordType: "MyAttachment",
fields: {
ext: { value: ".m4a" },
file: { value: buffer }
}
}
//console.debug(rec);
mydatabase.newRecordsBatch().create(rec).commit().then(function (response) {
if (response.hasErrors) {
console.log(">>> saveAttachFile record failed");
console.warn(response.errors[0]);
} else {
var createdRecord = response.records[0];
console.log(">>> saveAttachFile record success:", createdRecord);
}
});
The record is successful be saved.
But when I download the audio from icloud.developer.apple.com/dashboard .
the audio file is corrupt to play.
What's wrong with it. thank you to reply.
I was having the same problem and have found a working solution!
Remembering that CloudKitJS needs you to define your own fetch method, I implemented a custom one to see what was going on. I then attached a debugger on the custom fetch to inspect the data that was passing through it.
After stepping through the caller, I found that all asset values are transformed using its toString() method only when the library is embedded in NodeJS. This is determined by the absence of the global window object.
When toString() is called on a Buffer, its contents are encoded to UTF-8 (by default), which causes binary assets to become malformed. If you're using node-fetch for your fetch implementation, it supports Buffer and stream.Readable, so this toString() call does nothing but harm.
The most unobtrusive fix I've found is to swap the toString() method on any Buffer or stream.Readable instances passed as an asset field values. You should probably use stream.Readable, by the way, so that you don't load the entire asset in memory when uploading.
Anyway, here's what it looks like in practice:
// Put this somewhere in your implementation
const swizzleBuffer = (buffer) => {
buffer.toString = () => buffer;
return buffer;
};
// Use this asset value instead
{ asset: swizzleBuffer(fs.readFileSync(path)) }
Please be aware that this workaround mutates a Buffer in an ugly way (since Buffer apparently can't be extended). It's probably a good idea to design an API which doesn't use Buffer arguments so that you can mutate instances that only you create yourself to avoid unintended side effects anywhere else in your code.
Also, sure to vendor (make a local copy) of CloudKitJS in your project, as the behavior may change in the future.
ORIGINAL ANSWER
I ran into the same problem and solved it by encoding my data using Base64. It appears that there's a bug in their SDK which mangles Buffer instances containing non-ascii characters (which, um, seems problematic).
Anyway, try something like this:
const assetField = { value: Buffer.from(data.toString('base64')), 'ascii') }
Side note:
You'll need to decode the asset(s) on the device before using them. There's no way to do this efficiently without writing your own routines, as the methods included in Data / NSData instances requires all data to be in memory.
This is a problem with CloudKitJS (and not the native CloudKit client / service), so the other option is to write your own routine to upload assets.
Neither of these options seem particularly great, but rolling your own atleast means there aren't extra steps for clients to take in order to use the asset.
I am saving files as Buffers in my mongo database (using mongoose, nodejs, electron). For now, I'm keeping it simple with text-only files. I read a file in using
fs.readFile(filePath, function(err, data) {
if (err) {console.log(err);}
typeof callback == "function" && callback(data);
});
Then I create a new file in my database using the data variable. And, now I have something that looks like BinData(0,"SGVsbG8gV29ybGQK") stored in my mongodb. All is fine so far.
Now, what if I wanted to display that file in the UI? In this case, in Electron? I think there are two steps.
Step 1 The first is bringing this variable out of the DB and into the front-end. FYI: The model is called File and the variable that stores the file contents is called content.
So, I've tried File.content.toString() which gives me Object {type: "Buffer", data: Array[7]} which is not the string I'm expecting. What is happening here? I read here that this should work.
Step 2 Display this file. Now, since I'm only using text files right now, I can just display the string I get once Step 1 is working. But, is there a good way to do this for more complex files? Images and GIFs and such?
You should save the file mime.
And then set response.header MIME
response.setHeader("Content-Type", "image/jpeg");
response.write(binary)
response.end()
I'm trying to store images in mongo after downloading it with request
here is my code which causes a corrupted image to be stored in db.
request('http://test.jpg', function (error, response, image) {
db.images.insert(
{
file_name: 'test.jpg',
image: new Buffer(image)
},
function(err){
//mongojs callback
}
);
});
Please note I am using mongojs module and storing the images in regular document as BinData type.
Also if I write the image to a file, read it then save the image to the database then there is no corruption. But I don't want to do this as is my intention to avoid the file-system altogether.
I'm pretty this has something to do with encoding or buffers but I don't know enough about these to solve my problem.
If you don't want to use a proper image storing solution like gridfs you could base64 encode your images.
How can you encode a string to Base64 in JavaScript?
That gives you a string that you can store in mongo.