As a toy project for Electron, I'm trying to rewrite a WPF media management application and would like to get up to speed on what would be the best means of loading media (namely images, but later video) through node's fs and get it into a webpage for Electron to render.
Currently I've got a basic working setup using Base64 encoding:
function getImage(imageID, success) {
fs.readFile('f:/pictures/2.jpg', function(err, data) {
if (err) throw err;
success(Buffer.from(data).toString('base64'));
});
}
and rendering it like so in my html:
api.getImage(0, (result) => {
$('#mainContent').append('<img src="data:image/jpeg;base64,' + result + '" />');
});
This doesn't strike me as the most efficient method to do so. Base64 conversions are helpful for cross technology boundaries, but not the most efficient way to do things.
Some other considerations I had:
Host the file directory via node so I can link to the file directly from my webpage (this is not ideal as files can come from anywhere on the local filesystem)
Use node to copy the file from the filesystem to the served directory so I can link to it via html (this would be slow: disk IO and whatnot)
Pass the raw file buffer back to the webpage somehow (this seems like it should be the way to go, but not sure how to render it in "traditional" html)
Considering that I may want to process 50 or 100 images or so at a time (including possible resizing for thumbnail views: something I was able to do in WPF effectively by optimizing the decoding for the BitmapImage), what would be the general approach to follow?
To clarify, I'm not asking so much about async/process blocking as I am the data transfer itself.
Related
I am trying to load images from a local folder using fabric.js in node.
There seems to be very little up to date documentation on how to do this.
Most example use fabric.Image.fromURL(imageurl)
As far as I'm aware, this only works for web urls, not local paths.
Correct me if I'm wrong, but I have tried
fabric.Image.fromURL(imgpath, (img) => {
...
}
which throws the error Coul not load img: /image/path/img.jpg
Where
fs.readFile(imagepath, (err, i) => {
...
})
will successfully read the file, i will be a buffer.
What is the correct way to load a local image.
I know there is a fabric.Image.fromObject but I have no idea what type of object it wants.
I am currently loading the image into a 2d canvas object, converting it with canvas.toDataURL() and putting that url into fabric.Image.fromURL() which works but converting the image to a url is very slow due to large images. There must be a way to load the image directly and avoid this problem.
If you are using fabricjs 3+, that uses the new jsdom, you can use the file urls!
fabric.Image.fromURL(file://${__dirname}${filepath});
Check here on the fabricJS codebase how they handle reading files in browser and node for the visual test images
https://github.com/fabricjs/fabric.js/blob/master/test/lib/visualTestLoop.js#L139
try this one:
fabric.Image.fromURL(require("../../assets/mockup/100.png"), (img) => {...}
the end result that I need is to send multiple images to a web browser from a database.
The images are stored as blobs.
I know I can stream them out of the database and into a file and then I could just give the url to the file.
I also know I can hand off base64 string to the browser so it can render the image.
My question is which option is the most optimal? Or best practice? Keep in mind that if I go the stream method, I would have to check to see if the image has changed since the last time I displayed it...and if it has changed then I have to restream it out of the database.
I have been playing with the oracldb for node js and was able to successfully extract one blob into a file but I am also having trouble streaming multiple files.
This is a two question post:
Which is the most optimal:
1. Send Base64 string - I kind of like this method because i dont have to worry about streaming out the file and checking if it has changed since it is coming straight from the databse. My concern is can the browser/nodejs handle it? I know those strings can be very large. I could also be sending more than one image at a time.
Stream the blobs into files.
The second part question is how can i get multiple blobs out below is my code on streaming just one file, i found this example from github lobstream1.js
https://raw.githubusercontent.com/oracle/node-oracledb/master/examples/lobstream1.js
Focusing on the code:
// Stream a LOB to a file
var dostream = function(lob, cb) {
if (lob.type === oracledb.CLOB) {
console.log('Writing a CLOB to ' + outFileName);
lob.setEncoding('utf8'); // set the encoding so we get a 'string' not a 'buffer'
} else {
console.log('Writing a BLOB to ' + outFileName);
}
var errorHandled = false;
lob.on(
'error',
function(err) {
console.log("lob.on 'error' event");
if (!errorHandled) {
errorHandled = true;
lob.close(function() {
return cb(err);
});
}
});
lob.on(
'end',
function() {
console.log("lob.on 'end' event");
});
lob.on(
'close',
function() {
// console.log("lob.on 'close' event");
if (!errorHandled) {
return cb(null);
}
});
var outStream = fs.createWriteStream(outFileName);
outStream.on(
'error',
function(err) {
console.log("outStream.on 'error' event");
if (!errorHandled) {
errorHandled = true;
lob.close(function() {
return cb(err);
});
}
});
// Switch into flowing mode and push the LOB to the file
lob.pipe(outStream);
};
Fixed spooling out images with this method, I did change the dostream a bit.
for(var x = 0; x<result.rows.length;x++)
{
outputFileName = x + '.jpg';
console.log(outputFileName);
console.log(x);
var lob = result.rows[x][0];
dostream(lob,outputFileName);
// cb(null,lob);
}
Thank you for any help.
Given all the detail you provided in subsequent comments including the average image size, number of distinct images, memory available to Node.js, number of concurrent users, and the fact that it's "very critical to have the images up to date", here's my initial take...
For the first implementation, stick to the KISS principle and avoid over-engineering. Disable browser caching and don't cache images in Node.js. Instead, rely on the driver and Oracle Database to do the heavy lifting for you.
As for the table storing the images, try to use SecureFile LOBs over BasicFile LOBs (they are known to perform better) if possible. Also, look at the caching options available to both (CACHE, CACHE READS, and NOCACHE). Consider enabling the CACHE READS option based on your stated workload, but work with your DBA to ensure the buffer cache is sized appropriately so you will not impact others.
You can rely on the connection pool's connection request queue to help control how many people are fetching files concurrently. In fact, you might want to create a separate pool just for this purpose so that people fetching LOBs aren't blocking people doing other things in the application. For example, let's say you normally have one connection pool with 10 connections. You could create two connection pools with 5 connections each (use the connection pool cache to make this easy). Then, in the code path that fetches lobs, use the lob pool and use the other pool for everything else.
Given this setup, I'd also recommend NOT streaming the LOBs. Using the driver's ability to buffer the LOBs in Node.js will greatly simplify the code and you should have plenty of memory given such a small number of concurrent users/file fetches.
The biggest problem with this scenario that the images are pretty large and they'll always be flowing from the database through Node.js to the browser. But since you'll be on an internal network, this might not be much of a problem. If it does turn out to be a problem, you can start to add caching in either the browser or Node.js based on what makes the most sense.
Unless you do something like tiling or the base64 inline encoding, each image needs its own URL, so each invocation of node-oracledb would return just one image. You could do some kind of caching by writing to disk, but this seems extra IO - you will need to test to measure your own system's performance and memory requirements. Regarding accessing multiple images in node-oracledb there's some code in https://github.com/oracle/node-oracledb/issues/1041#issuecomment-459002641 that may be useful.
I'm trying to save a remote image file into a database, but I'm having some issues with it since I've never done it before.
I need to download the image and pass it along (with node-request) with a few other properties to another node api that saves it into a mysql database (using sequelize). I've managed to get some data to save, but when I download it manually and try to open it, it's not really usable and no image shows up.
I've tried a few things: getting the image with node-request, converting it to a base64 string (read about that somewhere) and passing it along in a json payload, but that didn't work. Tried sending it as a multipart, but that didn't work either. Haven't worked with streams/buffers/multipart all that much before and never in node. I've tried looking into node-request pipes, but I couldn't really figure out how possibly apply them to this context.
Here's what I currently have (it's a part es6 class so there's no 'function' keywords; also, request is promisified):
function getImageData(imageUrl) {
return request({
url: imageUrl,
encoding: null,
json: false
});
}
function createEntry(entry) {
return getImageData(entry.image)
.then((imageData) => {
entry.image_src = imageData.toString('base64');
var requestObject = {
url: 'http://localhost:3000/api/entry',
method: 'post',
json: false,
formData: entry
};
return request(requestObject);
});
}
I'm almost 100% certain the problem is in this part because the api just takes what it gets and gives it to sequelize to put into the table, but I could be wrong. Image field is set as longblob.
I'm sure it's something simple once I figure it out, but so far I'm stumped.
This is not a direct answer to your question but it is rarely needed to actually store an image in the database. What is usually done is storing an image on storage like S3, a CDN like CloudFront or even just in a file system of a static file server, and then storing only the file name or some ID of the image in the actual database.
If there is any chance that you are going to serve those images to some clients then serving them from the database instead of a CDN or file system will be very inefficient. If you're not going to serve those images then there is still very little reason to actually put them in the database. It's not like you're going to query the database for specific contents of the image or sort the results on the particular serialization of an image format that you use.
The simplest thing you can do is save the images with a unique filename (either a random string, UUID or a key from your database) and keep the ID or filename in the database with other data that you need. If you need to serve it efficiently then consider using S3 or some CDN for that.
I am working on more of a security dashboard, it watches for changes in files in the entire home directory with hundreds of sites (all Joomla, so a lot of files).
In order to keep on top of potential security issues we want to watch for file changes in an efficient way without creating unnecessary CPU/Memory overhead. We want to watch it at a faster interval but I know its more of a balancing act when you do want to keep it from using more cpu then a side process should.
I have tried to use "watch" with the following code, running in the home directory:
var watch, fs;
watch = require('watch');
fs = require('fs');
watch.createMonitor(__dirname,{interval:500,filter:function(file,stat){
if(file.indexOf('index.php')!=-1){
return true;
}else{
return false;
}
}},function(monitor){
monitor.filter(function(file){
console.log(file);
})
monitor.on('created',function(file,stat){
console.log(file + ' new');
});
monitor.on('changed',function(file,stat){
console.log(file + ' changed');
});
monitor.on('removed',function(file,stat){
console.log(file + ' deleted');
});
});
However this spikes the CPU to over 100% of a single core (sometimes 2) out of 8. Memory also takes up about 20% of 8gb pretty quickly as well. This is all just to create the watch event on all the files, so its before it can actually detect any file changes.
I know the issue with this is it goes through each file individually, and only does not track it if you filter that sort of file. Typically all I need to watch is the index.php in every directory, down to a point that it could be consistent (with some exceptions).
Is there a module already built to do this? Or is this something new? All modules I find assume its a smaller directory (like watching LESS or something) So not built for this sort of application at all.
Any ideas? I know this code will need to be scrapped as there is no way I can see to stop the CPU overhead.
Do not use package 'watch', just use fs.watch(...)
package 'watch':
consistent APIs
very slow because implement mostly in node, look source to see how it work
souce code: https://github.com/mikeal/watch/blob/master/main.js
fs.watch(..)
non-consistent APIs, not all OSs are supported.
very fast because it reused OS features
document: http://nodejs.org/docs/latest/api/fs.html#fs_fs_watch_filename_options_listener
Long-time reader, first-time poster.
I am using node v0.6.6 on OS X 10.7. I have not yet tried this in any other environment. I am using this client: https://github.com/elbart/node-memcache
When I use the following code, data randomly contains a few more bytes (as reported by console.log()), which leads to this image: http://imgur.com/NuaK4 (and many other JPG do this). favicon seems OK and HTML/CSS/javascript all work.
In other words: if I request the image, ~70% of the time the image is returned correctly; the other 30% - data reports a few more bytes and the image appears corrupt in the browser.
client.get(key, function(err, data) {
if (err) throw err;
if (data) {
res.writeHead(200, {'Content-Type': type, 'Content-Length': data.length});
console.log('Sending with length: ' + data.length);
res.end(data, 'binary');
}
});
I have been messing with this for several hours and I can honestly say I am stumped. I am hoping someone can show me the error in my ways. I tried searching if there was a way to properly store binary data with memcache but there's no relevant information.
Extra information: it happens with various JPG images; all images are around 100-300KB or less in filesize. For example, one image has reported the following sizes: 286442, 286443, 286441. This problem DOES NOT occur if I straight read data from disk and serve it with node.
Thanks in advance.
Edit I updated my node version and issue persists. Actual test source photo and corrupt photo can be found in my comment below (stackoverflow doesn't permit more links).
Elbart's node-memcache does not handle binary values correctly for the reasons Steve Campbell suggests: node-memcache does not give the client direct access to the buffer. By stringifying the buffers, binary data is corrupted.
Use the 'mc' npm. ( npm install mc )
Caveat: I'm the author of the 'mc' npm. I wrote it specifically to handle binary values over memcache's text protocol.