I am trying to set up a transform stream to pipe an image through with GM https://github.com/aheckmann/gm. So I can do something like:
readStream.pipe(resize()).pipe(writeStream);
I have used through2 along with gm to try and achieve this. It works but only parses half the image, leaving a large portion just grey.
'use strict';
var fs = require('fs')
, gm = require('gm').subClass({imageMagick: true})
, through2 = require('through2');
let readStream = fs.createReadStream('landscape.jpg');
let writeStream = fs.createWriteStream('landscape-out.jpg');
let resize = function(width, height) {
return through2(function(chunk, enc, callback){
gm(chunk)
.resize(800)
.gravity('Center')
.crop(800, 500, 0, 0)
.stream((err, stdout, stderr) => {
stdout.on('data', chunk => {
this.push(chunk);
});
stdout.on('end', () => {
callback();
});
});
});
}
readStream.pipe(resize()).pipe(writeStream);
In
through2(function(chunk, enc, callback){
chunk is only a small part of the image.
So the behavior you got seems normal, you are resizing the chunks of the image, not the image itself.
This said, in the doc,
// GOTCHA:
// when working with input streams and any 'identify'
// operation (size, format, etc), you must pass "{bufferStream: true}" if
// you also need to convert (write() or stream()) the image afterwards
// NOTE: this buffers the readStream in memory!
var readStream = fs.createReadStream('/path/to/my/img.jpg');
gm(readStream)
.size({bufferStream: true}, function(err, size) {
this.resize(size.width / 2, size.height / 2)
this.write('/path/to/resized.jpg', function (err) {
if (!err) console.log('done');
});
});
but it s going to buffer the picture in memory, so it s not optimum.
As you are using imagemagick, you shall just let it manage all that part of the processing. And later fs.readStream, the output.
From the doc
var writeStream = fs.createWriteStream('/path/to/my/reformatted.png');
gm('/path/to/my/img.jpg')
.stream('png')
.pipe(writeStream);
Related
I'm using node-resemble-js to compare two PNG images.
The comparison happens without issue and I get a successful/relevant response however I'm having trouble outputting the image diff.
var express = require('express');
var fs = require('fs');
var resemble = require('node-resemble-js');
var router = express.Router();
router.get('/compare', function(req, res, next) {
compareImages(res);
});
var compareImages = function (res) {
resemble.outputSettings({
largeImageThreshold: 0
});
var diff = resemble('1.png')
.compareTo('2.png')
.ignoreColors()
.onComplete(function(data){
console.log(data);
var png = data.getDiffImage();
fs.writeFile('diff.png', png.data, null, function (err) {
if (err) {
throw 'error writing file: ' + err;
}
console.log('file written');
});
res.render('compare');
});
};
module.exports = router;
It writes to diff.png as expected however it's not creating a valid image.
Any ideas where I'm going wrong? Feel like I'm pretty close but just unsure of final piece.
Thanks
Looks like there is a pack() method that needs to be called, which does some work and then streamifies the data. In that case you can buffer the stream and then call writeFile like this:
var png = data.getDiffImage();
var buf = new Buffer([])
var strm = png.pack()
strm.on('data', function (dat) {
buf = Buffer.concat([buf, dat])
})
strm.on('end', function() {
fs.writeFile('diff.png', buf, null, function (err) {
if (err) {
throw 'error writing file: ' + err;
}
console.log('file written');
})
})
or you can just pipe it like this, which is a little simpler:
png.pack().pipe(fs.createWriteStream('diff.png'))
Honestly, your approach made sense to me (grab the Buffer and write it) but I guess that data Buffer attached to what comes back from getDiffImage isn't really the final png. Seems like the docs are a bit thin but there's some info here: https://github.com/lksv/node-resemble.js/issues/4
I am trying to load the dimensions of an image from url. So far I've tried using GraphicsMagick but it gives me ENOENT error.
Here's the code so far I've written.
var gm = require('gm');
...
gm(img.attribs.src).size(function (err, size) {
if (!err) {
if( size.width>200 && size.height>200)
{
console.log('Save this image');
}
}
});
Where img.attribs.src contains the url source path of the image.
Update
value of img.attribs.src
http://rack.1.mshcdn.com/assets/header_logo.v2-30574d105ad07318345ec8f1a85a3efa.png
https://github.com/nodeca/probe-image-size it does exactly you asked about, without heavy dependencies. Also, it downloads only necessary peace of files.
Example:
var probe = require('probe-image-size');
probe('http://example.com/image.jpg', function (err, result) {
console.log(result);
// => {
// width: xx,
// height: yy,
// type: 'jpg',
// mime: 'image/jpeg',
// wUnits: 'px',
// hUnits: 'px'
// }
});
Disclaimer: I am the author of this package.
What you want to do is to download the file first. The easiest way is to use request module. The cool thing is that both request and gm can use streams. The only thing you need to remember when working with streams and gm's identify commands (like size, format, etc) you need to set bufferStream option to true. More info here.
var gm = require('gm');
var request = require('request');
var url = "http://strabo.com/gallery/albums/wallpaper/foo_wallpaper.sized.jpg";
var stream = request(url);
gm(stream, './img.jpg').size({ bufferStream: true }, function (err, size) {
if (err) { throw err; }
console.log(size);
});
You could also download file on disk (using request as well) an then use gm as normal.
I've used this library to perform the operation successfully.
You need the image-size NPM, this work fine at my end hope work at your
var sizeOf = require('image-size');
sizeOf('/images/'+adsImage, function (err, dimensions) {
var imgwidth= dimensions.width;
var imgheight= dimensions.height;
console.log(imgheight +" = = = = "+imgwidth)
if( imgwidth>200 && imgheight>200)
{
console.log('Save this image');
}
});
I wrote a pretty simple function that downloads an image from a given URL, resize it and upload to S3 (using 'gm' and 'knox'), I have no idea if I'm doing the reading of a stream to a buffer correctly. (everything is working, but is it the correct way?)
also, I want to understand something about the event loop, how do I know that one invocation of the function won't leak anything or change the 'buf' variable to another already running invocation (or this scenario is impossible because the callbacks are anonymous functions?)
var http = require('http');
var https = require('https');
var s3 = require('./s3');
var gm = require('gm');
module.exports.processImageUrl = function(imageUrl, filename, callback) {
var client = http;
if (imageUrl.substr(0, 5) == 'https') { client = https; }
client.get(imageUrl, function(res) {
if (res.statusCode != 200) {
return callback(new Error('HTTP Response code ' + res.statusCode));
}
gm(res)
.geometry(1024, 768, '>')
.stream('jpg', function(err, stdout, stderr) {
if (!err) {
var buf = new Buffer(0);
stdout.on('data', function(d) {
buf = Buffer.concat([buf, d]);
});
stdout.on('end', function() {
var headers = {
'Content-Length': buf.length
, 'Content-Type': 'Image/jpeg'
, 'x-amz-acl': 'public-read'
};
s3.putBuffer(buf, '/img/d/' + filename + '.jpg', headers, function(err, res) {
if(err) {
return callback(err);
} else {
return callback(null, res.client._httpMessage.url);
}
});
});
} else {
callback(err);
}
});
}).on('error', function(err) {
callback(err);
});
};
Overall I don't see anything that would break in your code.
Two suggestions:
The way you are combining Buffer objects is a suboptimal because it has to copy all the pre-existing data on every 'data' event. It would be better to put the chunks in an array and concat them all at the end.
var bufs = [];
stdout.on('data', function(d){ bufs.push(d); });
stdout.on('end', function(){
var buf = Buffer.concat(bufs);
})
For performance, I would look into if the S3 library you are using supports streams. Ideally you wouldn't need to create one large buffer at all, and instead just pass the stdout stream directly to the S3 library.
As for the second part of your question, that isn't possible. When a function is called, it is allocated its own private context, and everything defined inside of that will only be accessible from other items defined inside that function.
Update
Dumping the file to the filesystem would probably mean less memory usage per request, but file IO can be pretty slow so it might not be worth it. I'd say that you shouldn't optimize too much until you can profile and stress-test this function. If the garbage collector is doing its job you may be overoptimizing.
With all that said, there are better ways anyway, so don't use files. Since all you want is the length, you can calculate that without needing to append all of the buffers together, so then you don't need to allocate a new Buffer at all.
var pause_stream = require('pause-stream');
// Your other code.
var bufs = [];
stdout.on('data', function(d){ bufs.push(d); });
stdout.on('end', function(){
var contentLength = bufs.reduce(function(sum, buf){
return sum + buf.length;
}, 0);
// Create a stream that will emit your chunks when resumed.
var stream = pause_stream();
stream.pause();
while (bufs.length) stream.write(bufs.shift());
stream.end();
var headers = {
'Content-Length': contentLength,
// ...
};
s3.putStream(stream, ....);
Javascript snippet
function stream2buffer(stream) {
return new Promise((resolve, reject) => {
const _buf = [];
stream.on("data", (chunk) => _buf.push(chunk));
stream.on("end", () => resolve(Buffer.concat(_buf)));
stream.on("error", (err) => reject(err));
});
}
Typescript snippet
async function stream2buffer(stream: Stream): Promise<Buffer> {
return new Promise < Buffer > ((resolve, reject) => {
const _buf = Array < any > ();
stream.on("data", chunk => _buf.push(chunk));
stream.on("end", () => resolve(Buffer.concat(_buf)));
stream.on("error", err => reject(`error converting stream - ${err}`));
});
}
You can easily do this using node-fetch if you are pulling from http(s) URIs.
From the readme:
fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png')
.then(res => res.buffer())
.then(buffer => console.log)
Note: this solely answers "How to read a stream into a buffer?" and ignores the context of the original question.
ES2018 Answer
Since Node 11.14.0, readable streams support async iterators.
const buffers = [];
// node.js readable streams implement the async iterator protocol
for await (const data of readableStream) {
buffers.push(data);
}
const finalBuffer = Buffer.concat(buffers);
Bonus: In the future, this could get better with the stage 2 Array.fromAsync proposal.
// 🛑 DOES NOT WORK (yet!)
const finalBuffer = Buffer.concat(await Array.fromAsync(readableStream));
You can convert your readable stream to a buffer and integrate it in your code in an asynchronous way like this.
async streamToBuffer (stream) {
return new Promise((resolve, reject) => {
const data = [];
stream.on('data', (chunk) => {
data.push(chunk);
});
stream.on('end', () => {
resolve(Buffer.concat(data))
})
stream.on('error', (err) => {
reject(err)
})
})
}
the usage would be as simple as:
// usage
const myStream // your stream
const buffer = await streamToBuffer(myStream) // this is a buffer
I suggest loganfsmyths method, using an array to hold the data.
var bufs = [];
stdout.on('data', function(d){ bufs.push(d); });
stdout.on('end', function(){
var buf = Buffer.concat(bufs);
}
IN my current working example, i am working with GRIDfs and npm's Jimp.
var bucket = new GridFSBucket(getDBReference(), { bucketName: 'images' } );
var dwnldStream = bucket.openDownloadStream(info[0]._id);// original size
dwnldStream.on('data', function(chunk) {
data.push(chunk);
});
dwnldStream.on('end', function() {
var buff =Buffer.concat(data);
console.log("buffer: ", buff);
jimp.read(buff)
.then(image => {
console.log("read the image!");
IMAGE_SIZES.forEach( (size)=>{
resize(image,size);
});
});
I did some other research
with a string method but that did not work, per haps because i was reading from an image file, but the array method did work.
const DISCLAIMER = "DONT DO THIS";
var data = "";
stdout.on('data', function(d){
bufs+=d;
});
stdout.on('end', function(){
var buf = Buffer.from(bufs);
//// do work with the buffer here
});
When i did the string method i got this error from npm jimp
buffer: <Buffer 00 00 00 00 00>
{ Error: Could not find MIME for Buffer <null>
basically i think the type coersion from binary to string didnt work so well.
I suggest to have array of buffers and concat to resulting buffer only once at the end. Its easy to do manually, or one could use node-buffers
I just want to post my solution. Previous answers was pretty helpful for my research. I use length-stream to get the size of the stream, but the problem here is that the callback is fired near the end of the stream, so i also use stream-cache to cache the stream and pipe it to res object once i know the content-length. In case on an error,
var StreamCache = require('stream-cache');
var lengthStream = require('length-stream');
var _streamFile = function(res , stream , cb){
var cache = new StreamCache();
var lstream = lengthStream(function(length) {
res.header("Content-Length", length);
cache.pipe(res);
});
stream.on('error', function(err){
return cb(err);
});
stream.on('end', function(){
return cb(null , true);
});
return stream.pipe(lstream).pipe(cache);
}
in ts, [].push(bufferPart) is not compatible;
so:
getBufferFromStream(stream: Part | null): Promise<Buffer> {
if (!stream) {
throw 'FILE_STREAM_EMPTY';
}
return new Promise(
(r, j) => {
let buffer = Buffer.from([]);
stream.on('data', buf => {
buffer = Buffer.concat([buffer, buf]);
});
stream.on('end', () => r(buffer));
stream.on('error', j);
}
);
}
You can do this by:
async function toBuffer(stream: ReadableStream<Uint8Array>) {
const list = []
const reader = stream.getReader()
while (true) {
const { value, done } = await reader.read()
if (value)
list.push(value)
if (done)
break
}
return Buffer.concat(list)
}
or using buffer consumer
const buf = buffer(stream)
You can check the "content-length" header at res.headers. It will give you the length of the content you will receive (how many bytes of data it will send)
I am trying to use nodeJS to save a processed image stored in a base64 string.
var buff = new Buffer(base64data,'base64');
console.log(base64data);
var stream = fs.createWriteStream('/path/to/thefile.png');
stream.write(buff)
stream.end()
However, the resulting file is empty.
When I take the output of console.log(base64data); and decode it locally, it produces a valid png binary, so why is the file empty?
The file is a 3600x4800 px png file (i.e. it's huge), could this be a factor?
Also, I tried writeFile as well, no luck.
And yes, fs is require('fs')
Thanks
your stream.end() too soon as nothing is written. it is async function remember.
var buff = new Buffer(base64data,'base64');
console.log(base64data);
var stream = fs.createWriteStream('/path/to/thefile.png');
stream.write(buff);
stream.on("end", function() {
stream.end();
});
Better:
var buff = new Buffer(base64data,'base64');
console.log(base64data);
var stream = fs.createWriteStream('/path/to/thefile.png');
stream.write(buff);
stream.end();
stream.on('finish', () => {
//'All writes are now complete.'
});
stream.on('error', (error) => {...});
Using nodejs and imagemagick am able to re-size an image and send it to the browser with this.
var http = require('http'),
spawn = require('child_process').spawn;
http.createServer(function(req, res) {
var image = 'test.jpg';
var convert = spawn('convert', [image, '-resize', '100x100', '-']);
convert.stdout.pipe(res);
convert.stderr.pipe(process.stderr);
}).listen(8080);
The test image is read from the file-system, I want to alter so that test image is a binary string.
var image = 'some long binray string representing an image.......';
My plan is to store the binary strings in Mongodb and read them of dynamically.
Take a look at the node module node-imagemagick. There is the following example on the module's page to resize and image and write it to a file...
var fs = require('fs');
im.resize({
srcData: fs.readFileSync('kittens.jpg', 'binary'),
width: 256
}, function(err, stdout, stderr){
if (err) throw err
fs.writeFileSync('kittens-resized.jpg', stdout, 'binary');
console.log('resized kittens.jpg to fit within 256x256px')
});
You can alter this code to do the following...
var mime = require('mime') // Get mime type based on file extension. use "npm install mime"
, fs = require('fs')
, util = require('util')
, http = require('http')
, im = require('imagemagick');
http.createServer(function (req, res) {
var filePath = 'test.jpg';
fs.stat(filePath, function (err, stat) {
if (err) { throw err; }
fs.readFile(filePath, 'binary', function (err, data) {
if (err) { throw err; }
im.resize({
srcData: data,
width: 256
}, function (err, stdout, stderr) {
if (err) { throw err; }
res.writeHead(200, {
'Content-Type': mime.lookup(filePath),
'Content-Length': stat.size
});
var readStream = fs.createReadStream(filePath);
return util.pump(readStream, res);
});
});
});
}).listen(8080);
Ps. Haven't run the code above yet. Will try do it shortly, but it should give you an idea of how to asynchronously resize and stream a file.
Since you are using spawn() to invoke the ImageMagick command line convert, the normal approach is to write intermediate files to a temp directory where they will get cleaned up either immediately after use or as a scheduled/cron job.
If you want to avoid writing the file to convert, one option to try is base64 encoding your images and using the inline format. This is similar to how images are encoded in some HTML emails or web pages.
inline:{base64_file|data:base64_data}
Inline images let you read an image defined in a special base64 encoding.
NOTE: There is a limit on the size of command-line options you can pass .. Imagemagick docs suggest 5000 bytes. Base64-encoded strings are larger than the original (Wikipedia suggests a rough guide of 137% larger) which could be very limiting unless you're showing thumbnails.
Another ImageMagick format option is ephemeral:
ephemeral:{image_file}
Read and then Delete this image file.
If you want to avoid the I/O passing altogether, you would need a Node.js module that directly integrates a low-level library like ImageMagick or GD rather than wrapping command line tools.
What have you tried so far? You can use GridFS to store the image data and retrieve as a stream from there.. This in C#..Not sure if this helps..
public static void UploadPhoto(string name)
{
var server = MongoServer.Create("mongodb://localhost:27017");
var database = server.GetDatabase("MyDB");
string fileName = name;
using (var fs = new FileStream(fileName, FileMode.Open))
{
var gridFsInfo = database.GridFS.Upload(fs, fileName);
var fileId = gridFsInfo.Id;
//ShowPhoto(filename);
}
}
public static Stream ShowPhoto(string name)
{
var server = MongoServer.Create("mongodb://localhost:27017");
var database = server.GetDatabase("MyDB");
var file = database.GridFS.FindOne(Query.EQ("filename",name));
var stream = file.OpenRead())
var bytes = new byte[stream.Length];
stream.Read(bytes,0,(int)stream.Length);
return stream;
}
You can now use the stream returned by ShowPhoto.