CouchDB/Cradle How do I add images? - node.js

So Basically I want to let the user have the option of uploading an image when they register. I have no idea where to start forever. I know that CouchDB supports attachments, but how exactly does that work with Cradle.
I found the following code in Cradle's documentation
saveAttachment: function (/* id, [rev], attachmentName, contentType, dataOrStream */) {
So I know it can save attachments. How would I pass in the image then? I'm assuming that in the html, i have to use
form(action='/upload', enctype='multipart/form-data', method='post')
input(type='file', name='upload')
input(type='submit', value='Upload')
But where do I go from there? Wouldn't this step save the image on the server somewhere. Then do I somehow need to get the address of the image and pass that to cradle to save it as an attachment in the CouchDB database.
Thanks in advance if you can help me out!

You need to take the incoming stream from the form and then send a stream to CouchDB via Cradle.
Sending the stream to Cradle is probably the easy bit. This example shows how to do it with a local file:
db.saveAttachment(
doc.id,
doc.rev,
attachmentId,
mimetype,
fs.createReadStream(path),
function( err, data ){
console.log(data);
}
);
The trickier bit in my opinion is managing incoming files. They arrive as a multipart stream rather than being saved to a file. My preference would be to outsource that code to formidable, either directly, or indirectly via connect-form if you're using Connect or Express.
My current connect-form code can be summarised to this:
req.form.complete(function(err, fields, files){
if ( err ) // handle err
else
{
db.saveAttachment(
doc.id,
doc.rev,
attachmentId,
mimetype,
fs.createReadStream(files.name),
function( err, data ){
console.log(data);
}
);
}
});
This isn't optimal for speed as it creates an actual file on disk rather than streaming data from one place to the other, but it is convenient and may satisfy a lot of use cases.
Another package you should be aware of if you're dealing with image upload is node-imagemagick, which as you might expect from the name is the node.js wrapper for ImageMagick.

I wrote up some documentation on attachments that hopefully will be merged into the cradle readme soon. For now though here is the relevant section
Attachments
Cradle supports writing, reading, and removing attachments. The read and write operations can be either buffered or streaming
Writing
You can buffer the entire attachment body and send it all at once as a single request. The callback function will fire after the attachment upload is complete or an error occurs
Syntax
db.saveAttachment(idData, attachmentData, callbackFunction)
Example
Say you want to save a text document as an attachment with the name 'fooAttachment.txt' and the content 'Foo document text'
var doc = <some existing document>
var id = doc._id
var rev = doc._rev
var idAndRevData = {
id: id,
rev: rev
}
var attachmentData = {
name: 'fooAttachment.txt',
'Content-Type': 'text/plain',
body: 'Foo document text'
}
db.saveAttachment(idAndRevData, attachmentData, function (err, reply) {
if (err) {
console.dir(err)
return
}
console.dir(reply)
})
Streaming
You can use a read stream to upload the attachment body rather than buffering the entire body first. The callback function will fire after the streaming upload completes or an error occurs
Syntax
var doc = savedDoc // <some saved couchdb document which has an attachment>
var id = doc._id
var rev = doc._rev
var idAndRevData = {
id: id,
rev: rev
}
var attachmentData = {
name: attachmentName // something like 'foo.txt'
'Content-Type': attachmentMimeType // something like 'text/plain', 'application/pdf', etc.
body: rawAttachmentBody // something like 'foo document body text'
}
var readStream = fs.createReadStream('/path/to/file/')
var writeStream = db.saveAttachment(idData, attachmentData, callbackFunction)
readStream.pipe(writeStream)
When the streaming upload is complete the callback function will fire
Example
Attach a pdf file with the name 'bar.pdf' located at path './data/bar.pdf' to an existing document
var path = require('path')
var fs = require('fs')
// this document should already be saved in the couchdb database
var doc = {
_id: 'fooDocumentID',
_rev: 'fooDocumentRev'
}
var idData = {
id: doc._id,
rev: doc._rev
}
var filename = 'bar.pdf' // this is the filename that will be used in couchdb. It can be different from your source filename if desired
var filePath = path.join(__dirname, 'data', 'bar.pdf')
var readStream = fs.createReadStream
// note that there is no body field here since we are streaming the upload
var attachmentData = {
name: 'fooAttachment.txt',
'Content-Type': 'text/plain'
}
db.saveAttachment(idData, attachmentData, function (err, reply) {
if (err) {
console.dir(err)
return
}
console.dir(reply)
}, readStream)
Reading
Buffered
You can buffer the entire attachment and receive it all at once. The callback function will fire after the download is complete or an error occurs. The second parameter in the callback will be the binary data of the attachment
Syntax
db.getAttachment(documentID, attachmentName, callbackFunction)
Example
Say you want to read back an attachment that was saved with the name 'foo.txt'
var doc = <some saved document that has an attachment with name *foo.txt*>
var id = doc._id
var attachmentName = 'foo.txt'
db.getAttachment(id, attachmentName, function (err, reply) {
if (err) {
console.dir(err)
return
}
console.dir(reply)
})
Streaming
You can stream the attachment as well. If the attachment is large it can be useful to stream it to limit memory consumption. The callback function will fire once the download stream is complete. Note that there is only a single error parameter passed to the callback function. The error is null is no errors occured or an error object if there was an error downloading the attachment. There is no second parameter containing the attachment data like in the buffered read example
Syntax
var readStream = db.getAttachment(documentID, attachmentName, callbackFunction)
Example
Say you want to read back an attachment that was saved with the name 'foo.txt'. However the attachment foo.txt is very large so you want to stream it to disk rather than buffer the entire file into memory
var doc = <some saved document that has an attachment with name *foo.txt*>
var id = doc._id
var attachmentName = 'foo.txt'
var downloadPath = path.join(__dirname, 'foo_download.txt')
var writeStream = fs.createWriteStream(downloadPath)
var readStream = db.getAttachment('piped-attachment', 'foo.txt', function (err) { // note no second reply paramter
if (err) {
console.dir(err)
return
}
console.dir('download completed and written to file on disk at path', downloadPath)
})
readStream.pipe(writeStream)
Removing
You can remove uploaded attachments with a _id and an attachment name
Syntax
db.removeAttachment(documentID, attachmentName, callbackFunction)
Example
Say you want to remove an attachment that was saved with the name 'foo.txt'
var doc = <some saved document that has an attachment with name *foo.txt*>
var id = doc._id
var attachmentName = 'foo.txt'
db.removeAttachment(id, attachmentName, function (err, reply) {
if (err) {
console.dir(err)
return
}
console.dir(reply)
})

FYI, for future readers, the calling parameters have changed since then, so this appears to be no longer valid. Check the source as the documentation doesn't describe how to use it.

Related

Generating QR codes in serverless with aws-nodejs template

I need to write a lambda function and send a number in the api request field to genrate the number of QR codes and store them in a S3 bucket.I am using the serverless framework with the aws-nodejs template.
To describe the task briefly lets say I get a number input in the api request PathParameters and based on these number I have to generate those number of QR's using the qr npm package and then store these generated qr's in the s3 bucket
this is what i have been able to do so far.
module.exports.CreateQR = (event,context) =>{
const numberOfQR = JSON.parse(event.pathParameters.number) ;
for(let i=0;i<numberOfQR;i++){
var d= new Date();
async function createQr(){
let unique, cipher, raw, qrbase64;
unique = randomize('0', 16);
cipher = key.encrypt(unique);
raw = { 'version': '1.0', data: cipher, type: 'EC_LOAD'}
// linkArray.forEach( async (element,index) =>{
let qrcode = await qr.toDataURL(JSON.stringify(raw));
console.log(qrcode);
// fs.writeFileSync('./qr.html', `<img src="${qrcode}">`)
const params = {
Bucket:BUCKET_NAME,
Key:`QR/${d}/img${i+1}.jpg`,
Body: qrcode
};
s3.upload(params , function(err,data){
if(err){
throw err
}
console.log(`File uploaded Successfully .${data.Location}`);
});
}
createQr();
}
};
I have been able to upload a given number of images to the bucket but the issue i am facing is the images are not going in the order.I think the problem is with the asynchronous code. Any idea how to solve this issue
that's because you're not awaiting the s3 to upload, but instead you have a callback.
you should use the .promise of s3 and then await it, so you'll wait the file to be uploaded before move to the next one
I changed the example of code
See docs:
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html#promise-property
// see the async keyword before the lambda function
// we need it for use the await keyword and wait for a task to complete before continue
module.exports.CreateQR = async (event,context) =>{
const numberOfQR = JSON.parse(event.pathParameters.number) ;
// moved your function out of the loop
function createQr(){
// ...
// here we call .promise and return it so we get a Task back from s3
return s3.upload(params).promise();
}
for(let i=0;i<numberOfQR;i++){
// ....
// here we await for the task, so you will get the images being created and uploaded in order
await createQr();
}
};
hope it guide you towards a solution.

How to send File through Websocket along with additional info?

I'm developing a Web application to send images, videos, etc. to two monitors from an admin interface. I'm using ws in Node.js for the server side. I've implemented selecting images available on the server and external URLs and sending them to the clients, but I also wanted to be able to directly send images selected from the device with a file input. I managed to do it using base64 but I think it's pretty inefficient.
Currently I send a stringified JSON object containing the client to which the resource has to be sent, the kind of resource and the resource itself, parse it in the server and send it to the appropriate client. I know I can set the Websocket binaryType to blob and just send the File object, but then I'd have no way to tell the server which client it has to send it to. I tried using typeson and BSON to accomplish this, but it didn't work.
Are there any other ways to do it?
You can send raw binary data through the WebSocket.
It's quite easy to manage.
One option is to prepend a "magic byte" (an identifier that marks the message as non-JSON). For example, prepend binary messages with the B character.
All the server has to do is test the first character before collecting the binary data (if the magic byte isn't there, it's probably the normal JSON message).
A more serious implementation will attach a header after the magic byte (i.e., file name, total length, position of data being sent etc').
This allows the upload to be resumed on disconnections (send just the parts that weren't acknowledged as received.
Your server will need to split the data into magic byte, header and binary_data before processing. but it's easy enough to accomplish.
Hope this help someone.
According to socket.io document you can send either string, Buffer or mix both of them
On Client side:
function uploadFile(e, socket, to) {
let file = e.target.files[0];
if (!file) {
return
}
if (file.size > 10000000) {
alert('File should be smaller than 1MB')
return
}
var reader = new FileReader();
var rawData = new ArrayBuffer();
reader.onload = function (e) {
rawData = e.target.result;
socket.emit("send_message", {
type: 'attachment',
data: rawData
} , (result) => {
alert("Server has received file!")
});
alert("the File has been transferred.")
}
reader.readAsArrayBuffer(file);
}
on server side:
socket.on('send_message', async (data, cb) => {
if (data.type == 'attachment') {
console.log('Found binary data')
cb("Received file successfully.")
return
}
// Process other business...
});
I am using pure WebSocket without io, where you cannot mix content - either String or Binary. Then my working solution is like this:
CLIENT:
import { serialize } from 'bson';
import { Buffer } from 'buffer';
const reader = new FileReader();
let rawData = new ArrayBuffer();
ws = new WebSocket(...)
reader.onload = (e) => {
rawData = e.target.result;
const bufferData = Buffer.from(rawData);
const bsonData = serialize({ // whatever js Object you need
file: bufferData,
route: 'TRANSFER',
action: 'FILE_UPLOAD',
});
ws.send(bsonData);
}
Then on Node server side, the message is catched and parsed like this:
const dataFromClient = deserialize(wsMessage, {promoteBuffers: true}) // edited
fs.writeFile(
path.join('../server', 'yourfiles', 'yourfile.txt'),
dataFromClient.file, // edited
'binary',
(err) => {
console.log('ERROR!!!!', err);
}
);
The killer is promoteBuffer option in deserialize function.

How to save pdf to android file system and then view PDF - react-native

I am using the react-native-fs and I am trying to save a base64 of a pdf file to my android emulators file system.
I receive base64 encoded pdf from the server.
I then decode the base64 string with the line:
var pdfBase64 = 'data:application/pdf;base64,'+base64Str;
saveFile() function
saveFile(filename, pdfBase64){
// create a path you want to write to
var path = RNFS.DocumentDirectoryPath + '/' + filename;
// write the file
RNFS.writeFile(path, base64Image, 'base64').then((success) => {
console.log('FILE WRITTEN!');
})
.catch((err) => {
console.log("SaveFile()", err.message);
});
}
Error
When I try saving the pdfBase64 the saveFile() function catches the following error:
bad base-64
Question
Can anyone tell where or what I am doing wrong?
Thanks.
For anyone having the same problem, here is the solution.
Solution
react-nativive-pdf-view must take the file path to the pdf_base64.
Firstly, I used the react-native-fetch-blob to request the pdf base64 from the server.(Because RN fetch API does not yet support BLOBs).
Also I discovered that react-native-fetch-blob also has a FileSystem API which is way better documented and easier to understand than the 'react-native-fs' library. (Check out its FileSystem API documentation)
Receiving base64 pdf and saving it to a file path:
var RNFetchBlob = require('react-native-fetch-blob').default;
const DocumentDir = RNFetchBlob.fs.dirs.DocumentDir;
getPdfFromServer: function(uri_attachment, filename_attachment) {
return new Promise((RESOLVE, REJECT) => {
// Fetch attachment
RNFetchBlob.fetch('GET', config.apiRoot+'/app/'+uri_attachment)
.then((res) => {
let base64Str = res.data;
let pdfLocation = DocumentDir + '/' + filename_attachment;
RNFetchBlob.fs.writeFile(pdfLocation, pdf_base64Str, 'base64');
RESOLVE(pdfLocation);
})
}).catch((error) => {
// error handling
console.log("Error", error)
});
}
What I was doing wrong was instead of saving the pdf_base64Str to the file location like I did in the example above. I was saving it like this:
var pdf_base64= 'data:application/pdf;base64,'+pdf_base64Str;
which was wrong.
Populate PDF view with file path:
<PDFView
ref={(pdf)=>{this.pdfView = pdf;}}
src={pdfLocation}
style={styles.pdf}
/>
There is a new package to handle the fetching (based on react-native-fetch-blob) and displaying of the PDF via URL: react-native-pdf.
Remove application type in base64 string and it's working for me
var pdfBase64 = 'data:application/pdf;base64,'+base64Str;
To
var pdfBase64 = base64Str;

hitting a multipart url in nodejs

I have a client code using form-data module to hit a url that returns a content-type of image/jpeg. Below is my code
var FormData = require('form-data');
var fs = require('fs');
var form = new FormData();
//form.append('POLICE', "hello");
//form.append('PAYSLIP', fs.createReadStream("./Desert.jpg"));
console.log(form);
//https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xfp1/v/t1.0- 1/c8.0.50.50/p50x50/10934065_1389946604648669_2362155902065290483_n.jpg?oh=13640f19512fc3686063a4703494c6c1&oe=55ADC7C8&__gda__=1436921313_bf58cbf91270adcd7b29241838f7d01a
form.submit({
protocol: 'https:',
host: 'fbcdn-profile-a.akamaihd.net',
path: '/hprofile-ak-xfp1/v/t1.0-1/c8.0.50.50/p50x50/10934065_1389946604648669_2362155902065290483_n.jpg?oh=13640f19512fc3686063a3494c6c1&oe=55ADCC8&__gda__=1436921313_bf58cbf91270adcd7b2924183',
method: 'get'
}, function (err, res) {
var data = "";
res.on("data", function (chunks) {
data += chunks;
});
res.on("end", function () {
console.log(data);
console.log("Response Headers - " + JSON.stringify(res.headers));
});
});
I'm getting some chunk data and the response headers i received was
{"last-modified":"Thu, 12 Feb 2015 09:49:26 GMT","content-type":"image/jpeg","timing-allow-origin":"*","access-control-allow-origin":"*","content-length":"1443","cache-control":"no-transform, max-age=1209600","expires":"Thu, 30 Apr 2015 07:05:31 GMT","date":"Thu, 16 Apr 2015 07:05:31 GMT","connection":"keep-alive"}
I am now stuck as how to process the response that i received to a proper image.I tried base64 decoding but it seemed to be a wrong approach any help will be much appreciated.
I expect that data, once the file has been completely downloaded, contains a Buffer.
If that is the case, you should write the buffer as is, without any decoding, to a file:
fs.writeFile('path/to/file.jpg', data, function onFinished (err) {
// Handle possible error
})
See fs.writeFile() documentation - you will see that it accepts either a string or a buffer as data input.
Extra awesomeness by using streams
Since the res object is a readable stream, you can simply pipe the data directly to a file, without keeping it in memory. This has the added benefit that if you download really large file, Node.js will not have to keep the whole file in memory (as it does now), but will write it to the filesystem continuously as it arrives.
form.submit({
// ...
}, function (err, res) {
// res is a readable stream, so let's pipe it to the filesystem
var file = fs.createWriteStream('path/to/file.jpg')
res.on('end', function writeDone (err) {
// File is saved, unless err happened
})
.pipe(file) // Send the incoming file to the filesystem
})
The chunk you got is the raw image. Do whatever it is you want with the image, save it to disk, let the user download it, whatever.
So if I understand your question clearly, you want to download a file from an HTTP endpoint and save it to your computer, right? If so, you should look into using the request module instead of using form-data.
Here's a contrived example for downloading things using request:
var fs = require('fs');
var request = require('request')
request('http://www.example.com/picture.jpg')
.pipe(fs.createWriteStream('picture.jpg'))
Where 'picture.jpg' is the location to save to disk. You can open it up using a normal file browser.

What is the correct way to encode image data in nodejs buffers?

I'm trying to fetch an image, apply a transform and save it in a database like mongodb. Her's my code
var stor = function(inStream, sizeType) {
console.log('entering stor function');
var hashCode = '';
var img = new Buffer(1024 * 1024 * 5 * 1.1, 'binary'); //5mb size + 10% space
var hash = crypto.createHash('sha1');
inStream.on('data', function (chunk){
Buffer.concat([img, chunk]);
hash.update(chunk);
});
inStream.on('end', function() {
hashCode = hash.digest('hex');
var retUrl = "http://playground.com/" + hashCode;
//post this url using requests, set encoding : binary
});
};
server.post('/resize', function(req, res) {
req.accepts('application/json');
console.log('received a resize request for image =', req.body.host + req.body.path);
var request = requests.get({
url: req.body.url,
headers: {'accept-encoding': 'gzip'}
});
//error handling
request.on('response', function (response) {
console.log('succesfully fetched image...');
response.setEncoding('binary');
//save original in stor
stor(response, 'original');
res.writeHead(201);
res.send();
});
});
module.exports = server;
When i do this, where i receive some image from the internet and then save it in my database for future use, the image saved data in the database is not the original image i stored. It is corrupt. I have narrowed the problem down to the encoding of the data I buffer, in the function store( variable 'img'). I did this by directly piping the data from response to the post to database call. I can't do that for my purpose because i need to compute the hash of the image.
I want to know if my assumptions are correct.
Images from the internet can be read as 'binary'.
You can load that data onto a buffer as 'binary'.
PUT the image onto a store with encoding set to 'binary'.
I think one or all of these assumptions are wrong, as i get back only corrupted data back from the database.
The issue was that I was using exec. Exec outputs a buffer. Using spawn solved this issue. Spawn outputs a STREAM, which handles binary correctly. Ofcourse, I also set the encoding to binary as well.

Resources