How to save records with Asset field use server-to-server cloudkit.js - node.js

I want to use server-to-server cloudkit js. to save record with Asset field.
the Asset field is a m4a audio. after saved, the audio file is corrupt to play
The Apple's Doc is not clear about the Asset field.
In a record that is being saved to the database, the value of an Asset field must be a window.Blob type. In the code fragment above, the type of the assetFile variable is window.File.
Docs:
https://developer.apple.com/documentation/cloudkitjs/cloudkit/database/1628735-saverecords
but in nodejs ,there is no Blob or .File, I filled it with a buffer like this code:
var dstFile = path.join(__dirname,"../test.m4a");
var data = fs.readFileSync(dstFile);
let buffer = Buffer.from(data);
var rec = {
recordType: "MyAttachment",
fields: {
ext: { value: ".m4a" },
file: { value: buffer }
}
}
//console.debug(rec);
mydatabase.newRecordsBatch().create(rec).commit().then(function (response) {
if (response.hasErrors) {
console.log(">>> saveAttachFile record failed");
console.warn(response.errors[0]);
} else {
var createdRecord = response.records[0];
console.log(">>> saveAttachFile record success:", createdRecord);
}
});
The record is successful be saved.
But when I download the audio from icloud.developer.apple.com/dashboard .
the audio file is corrupt to play.
What's wrong with it. thank you to reply.

I was having the same problem and have found a working solution!
Remembering that CloudKitJS needs you to define your own fetch method, I implemented a custom one to see what was going on. I then attached a debugger on the custom fetch to inspect the data that was passing through it.
After stepping through the caller, I found that all asset values are transformed using its toString() method only when the library is embedded in NodeJS. This is determined by the absence of the global window object.
When toString() is called on a Buffer, its contents are encoded to UTF-8 (by default), which causes binary assets to become malformed. If you're using node-fetch for your fetch implementation, it supports Buffer and stream.Readable, so this toString() call does nothing but harm.
The most unobtrusive fix I've found is to swap the toString() method on any Buffer or stream.Readable instances passed as an asset field values. You should probably use stream.Readable, by the way, so that you don't load the entire asset in memory when uploading.
Anyway, here's what it looks like in practice:
// Put this somewhere in your implementation
const swizzleBuffer = (buffer) => {
buffer.toString = () => buffer;
return buffer;
};
// Use this asset value instead
{ asset: swizzleBuffer(fs.readFileSync(path)) }
Please be aware that this workaround mutates a Buffer in an ugly way (since Buffer apparently can't be extended). It's probably a good idea to design an API which doesn't use Buffer arguments so that you can mutate instances that only you create yourself to avoid unintended side effects anywhere else in your code.
Also, sure to vendor (make a local copy) of CloudKitJS in your project, as the behavior may change in the future.
ORIGINAL ANSWER
I ran into the same problem and solved it by encoding my data using Base64. It appears that there's a bug in their SDK which mangles Buffer instances containing non-ascii characters (which, um, seems problematic).
Anyway, try something like this:
const assetField = { value: Buffer.from(data.toString('base64')), 'ascii') }
Side note:
You'll need to decode the asset(s) on the device before using them. There's no way to do this efficiently without writing your own routines, as the methods included in Data / NSData instances requires all data to be in memory.
This is a problem with CloudKitJS (and not the native CloudKit client / service), so the other option is to write your own routine to upload assets.
Neither of these options seem particularly great, but rolling your own atleast means there aren't extra steps for clients to take in order to use the asset.

Related

gRPC: How to call a remote procedure with combination of static and stream parameters?

I'm trying to transfer a file using gRPC. I can send the data, broken into chunks, using gRPC stream. I'm looking for way to also transfer the filename with the data. I'm sure there is an obvious solution that I'm missing. But here are a few approaches that I can think of
Sending filename with each chunk, which as the obvious disadvantage of retransmitting the same data. The .proto file will look like
service KeyValueStore {
rpc upload (stream FileData) returns (UploadStatus) {}
}
message FileData {
string filename = 1;
bytes data = 2;
}
Sending the filename as the first chunk. The receiver will need to be aware of such encoding.
But I'm looking for a non-hacky solution.
I was hoping to have a solution like
service KeyValueStore {
rpc upload (FileName, stream FileData) returns (UploadStatus) {}
}
But it's not possible and also discouraged according to answer here
In general, is there a cleaner way to call a procedure with a combination of normal and stream parameters? or achieve the same effect?
The post you linked is correct. Your input will be a single protocol buffer and in general, it should be named something like "FooRequest". The same is true for the response object, which should be called something like "FooResponse". Decoupling the request and response objects from their contents will give you room to change your API in a backwards-compatible manner over time.
The fact that we don't support multiple request types is not a barrier in practice, because protos can be nested arbitrarily. Consider an API like this.
message FileData {
string filename = 1;
bytes data = 2;
}
message UploadRequest {
oneof payload {
string filename = 1;
FileData file_data = 2;
}
}
service KeyValueStore {
rpc upload (stream UploadRequest) returns (UploadResponse) {}
}
Of course, from the server's perspective, it is now possible for a misbehaving client to send a filename in the middle of the stream. Or, conversely, to start sending chunks of data without first sending a filename.
You could decide that a client must send a filename as the first message. Or perhaps it's okay as long as the filename is sent before the stream ends. Or perhaps sending the filename is entirely optional and not sending one will result in a default value for the filename.
Your decision on these points will be part of your API, but will not be enforced automatically by protobuf as an IDL. You'll need to explicitly handle these corner cases in your server code. Please remember though that, since these are API considerations, they should be written somewhere in your protobuf file. Do your very best to ensure that every message, RPC, and field has a clear and concise docstring.

Optimal method for nodejs to hand of image from database to browser

the end result that I need is to send multiple images to a web browser from a database.
The images are stored as blobs.
I know I can stream them out of the database and into a file and then I could just give the url to the file.
I also know I can hand off base64 string to the browser so it can render the image.
My question is which option is the most optimal? Or best practice? Keep in mind that if I go the stream method, I would have to check to see if the image has changed since the last time I displayed it...and if it has changed then I have to restream it out of the database.
I have been playing with the oracldb for node js and was able to successfully extract one blob into a file but I am also having trouble streaming multiple files.
This is a two question post:
Which is the most optimal:
1. Send Base64 string - I kind of like this method because i dont have to worry about streaming out the file and checking if it has changed since it is coming straight from the databse. My concern is can the browser/nodejs handle it? I know those strings can be very large. I could also be sending more than one image at a time.
Stream the blobs into files.
The second part question is how can i get multiple blobs out below is my code on streaming just one file, i found this example from github lobstream1.js
https://raw.githubusercontent.com/oracle/node-oracledb/master/examples/lobstream1.js
Focusing on the code:
// Stream a LOB to a file
var dostream = function(lob, cb) {
if (lob.type === oracledb.CLOB) {
console.log('Writing a CLOB to ' + outFileName);
lob.setEncoding('utf8'); // set the encoding so we get a 'string' not a 'buffer'
} else {
console.log('Writing a BLOB to ' + outFileName);
}
var errorHandled = false;
lob.on(
'error',
function(err) {
console.log("lob.on 'error' event");
if (!errorHandled) {
errorHandled = true;
lob.close(function() {
return cb(err);
});
}
});
lob.on(
'end',
function() {
console.log("lob.on 'end' event");
});
lob.on(
'close',
function() {
// console.log("lob.on 'close' event");
if (!errorHandled) {
return cb(null);
}
});
var outStream = fs.createWriteStream(outFileName);
outStream.on(
'error',
function(err) {
console.log("outStream.on 'error' event");
if (!errorHandled) {
errorHandled = true;
lob.close(function() {
return cb(err);
});
}
});
// Switch into flowing mode and push the LOB to the file
lob.pipe(outStream);
};
Fixed spooling out images with this method, I did change the dostream a bit.
for(var x = 0; x<result.rows.length;x++)
{
outputFileName = x + '.jpg';
console.log(outputFileName);
console.log(x);
var lob = result.rows[x][0];
dostream(lob,outputFileName);
// cb(null,lob);
}
Thank you for any help.
Given all the detail you provided in subsequent comments including the average image size, number of distinct images, memory available to Node.js, number of concurrent users, and the fact that it's "very critical to have the images up to date", here's my initial take...
For the first implementation, stick to the KISS principle and avoid over-engineering. Disable browser caching and don't cache images in Node.js. Instead, rely on the driver and Oracle Database to do the heavy lifting for you.
As for the table storing the images, try to use SecureFile LOBs over BasicFile LOBs (they are known to perform better) if possible. Also, look at the caching options available to both (CACHE, CACHE READS, and NOCACHE). Consider enabling the CACHE READS option based on your stated workload, but work with your DBA to ensure the buffer cache is sized appropriately so you will not impact others.
You can rely on the connection pool's connection request queue to help control how many people are fetching files concurrently. In fact, you might want to create a separate pool just for this purpose so that people fetching LOBs aren't blocking people doing other things in the application. For example, let's say you normally have one connection pool with 10 connections. You could create two connection pools with 5 connections each (use the connection pool cache to make this easy). Then, in the code path that fetches lobs, use the lob pool and use the other pool for everything else.
Given this setup, I'd also recommend NOT streaming the LOBs. Using the driver's ability to buffer the LOBs in Node.js will greatly simplify the code and you should have plenty of memory given such a small number of concurrent users/file fetches.
The biggest problem with this scenario that the images are pretty large and they'll always be flowing from the database through Node.js to the browser. But since you'll be on an internal network, this might not be much of a problem. If it does turn out to be a problem, you can start to add caching in either the browser or Node.js based on what makes the most sense.
Unless you do something like tiling or the base64 inline encoding, each image needs its own URL, so each invocation of node-oracledb would return just one image. You could do some kind of caching by writing to disk, but this seems extra IO - you will need to test to measure your own system's performance and memory requirements. Regarding accessing multiple images in node-oracledb there's some code in https://github.com/oracle/node-oracledb/issues/1041#issuecomment-459002641 that may be useful.

How to capture only the fields modified by user

I am trying to build a logging mechanism, to log changes done to a record. I am currently logging previous and new record. However, as the site is very busy, I expect the logfile to grow seriously huge. To avoid this, I plan to only capture the modified fields only.
Is there a way to capture only the modifications done to a record (in REACT), so my {request.body} will have fewer fields?
My Server-side is build with NODE.JS and the client-side is REACT.
One approach you might want to consider is to add an onChange(universal) or onTextChanged(native) listener to the text field and store the form update in a local state/variables.
Finally, when a user makes an action (submit, etc.) you can send the updated data to the logging module.
The best way I found and works for me is …
on the api server-side, where I handle the update request, before hitting the database, I do a difference between the previous record and {request.body} using lodash and use the result to send to my update database function
var _ = require('lodash');
const difference = (object, base) => {
function changes(object, base) {
return _.transform(object, function (result, value, key) {
if (!_.isEqual(value, base[key])) {
result[key] = (_.isObject(value) && _.isObject(base[key])) ? changes(value, base[key]) : value;
}
});
}
return changes(object, base);
}
module.exports = difference
I saved the above code in a file named diff.js and included it in my server-side file.
It worked good.
Thanks for giving the idea...

Can you pass meta data along with a stream?

When I pipe something like an image file through a stream is there any way to send an meta object along with it?
My server gets sent an image from a user. The image gets pushed through a set of streams that perform various actions.
The final stream emits a data event, it passes the resulting image buffer into a callback but I lose all context for the user. I need to keep the resulting image tied to the user's id and some other meta data.
Ideal:
stream.on('data', function(img, meta){
...
})
Thanks for any possible solutions!
In short, no, there's nothing built into Node.js to support including metadata with streams. You do have some other options, though, including:
You could use a closure to track the meta data separately from the stream. For example:
function handleImage(imageStream) {
var meta = {...};
imageStream.pipe(otherStreams).on('data', function(image) {
// you now have `image` and `meta` variables at your disposal here.
}
}
The downside of this is that the metadata is not available to your otherStreams.
This is a good solution if your other streams are third-party code outside of your control, of if they don't need to know about the metadata.
You could do something similar to HTTP headers, where all the data up to a certain point is meta data, and everything after it is the image. (In HTTP, the deliminator is wherever \n\n occurs first.) All of your streams in the chain have to know about this and handle it though.
If you know your metadata will always be in one chunk and none of your streams split or merge chunks, then you could simplify this a bit and just say that the first (or last) chunk is always metadata.
Switch to an object stream like Amoli mentioned in his answer. Here you would pass {image: imgData, meta: {...}}. You would then have to update your other streams to expect this format.
The main downside of this method, though, is that you either have to pass the metadata multiple times, cache it somewhere for each stream that needs it, or pass your entire image as one chunk (which kind of kills the entire point of "streams"). And, from what I've been told, node.js can optimize text/binary streams better than object streams. So, this probably isn't a good approach for your situation.
https://github.com/dominictarr/mux-demux might be helpful here. It combines multiple streams into one, so you could have separate image and meta streams. I'm not sure how well it would work for your situation though. You'd probably need to update all of your streams to be aware of it.
I know I said that all but the first option require modifying the other streams, but there is a way around that: you could create a generic "stream wrapper" that splits up the image and meta data and passes just the image data through the main stream, and has the meta data bypass it and go on to the next one down the chain. This gets ugly fast though, so probably not the best idea.
Basically, whenever you want to read or write any objects which are not strings or buffers, you’ll need to put your stream into objectMode
Example (source):
function S3Lister (s3, options) {
options || (options = {});
stream.Readable.call(this, { objectMode : true });
this.s3 = s3; // a knox-like client.
this.marker = options.start;
this.connecting = false;
this.ended = false;
}
util.inherits(S3Lister, stream.Readable);
We set the stream to use objectMode as we want to return not just data but also some metadata.
For more information:
Node.js Docs stream object mode
An introduction to nodes streams
I created a module called metastream for this type of thing. (It is in npm).

Access extension data on other pages

I want to save some value (say userid) from my extension and then want to access it on other pages. I cannot do it using cookie because I can access cookie only within the extension.
Is there a way to do that?
I am sorry, it might just be me being thick but I still don't really understand what you are trying to do.
If you are wanting to create a variable that persists across different domains, you need to use chrome.storage. It is similar to HTML5 localStorage but with some significant improvements (persisting across domains being a main one for me). You can read more about setting and getting the values here:
http://developer.chrome.com/extensions/storage.html
If this isn't what you need, please try to give a specific example of what you are trying to do and I will try again.
EDIT: Please see below for examples.
You can set a variable into storage like this (the function is an optional param):
var storage = chrome.storage.local;
var myVal="foo";
storage.set({'myVar': myVal}, function() {
alert('myVar has been saved in local storage');
});
You can retrieve it like this (function is NOT optional):
storage.get('myVar', function(items) {
if (items.myVar) {
alert(items.myVar);
}
});
You can return default values when 'myVar' is undefined by storage.get({'myVar': ''}) - this saves checking with the "if (items.myVar) {" line.
EDIT: Please see below for issues to watch out for.
Firstly, the methods are asynchronous. So...
alert("1");
storage.get('myVar', function(items) {
alert("2");
});
alert("3");
will probably output 1, then 3, then 2. Not a big deal but saves a lot of restructuring later if you know beforehand.
Secondly,
please note that there is a limit to the amount of storage space you have.
For chrome.storage.sync (for which the variable are automatically synced across all of that users browsers) the storage space is very limited at about 100kb. (there are other limits as well, such as a max of 512 separate variables - see the documentation link above). Basically, chrome.storage.sync is for a few small variables only.
chrome.storage.local has much more storage space at about 5mb (unless you set the unlimited storage permission). Unlike chrome.storage.sync there is no other limits on storage chrome.storage.local.
If a set will take you over the storage limit, then it fails.
Thirdly, using a variable as your variable name does NOT work:
var myVal="foo";
var myVarName='myVar';
storage.set({myVarName: myVal}, function() {
alert('THIS WILL NOT WORK');
});
If like me, you really wanted/needed to do something like this, the way to do something like this is by creating an object and assigning it the variables as properties and then putting the object in storage. For example:
var myVarName='myVar';
var mySecondVarName='mySecondVar';
var myVal="foo";
var mySecondVal="bar";
var obj = {};
obj[myVarName] = myVal;
obj[mySecondVarName] = mySecondVal;
//add as many vars as you want to save to the storage
//(as long as you do not exceed the max storage space as mentioned above.)
storage.local.set(obj);
you can then get the values with:
chrome.storage.local.get(myVarName, function (items) {
if (items[myVarName]) {
alert('You got the value that was set to myVarName');
}
});
Or if you wanted the value for mySecondVarName:
chrome.storage.local.get(mySecondVarName, function (items) {
if (items[mySecondVarName]) {
alert('You got the value that was set to mySecondVarName');
}
});
And these can be nested.

Resources