I have a really annoying issue. Basically my code works in all other places apart from in this one instance where wait.for doesn't wait for the redis command to finish before moving on. The scenario is I trying to process an XML file and update records within a redis DB as the XML file is being read. The part that is breaking is where I'm trying to retrieve a list of customer card numbers within a region before updating the overall customer record and counts. But it is not waiting.
Here is the code:
function processCredit(creditObj)
{
.....
console.log('CREDIT CARD=1');
wait.launchFiber( updateCustomer, cardNumber );
}
function updateCustomer(cardNumber)
{
var foundCard=0;
console.log('CREDIT CARD=2');
var customerIDs = wait.forMethod(redisClient, 'zrange', 'cn:'+cardNumber+':'+stores[siteId]['region'], 0, -1);
console.log('CREDIT CARD=3 card number='+cardNumber);
....
}
"CREDIT CARD=3" statement always gets printed at the end of the log file, not directly after "CREDIT CARD=2"
I've used wait.for in many other places throughout my programme and it has always worked. I don't understand why it doesn't in this case.
Thanks
Related
thanks for your help, I am new to firebase, I am designing an application with Node.js, what I want is that every time it detects changes in a document, a function is invoked that creates or updates the file system according to the new structure of data in the firebase document, everything works fine but the problem I have is that if the document is updated with 2 or more attributes the makeBotFileSystem function is invoked the same number of times which brings me problems since this can give me performance problems or file overwriting problems since what I do is generate or update multiple files.
I would like to see how the change can be expected but wait until all the information in the document is finished updating, not attribute by attribute, is there any way? this is my code:
let botRef = firebasebotservice.db.collection('bot');
botRef.onSnapshot(querySnapshot => {
querySnapshot.docChanges().forEach(change => {
if (change.type === 'modified') {
console.log('bot-changes ' + change.doc.id);
const botData = change.doc.data();
botData.botId = change.doc.id;
//HERE I CREATE OR UPDATE FILESYSTEM STRUCTURE, ACCORDING Data changes
fsbotservice.makeBotFileSystem(botData);
}
});
});
The onSnapshot function will notify you anytime a document changes. If property changes are commited one by one instead of updating the document all at once, then you will receive multiple snapshots.
One way to partially solve the multiple snapshot thing would be to change the code that updates the document to commit all property changes in a single operation so that you only receive one snapshot.
Nonetheless, you should design the function triggered by the snapshot so that it can handle multiple document changes without breaking. Given that document updates will happen no matter if by single/multiple property changes your code should be able to handle those. IMHO the problem is the filesystem update rather than how many snaphots are received
You should use docChanges() method like this:
db.collection("cities").onSnapshot(querySnapshot => {
let changes = querySnapshot.docChanges();
for (let change of changes) {
var data = change.doc.data();
console.log(data);
}
});
I am trying to build a logging mechanism, to log changes done to a record. I am currently logging previous and new record. However, as the site is very busy, I expect the logfile to grow seriously huge. To avoid this, I plan to only capture the modified fields only.
Is there a way to capture only the modifications done to a record (in REACT), so my {request.body} will have fewer fields?
My Server-side is build with NODE.JS and the client-side is REACT.
One approach you might want to consider is to add an onChange(universal) or onTextChanged(native) listener to the text field and store the form update in a local state/variables.
Finally, when a user makes an action (submit, etc.) you can send the updated data to the logging module.
The best way I found and works for me is …
on the api server-side, where I handle the update request, before hitting the database, I do a difference between the previous record and {request.body} using lodash and use the result to send to my update database function
var _ = require('lodash');
const difference = (object, base) => {
function changes(object, base) {
return _.transform(object, function (result, value, key) {
if (!_.isEqual(value, base[key])) {
result[key] = (_.isObject(value) && _.isObject(base[key])) ? changes(value, base[key]) : value;
}
});
}
return changes(object, base);
}
module.exports = difference
I saved the above code in a file named diff.js and included it in my server-side file.
It worked good.
Thanks for giving the idea...
It's a simple concept but yet finding the right answer is ridiculously stressing. Use Firebase Functions w/ Node.js to pull the "itemsExpires" (date) from the Firebase database. The parent node is the userId (random string) and each item is stored under a key node (another random string).. So, here's what the firebase database looks like:
firebase-database-name
+ 82hfijcbwjbjfbergjagbk_USERID
+ "My Stuff"
+ gnjfgsdkfjsdf_ITEMkey
-- "item name": whatever
-- "itemExpires": 05-01-2017
-- "itemType": whatever too
+ an3jeejwiag_ITEMkey
-- "item name": whatever
-- "itemExpires": 06-01-2017
-- "itemType": whatever too
+ zzzndjgabblsbl_ITEMkey
-- "item name": whatever
-- "itemExpires": 07-01-2017
-- "itemType": whatever too
I'm not asking for someone to write the code, a good reference will do but there are so many ways to call data and all I'm finding are the ways to Call using a structured tree and not one with random id's and keynumbers.
*** Basically, my goal here is to run a 3rd party cron job through Firebase Functions that runs through each item entry and checks the expiration date against today's date. This is the wall I'm against.
Bradley I am yet unclear as to what you want to do exactly. I suppose you intend on having multiple users (not just one as in the example) with multiple Items and compare the current date against the expiration date of all items for every user at a specified time (using cron). There are some considerations you should take into account here :
Do you really need cron ? ( or can you solve your problems more easily and natively with a javascript plain setInterval() ? )
How often are you going to check your entire database and how big is that database?
OK. So to explain, the first consideration is just a thought and the logic behind it should be pretty obvious. The second consideration takes some explaining. Since I believe your firebase data will NOT be static and will constantly change you need to find a way to get these changes inside you node script.
If you do not intend on calling your scheduled task too often and the database is not a mammoth (taking ages to load) you could just do the following :
const firebase = require('firebase-admin');
const serviceAccount = require('yourServiceAccountPath.json');
firebase.initializeApp({
credential: firebase.credential.cert(serviceAccount),
databaseURL: "youDatabaseURL"
});
setInteval(function(){ // or cron schedule
firebase.database().ref().once('value',function(snapshot){
let allYourUsers = snapshot.val();
// iterate through them all
// and do what you gotta do
});
},10000); // your interval in milliseconds
However this approach just loads once all your database each time you want to check all items. If you have other data in the database, consider adding users to a seperate path and load just that path. This approach is not recommended if your users are quite many and/or you want them to be checked very often. If such is the case and your data does not change very often you could consider this alternative:
Here you use the on function to update your data as it is edited and set the checking part seperate like so :
const firebase = require('firebase-admin');
const serviceAccount = require('yourServiceAccountPath.json');
firebase.initializeApp({
credential: firebase.credential.cert(serviceAccount),
databaseURL: "youDatabaseURL"
});
const databaseRef=firebase.database().ref();
let allYourUsers;
let allYourUsersStaticCopy;
databaseRef.on('value',function(snapshot){
allYourUsers = snapshot.val();
});
setInteval(function(){ // or cron schedule
if ( allYourUsers ) { // to ensure that data has loaded at least once
// (startup considerations)
allYourUsersStaticCopy = Object.assign({},allYourUsers);
// iterate through the static copy in order to avoid
// you data changing while you are accesing it
// and do what you gotta do
}
},10000); // your interval in milliseconds
The upside with the second piece of code is that your data is loaded every time there is a change and not every time your check runs. If however your data changes very often (additions,deletions and edits) this approach might not be optimum.
In the case that your script runs often enough, the database is big enough, and the changes are often enough to prevent any of the above to be efficient, you might want to consider the optimum solution : loading your users once and then attaching listeners for the child added,removed and changed to update your existing users object. Thus you receive only changes and not a whole snapshot. If such is the case you can read about the mentioned listeners in the firebase docs and I can squeeze in some extra code. Just let me know.
Hope this long post helps!
Assuming you have Firebase set up properly within your node project, you can do a one time read for your ITEMkey entries. Something like this:
var db = admin.database();
var ref = db.ref("82hfijcbwjbjfbergjagbk_USERID").child("My Stuff");
ref.once("value", function(snapshot) {
var contents = snapshot.val();
// Data returned here will be an object with all children
// nodes under "My Stuff". You can access it by calling
// snapshot.val() like I did above.
}
I just started developing nodejs. I'm confused to use async model. I believe there is a way to turn most of SYNC use cases into ASYNC way. Example, by SYNC, we load some data and wait until it returns then show them to user; by ASYNC, we load data and return, just tell the user data will be presented later. I can understand why ASYNC is used in this scenario.
But here I have a use case. I'm building an web app, allowing user to place a order (buying something). Before saving the order data into db, I want to put some user data together with order data (I'm using document NoSql db by the way). So I think by SYNC, after I get order data, I make a SYNC call to database and wait for its returned user data. After I get returned data, integrate them together and ingest into db.
I think there might be an issue if I make ASYNC call to db to query user data because user data may be returned after I save data to db. And that's not what I want.
So in this case, how can I do this thing ASYNCHRONOUSLY?
Couple of things here. First, if your application already has the user data (the user is already logged in), then this information should be stored in session so you don't have to access the DB. If you are allowing the user to register at the time of purchase, you would simply want to pass a callback function that handles saving the order into your call that saves the user data. Without knowing specifically what your code looks like, something like this is what you would be looking for.
function saveOrder(userData, orderData, callback) {
// save the user data to the DB
db.save(userData, function(rec) {
// if you need to add the user ID or something to the order...
orderData.userId = rec.id; // this would be dependent on your DB of choice
// save the order data to the DB
db.save(orderData, callback);
});
}
Sync code goes something like this. step by step - one after other. There can be ifs and loops (for) etc. all of us get it.
fetchUserDataFromDB();
integrateOrderDataAndUserData();
updateOrderData();
Think of async programming with nodejs as event driven. Like UI programming - code (function) is executed when an event occurs. E.g. On click event - framework calls back registered clickHandler.
nodejs async programming can also be thought on these lines. When db query (async) execution completes, your callback is called. When order data is updated, your callback is called. The above code goes something like this:
function nodejsOrderHandler(req,res)
{
var orderData;
db.queryAsync(..., onqueryasync);
function onqueryasync(userdata)
{
// integrate user data with order data
db.update(updateParams, onorderudpate);
}
function onorderupdate(e, r)
{
// handler error
write response.
}
}
javascript closure provides the way to keep state in variables across functions.
There is certainly much more to async programming and there are helper modules that help with basic constructs like chain, parallel, join etc as you write more involved async code. but this probably gives you a quick idea.
The following code is fetch data (fill data for the first time) part of my tableViewController. I am using an NSManagedDocument's managedObjectContext to fill (pre populate) my database. The source is an array that I clean up from my TXT file which rests directly in the Xcode's resources folder. After this creation, I have document cases like closed / open and normal.
The following code inputs and fetches my data onto the table correctly with a fetched results controller request. However, when the data is loading in the thread that is meant to free the UI from this one time data creation (26854 object names) into managedObject.name attribute heavy operation, the tableview and my UI is frozen (for 1-15 seconds that is I think while populating in document.managedObjectContext for the first time for my database).
After 10-15 seconds data is loaded and shows correctly. However, when I stop the simulator and restart the app in simulator, although I save the document as seen in below code, and I use the same fetch results controller setup (and request) the table view shows empty, it is movable in this case (The document state shows open and normal at this stage and file path is same, I checked... It seems like neither autosave nor explicit saveForOverwriting I use work... Or is it something else? I tried a lot of things and I'll go crazy soon. I think it has something to do with my multithreading.
self.managedObjectNames is the array property in the table view and I set it from the TXT file during my table view's loadView:
Is there anybody out there who can show the mistake here? Is it that I give self.managedObjectNames in the method of entity creation category.
Thanks!
- (void)fetchDataIntoDocument:(UIManagedDocument *)document {
dispatch_queue_t fetchQ = dispatch_queue_create("Data fetcher", NULL);
dispatch_async(fetchQ, ^{
[document.managedObjectContext performBlock:^{
for (int i = 0; i < 26854; i++) {
[managedObject managedObjectWithId:[NSNumber numberWithInt:i] andArray:self.managedObjectNames inManagedObjectContext:document.managedObjectContext];
}
// NSLog(#"Save baby!!?");
[document saveToURL:document.fileURL forSaveOperation:UIDocumentSaveForOverwriting completionHandler:nil];
}];
});
dispatch_release(fetchQ);
}
The reason why your UI is blocked for 10-15 seconds is because the document.managedObjectContext has been created with NSMainQueueConcurrencyType. That means that the performBlock: method will be executed on the main queue.
Creating the fetchQ in your code does not have any reason. It would have a reason if fetching of data would take some considerable amount of time but adding them would be fast (e.g. creating/modifying only few objects):
dispatch_async(fetchQ, ^{
// fetch data here (e.g. fetchAttribute may take few seconds)
NSString *attribute = fetchAttribute();
[document.managedObjectContext performBlock:^{
MyObject *o;
o = [NSEntityDescription insertNewObjectForEntityForName:#"MyObject"
inManagedObjectContext:document.managedObjectContext];
o.myAttribute = attribute;
}];
});
However I don't know answer to your main question.