Node JS best practices for static string mappers - node.js

I am running a Node JS app (Sails JS, but shouldn't matter) and MongoDB (also shouldn't matter).
I have a database model that the WebApp saves many instances of it. For simplicity, let us take a UserActivityHistory model example (the example is not my real case, so nevermind the logic).
One of the members of the model is "ActivityType", which can be a medium-long text but can be chosen from a static set. For example:
ActivityType may be one of the following:
"User added a new account".
"User changed account password".
So, assume that I am going to save thousands of instances, I prefer to save activity type identifier/code for memory performance. For example:
Instead of saving "User added a new account", the WebApp should save "UANC".
When the user wants to view the "UserActivityHistory", I want to have a static mapper (maybe global variable?) that is some kind of dictionary as follows:
["UNAC"] = "User added a new account"
["UCAP"] = "User changed account password"
Which means, I receive the code of the activity and returns the text from a static dictionary that is saved in the server memory running the application.
What is the most efficient way and best practices to achieve this mapping?
The ways I thought of:
1) Global variables: saving a global dictionary and add/remove values from it by API calls. I am not sure if this is a wise idea, I still do not see why not, but still deep in my heart something is telling me do not trust global variables.
2) Saving mappers in another database: this requires another call to the database for each set of activities. (doesn't seem so heavy to me, if I load many activities at once, but still there could be a better solution).
Note that I want to add dynamic activities, so having a function with a switch case is not a proper solution.

If you want to avoid having to redeploy your app everytime a string changes then I would recommend you look at pulling these from some persistant storage (e.g. external file / DB).
Regardless of which you choose, given they are application settings, you would pull them once either as the app starts or on first access. This would give you the flexibility of changing them out-with the app (or perhaps even from the app).
To answer your comments
If I pull them as the app starts on first access, where should I save them? global variable?
Well the natural thing to do here would be to have it as it's own module, this would give you a good place to have other methods for manipulating the list & persisting to storage. If you go for the singleton approach then you could load the resource on first access e.g.
ActivityType.js
var fs = require('fs');
var cache = null;
function loadCache(callback) {
if (!cache) {
fs.readFile('/path-to-strings', function (err, data) {
if (!err) {
// for the purpose of the example, lets assume you are storing the
// strings as a JSON object e.g. { 'key1': 'value1', 'key2': 'value2' }
cache = JSON.parse(data);
callback(null, cache);
} else {
callback(err, null);
}
});
} else {
callback(null, cache);
}
};
module.exports = {
get: function(key, callback) {
loadCache(function(err, cache) {
callback(err, (cache || {})[key]);
});
},
add: function(key, value, callback) {
loadCache(function(err, cache) {
if (err) {
if (callback) callback(err);
} else {
cache[key] = value;
}
});
},
remove: function(key, callback) {
loadCache(function(err, cache) {
if (err) {
if (callback) callback(err);
} else {
delete cache[key];
}
});
},
save: function(callback) {
loadCache(function(err, cache) {
if (err) {
if (callback) callback(err);
} else {
var data = JSON.stringify(cache, null, 4); // save formatted for easier editing
fs.writeFile('/path-to-strings', data, callback);
}
});
}
};
App.js
var ActivityType = require('./ActivityType');
...
ActivityType.get('UNAC', function(err, text) {
if (!err) {
console.log(text);
} else {
console.log('Unable to load string');
}
});
ActivityType.add('UCAP', 'User changed account password');
ActivityType.remove('UNAC');
ActivityType.save(function(err) {
if (!err) {
console.log('Saved successfully!');
} else {
console.log('Unable to save strings');
}
});
Alternatively, you could have an explicit load or init method you could call during app startup.
How expensive is the redeployment of the app?
That's not something I could answer as there is not enough information here, only you know how expensive a redeployment of your app would be. Regardless, surely updating a DB record or changing a resource file is far simpler than a redeployment?

Related

Use redis to build a real time chat with socket.io and NodeJs

I want to built a real time chat system for my project but actually I have some problems with Redis because I want my data stored as better as possible.
My problem:
I'd like to use Socket Io to do real time chatting in a closed group (of two people), but how to store messages?
Redis is a key value store and that means that if i want to store something i need to add an unique key to my data before getting stored.
If the same user posts more than one messages which keys would I use inside redis? I'm thinking about unique ids as unique keys but since I want to be able to fetch this comments when a user log the chat page, but if I do that I need to write another database that relate chat ids to the user that posted that message
Am I forgetting anything? Is there a best method to do this?
Sorry for my bad English.
Redis is more then key-value store.
So you want the following:
chat messages,
two-person discussions,
you did not mention time constraints, so lets assume that you archive messages after a while,
you also don't say if you want separate "threads" between two people, like forums or continuous messages, like facebook. I'm assuming continuous.
For each user, you have to store messages he sends. Let's say APP_NAMESPACE:MESSAGES:<USER_ID>:<MESSAGE_ID>. We add userId here so that we can easily retreive all messages sent by a single user.
And, for each two users, you need to track their conversations. As a key, you can simply use their userids APP_NAMESPACE:CONVERSATIONS:<USER1_ID>-<USER2_ID>. To make sure you always get the same, shared conversation for the two users, you can sort their ids alfabetically, so that users 132 and 145 will both have 132:145 as conversation key
So what to store in "conversations"? Let's use a list: [messageKey, messageKey, messageKey].
Ok, but what is now the messageKey? Combo of userId above and a messageId (so we can get the actual message).
So basically, you need two things:
Store the message and give it an ID
Store a reference to this message to the relevant conversation.
With node and standard redis/hiredis client this would be somehting like (I'll skip the obvious error etc checks, and I'll write ES6. If you cannot read ES6 yet, just paste it to babel):
// assuming the init connects to redis and exports a redisClient
import redisClient from './redis-init';
import uuid from `node-uuid`;
export function storeMessage(userId, toUserId, message) {
return new Promise(function(resolve, reject) {
// give it an id.
let messageId = uuid.v4(); // gets us a random uid.
let messageKey = `${userId}:${messageId}`;
let key = `MY_APP:MESSAGES:${messageKey}`;
client.hmset(key, [
"message", message,
"timestamp", new Date(),
"toUserId", toUserId
], function(err) {
if (err) { return reject(err); }
// Now we stored the message. But we also want to store a reference to the messageKey
let convoKey = `MY_APP:CONVERSATIONS:${userId}-${toUserId}`;
client.lpush(convoKey, messageKey, function(err) {
if (err) { return reject(err); }
return resolve();
});
});
});
}
// We also need to retreive the messages for the users.
export function getConversation(userId, otherUserId, page = 1, limit = 10) {
return new Promise(function(resolve, reject) {
let [userId1, userId2] = [userId, otherUserId].sort();
let convoKey = `MY_APP:CONVERSATIONS:${userId1}-${userId2}`;
// lets sort out paging stuff.
let start = (page - 1) * limit; // we're zero-based here.
let stop = page * limit - 1;
client.lrange(convoKey, start, stop, function(err, messageKeys) {
if (err) { return reject(err); }
// we have message keys, now get all messages.
let keys = messageKeys.map(key => `MY_APP:MESSAGES:${key}`);
let promises = keys.map(key => getMessage(key));
Promise.all(promises)
.then(function(messages) {
// now we have them. We can sort them too
return resolve(messages.sort((m1, m2) => m1.timestamp - m2.timestamp));
})
.catch(reject);
});
});
}
// we also need the getMessage here as a promise. We could also have used some Promisify implementation but hey.
export function getMessage(key) {
return new Promise(function(resolve, reject) {
client.hgetall(key, function(err, message) {
if (err) { return reject(err); }
resolve(message);
});
});
}
Now that's crude and untested, but that's the gist of how you can do this.
Is redis is a constraint in your project?
you can go through this http://autobahn.ws/python/wamp/programming.html

How can I cancel MongoDB query from .each callback

I implemented a little NodeJs web server that stores log entries and provides a backend for a web based log browser. The web interface provides also an "Export to CVS" function and lets user download the logs in CVS format. My code looks similar to this:
this.log_entries(function(err, collection) {
collection.find(query)
.sort({_id: 1})
.each(function (err, doc) {
if(doc){
WriteLineToCSVFile(doc);
}
else {
ZipCSVFileAndSendIt();
}
});
});
The export operation may take a significant amount of time and disk space in case if a user didn't specify the right filters for the query. I need to implement a fail safe mechanism preventing this. One important requirement is that user should be able to abort the ongoing export operation at any point in time. Currently my solution is that I stop writing the data to the CSV file, however the callback passed to the .each() still gets called. I could not find any information how to stop the each loop. So the question is how can I do this?
UPDATE, THE ANSWER:
Use cursor.nextObject()
For the correct answer see the comments by #dbra below: db.currentOp() and db.killOp() doesn't work for this case.
The final solution looks like this:
this.log_entries(function(err, collection) {
var cursor = collection.find(query);
cursor.sort("_id", 1, function(err, sorted) {
function exportFinished(aborted) {
...
}
function processItem(err, doc) {
if(doc === null ) {
exportFinished(false);
}
else if( abortCurrentExport ) {
exportFinished(true);
}
else {
var line = formatCSV(doc);
WriteFile(line);
process.nextTick(function(){
sorted.nextObject(processItem);
});
}
}
sorted.nextObject(processItem);
});
});
Note the usage of process.nextTick - without it there will be a stack overflow!
You could search the running query with db.currentOp and then kill it with db.killOp, but il would be a nasty solution.
A better way could be working with limited subsequent batches; the easier way would be a simple pagination with "limit" and "skip", but it depends on how your collection changes while you read it.

How do you save multiple records in SailsJS in one action?

I am wondering what the best practice would be for saving multiple records in an action that makes changes, particularly add and remove, to multiple records. The end goal is to have one function have access to all of the changed data in the action. Currently I am nesting the saves in order for the innermost saves to have access to the data in all the updated records. Here is an example of how I am saving:
record1.save(function (error, firstRecord) {
record2.save(function (erro, secondRecord) {
record3.save(function (err, thirdRecord) {
res.send({recordOne: firstRecord, recordTwo: secondRecord, recordThree: thirdRecord});
});
});
});
With this structure of saving, recordOne, recordTwo, and recordThree display the expected values on the server. However, checking localhost/1337/modelName reveals that the models did not properly update and have incorrect data.
You could use the built in promise engine Bluebird to to that.
Promise.then(function() {
return [record1.save(),record2.save(),record3.save()];
}).spread(function(record1_saved,record2_saved,record3_saved){
res.send({recordOne: record1_saved, recordTwo: record2_saved, recordThree: record3_saved});
req.namespamce = this;
}).catch(function(err){
res.badRequest({
error: err.message
});
req.namespamce = this;
});

Delete multiple entities at once using Azure Table Storage Node.js interface

The only delete operation I get deletes one at a time: https://www.windowsazure.com/en-us/develop/nodejs/how-to-guides/table-services/#delete-entity
What I want is equivalent to the SQL statement
DELETE FROM MyTable WHERE PartitionKey = 'something'
Also on that page is a way to send a batch (although I could not get this to work with delete, anyone know why?). However, I'd first have to do a select to get the list of entities I want to delete in order to get their RowKeys. I was wondering whether it's possible to do it in only one request to Azure.
Thanks in advance.
UPDATE: Here's the code I tried and it doesn't work. I have confirmed that all arguments are correct when the function is called.
// subAccts all have PartitionKey = pKey
function deleteAccount(pKey, rKey, subAccts, callback) {
var tasks = subAccts; // rename for readability
tasks.push({ PartitionKey: pKey, RowKey: rKey });
tableService.beginBatch();
async.forEach(tasks, function(task, callback) {
tableService.deleteEntity(myTable, task, function(error) {
if (!error) {
callback(null);
}
else {
console.log(error);
callback(error);
}
});
}, function(error) {
if (error) {
console.log(error);
callback(error);
return;
}
tableService.commitBatch(callback);
});
}
If you don't know the list of entities you want to delete in advance, you'll have to query first to find them.
You might consider restructuring your table, though. If, instead of just putting these entities in the same partition, you could put them all in the same table (by themselves), you could delete the table. This is commonly used for deleting old logs... create tables like log_january, log_february, and then you can just delete an entire month at a time with a single command.
EDIT
There appears to be a bug in the library. Try this edit as a workaround:
BatchServiceClient.prototype.addOperation = function (webResource, outputData) {
if (azureutil.objectIsNull(outputData)) {
outputData = '';
}
if (webResource.httpVerb !== 'GET') {
webResource.headers[HeaderConstants.CONTENT_ID] = this.operations.length + 1;
if (webResource.httpVerb !== 'DELETE') {
webResource.headers[HeaderConstants.CONTENT_TYPE] = 'application/atom+xml;type=entry';
} else {
delete webResource.headers[HeaderConstants.CONTENT_TYPE];
}
...
I've created a pull request to fix this in the dev branch: https://github.com/WindowsAzure/azure-sdk-for-node/pull/300.
Until this is fixed, you can always clone my fork (https://github.com/smarx/azure-sdk-for-node), checkout the dev branch, and npm install that, rather than hand-editing the code.

Trying to understand how the node.js programming model works

I've been reading about node.js recently (like many others). I find interesting for some use cases, but am a bit struggling to understand the inner workings, specifically the interaction between closure functions and the process flow of the code.
Let's say I have a function which accepts a key-value array. The function must check that the values follow certain data-quality guidelines (for example some keys must have a value, other keys must have numbers as values etc) before storing the data somewhere (for the purpose of this question let's assume data validation has to be done in the application itself).
In "regular" developments models I'd write something like this:
resultName = validateName(data.name)
resultAddress = validateAddress(data.address)
resultID = validateID(data.id)
if (resultName && resultAddress && resultID) {
store(data)
else {
sendErrorToUser(data)
}
Get the results of the validations, and either explain the error(s) to the user or store data and return some kind of confirmation. The flow is very clear.
The way I understand node.js, the way to do this would be to delegate the validations to a different function (to avoid waiting for each validation to finish), and supply two callback functions to the functions which validate the chunks of data:
* a callback to call when validation is successful
* a callback to call when validation fails
It's easy to now return to the user with a "please wait" message, but I have to wait for all validations to clear (or fail) before storing the data or explaining the problem to the user. As a simple way to figure out if all the validations are done I thought of using a variable that counts the number of functions that called the callback, and emitting a "validation complete" event to store the validated data (or get back to the user with any errors). Or, alternatively, emit an event after each validation is complete and in that event's code check if all validations are complete before emitting the "store" / "error" events.
My question is -- am I approaching this correctly? Or is there a more suitable way to do these kinds of things with node.js (or similar event-based systems).
Thank you!
Alon
Are your validations asynchronous? If they are not you can use the code you posted, the "regular" one.
If the validations are asynchronous (checking uniqueness of an email for instance), you need to provide callbacks:
var validateUniqueEmail = function (data, callback) {
db.find({email: data.email}, function (err, result) {
callback(err, result === null);
})
};
var validateAndStore = function (data, callback) {
asyncValidation(data, function (err, is_valid) {
if (err) {
callback(err, null);
} else if (!is_valid) {
callback('Email is not unique', null);
} else {
db.store(data, callback);
}
});
}
The code above can be simplified a lot by using some validator or ORM modules already existing
example: mongolia validator module.
Let's go. Basically, what you want to do is something along the lines of :
var validate(data, cb){
var allOk = true;
for(var key in data){
allOk = allOk && validate[key](data.key); // validator depends on the key
}
if (allOk) cb(null, data); else cb(new Error "bleh");
}
This could be done the following way (note how we pass the failed keys as the first (error) argument to the callback):
var validate(data, cb){
var status = {true:[], false:[]},
total = Object.keys(data).length,
done = 0;
for (var key in data)
(function(key){
validate[key](data[key], function(ok){
status[ok].push(key);
if (++done == total){
status[false].length ? cb(status[false]) : cb(null);
}
});
})(key);
}
Which you can use this way :
validate(data, function(failures){
if (failures){
// tell the user the input does not validate on the keys stored in failures
} else {
// all ok
store(data);
}
});
Correct me if I'm wrong, but I think what you're asking is how to handle the response from multiple asynchronous calls.
Here's how I do it (using your validation example):
var result = {};
function isAllDataAvailable() {
return result.name !== undefined
&& result.address !== undefined
&& result.id !== undefined;
}
function callback(error) {
if (error) {
showError(error);
// terminate here (?)
return;
}
if (isAllDataAvailable()) {
showOutput();
}
}
validateName(data, callback);
validateAddress(data, callback);
validateEmail(data, callback);
The key here is the result object, which starts out as empty. As each field gets validated, it gets added to the result object (by the validation functions, which I've left out in the above snippet). I've used a single callback method, but you could have multiple, say callbackName, callbackAddress, etc. The validation results are processed only if and when the result object has been fully populated, which is checked in isAllDataAvailable.
Hope this helps.
Consider using: https://github.com/asaf/nodejs-model
It will make your life much easier when dealing with validators.

Resources