Making MongoDB Update/Write Queries Faster - node.js

Question

I am looking for the most efficient way to store realtime data in MongoDB for a MeanJS web application I am working on.
I have the following example schema:
SomeModel: {
name: {
type: String,
default: '',
required: 'Please Enter Name',
trim: true
},
data: {
type: Schema.Types.Mixed,
default: {}
},
data_keys: {
type: Schema.Types.Mixed,
default: {}
},
websocket_url: {
type: String,
default: '',
},
created: {
type: Date,
default: Date.now
},
user: {
type: Schema.ObjectId,
ref: 'User'
}
}
The 'data' field may have data like this, but it depends on the 'model' being subscribed to, each model's data may have a slightly different format.
data: {
balance: {
currentBalance: 100,
availableBalance: 80,
/* Additional Account Details */
},
orders: [{
/* Some Array of Order Details */
}],
/* Additional Data Properties */
}
For each 'someModel' object I am trying to connect to a websocket server, subscribe to updates and then write them to the database.
I am trying to use something like this:
some_ws = new WebSocket(someModel.websocket_url);
some_ws.on('message', function incoming(msg) {
var message = JSON.parse(msg);
try {
// Update 'someModel.data' in memory.
Object.keys(message['data']).forEach(function(key) {
someModel.data[key] = message['data'][key];
});
// Write out to Database.
SomeModel
.update({_id: someModel._id}, {data: someModel.data, data_keys: someModel.data_keys})
.exec(function (err, nItems) {
if(err) {
console.log("ERROR Saving SomeModel Data: %s", err);
} else {
// console.log("Saved Data for: %s", someModel.name);
}
});
} catch (exception) {
console.log(clc.red("Exception Caught: %s"), util.inspect(exception));
console.log(clc.cyan("DEBUG:: Message: %s"), util.inspect(message));
}
});
I'm finding I'm getting almost continuous updates from the websocket connection and that the 'update' queries are slowing down the 'read queries' that need to happen in the front end of the application.
I'd like to be able to store the 'current' data for the model in this 'someModel.data' object, and then every minute write to a 'model_log' table with a "snapshot" of what data associated with the model at that particular time:
eg:
model_log schema: {
model: {
type: 'Schema.ObjectId',
ref: 'SomeModel',
},
data: {
/* Model Data */
},
timestamp: {
type: Date,
default: Date.now,
}
}
so I can do: model_log.find({'timestamp': { $gte: startDate, $lte: endDate } });
and get back:
[
{
model: ObjectId('someModelId'),
data: {
someData: someValues,
otherData: otherValues,
},
timestamp: March 15, ‎2016‎ ‎12:‎00‎ ‎AM
},
{
model: ObjectId('someModelId'),
data: {
someData: someNewValues,
otherData: otherNewValues,
},
timestamp: March 15, 2016 12:01 AM,
},
...
]
How can I make this more efficient or make these write/update operations faster?
Thanks,

Answer 1

there are few options:
split load by using collection sharding
create replica set where secondary server can reply for queries and primary is responsible to serve data and push changes
using wired tiger storage engine put collection into memory (I'am unsure if we can do it in community)
use SSD HDD to reduce write latency
switch to wired tiger (as we have here document lock level instead of collection lock level)
nr 3 & 5 can be tested in separation on dev machine