Aggregate time series data - node.js

Question

I am using mongoDB and mongoose to store metrics data. It is stored as a document for an array of metrics referencing the project it's stored for and metric type.
The schema for this looks like this:
exports.metricReportSchema = new Schema({
metrics: [{
metric: {
type: mongoose.Schema.Types.ObjectId,
ref: 'metricSchema',
required: true
},
value: {
type: String,
required: true
}
}],
project: {
type: mongoose.Schema.Types.ObjectId,
ref: 'projectSchema',
required: true
},
reportDate: Date
});
And the actual document looks like the following:
db.metricreports.findOne() {
"_id" : ObjectId("58a60e8459dd3d12ef8c5d51"),
"reportDate" : ISODate("2017-02-16T20:41:40.657Z"),
"project" : ObjectId("58a20f5f04ef5789d3ef8faa"),
"metrics" : [
{
"metric" : ObjectId("58a20f5f04ef5789d3ef8fb7"),
"value" : "781",
"_id" : ObjectId("58a60e8459dd3d12ef8c5d52")
}, {
"metric" : ObjectId("58a21106fc2aef8a10ded196"),
"value" : "566",
"_id" : ObjectId("58a60e8459dd3d12ef8c5d53")
}, {
"metric" : ObjectId("58a2141bded78e8ad8384f97"),
"value" : "501",
"_id" : ObjectId("58a60e8459dd3d12ef8c5d54")
}, {
"metric" : ObjectId("58a2141bded78e8ad8384f94"),
"value" : "44",
"_id" : ObjectId("58a60e8459dd3d12ef8c5d55")
}, {
"metric" : ObjectId("58a2141bded78e8ad8384f93"),
"value" : "645",
"_id" : ObjectId("58a60e8459dd3d12ef8c5d56")
}
],
"__v" : 0
}
Over time, there are multiple documents of this kind that store slices of data for multiple metrics. It is very convenient for selecting and displaying static reports on metrics for multiple projects and whatnot.
Now, this is becoming little complex when I try to build a time series report for an individual metric for a project.
Basically, what I would need to do is to scan multiple metricReport documents and extract individual single metrics' data from all available reports over time. Let's say I have 10 metricReports that each contain data for 10 different metrics and I only want to extract one, this could probably look like this:
{
"_id": "...",
"project": "...",
"metric": "...",
"data": {
"2016-02-02": "22",
"2016-02-03": "453",
...
}
}
I could not find a way to do this with out-of-box mongoDB querying and filtering functionality and wanted to ask for advice:
Is my approach of storing multiple metrics in a single document reasonable? Would I be better of keeping metrics as individual documents and then "merging" them somehow?
Is there a way to achieve what I need without doing this with nodejs (I assume this is not going to be very fast thing - grabbing the documents and then iterating them to create a new structure alongside and pushing it out)?
Is there a better way to do this? Virtual models or something in mongoose that could help? I understand that mongoDB may not be the right choice for time series data but it's not the only part of functionality and mongoDB/mongoose combination seems to be serving the other purposes nicely and I don't want to change the technology mid-way.

Answer 1

Yes, but keep in mind that documents have a limited size (16 MB IIRC), so if your data is unbounded, this structure will not work as your "metrics" array will grow past that.
Ultimately yes, even if you can't figure out a decent filter query, Mongo has MapReduce which will allow you to do what you want, though it won't be easy. I'd use Node for this.
There's no silver bullet here. Mongo is excellent if you need to aggregate data and store it as arbitrary JSON (i.e. to be consumed by an app), and not so great at doing complex joins/views of data. Any joins at the application level will be slow. If you want performance, you'll have to aggregate your reports into individual documents, and save & serve them. If you have live data, this will be more difficult as you'll need to handle updates.