mongodb - Find first subdocument that matches query - node.js

Question

I've created a database schema very similar to the documentation on time series data here.
I ended up with the following documents:
{
"_id" : "unique_name_1",
"data" : {
"2017" : {
"5" : {
"21" : {
"61" : [
{
"timestamp" : 1498162890460.0,
"value" : true
}
],
"84" : [
{
"timestamp" : 1498183202126.0,
"value" : false
}
]
},
"22" : {
"24" : [
{
"timestamp" : 1498215602957.0,
"value" : true
}
],
"61" : [
{
"timestamp" : 1498249322863.0,
"value" : false
}
]
}
},
"9" : {
"16" : {
"36" : [
{
"timestamp" : 1508249031987.0,
"value" : true
},
{
"timestamp" : 1508249429052.0,
"value" : false
}
]
}
}
}
}
}
The first subdocument under 'data' represents the year. The second subdocument represents the month. The third subdocument represents the day. And the fourth subdocument represents the 15 minute interval of the day in which the event happened.
I want to query the database and get the first true (and maybe the immediate false that follows if possible) every day while ignoring all subsequent entries.
The first entry every day may or may not be true. The data does not necessarily always go from true to false back to true. For example, I may have several trues in a row, in which case I would want to ignore all subsequent true values.
This structure is really great for querying specific times, if you know them, but I'm at a loss for querying specific values. Should I just query the entire document and parse it on the front end? That becomes more difficult if I want to run the same query on hundreds of documents.
Thanks for the help.

Answer 1

To analyze your data in context of single day you have to use a long series of $objectToArray combined with $unwind. That's because the only way to figure out what represents day, month, year is to analyze the level of nesting. However once you do that, the rest might be quite simple. Try:
db.col.aggregate([
{
$project: {
data: { $objectToArray: "$data" }
}
},
{
$unwind: "$data"
},
{
$project: {
year: "$data.k",
data: { $objectToArray: "$data.v" }
}
},
{
$unwind: "$data"
},
{
$project: {
year: 1,
month: "$data.k",
data: { $objectToArray: "$data.v" }
}
},
{
$unwind: "$data"
},
{
$project: {
year: 1,
month: 1,
day: "$data.k",
data: { $objectToArray: "$data.v" }
}
},
{
$unwind: "$data"
},
{
$project: {
year: 1,
month: 1,
day: 1,
interval: "$data.k",
data: "$data.v"
}
},
{
$unwind: "$data"
},
{
$match: { "data.value": true }
},
{
$sort: {
"data.timestamp": 1
}
},
{
$group: {
_id: {
year: "$year",
month: "$month",
day: "$day"
},
firstPositive: {
$first: "$data"
}
}
}
])

Answer 2

Note that because you're using key/value pairs rather than an array, you would need to use wildcard keys in order to perform such a query. This simply isn't possible in MongoDB.
That being said, there's a new aggregation operator as of version 3.6 called $objectToArray which may prove useful for resolving this problem. I would recommend taking your research there and attempting to develop the appropriate aggregation pipeline with that operator in mind.
If you get stuck on the pipeline, then that would be a good time to return with a new question showing what you've managed to write up so far. Until then, there's unfortunately not much more that can be done to help you work around your existing document structure.
The alternatives, of course, are either to completely restructure your documents (the nuclear option), which may not be a possibility if you're working on a production system which would require updating all relevant pieces of code, or to process this data directly on the backend instead of through a MongoDB query.