Check whether or not a booking is possible - node.js

I'm currently implementing a booking system. The seller can specify how many items of the given type are available. The rule is simple: There can never be more bookings then available items.
Now I would like to find out with how many existing bookings a new booking is in conflict in order to check if the limit is reached.
The following diagram should give you a little insight on what I'm trying to do.
https://ibb.co/4pJk8XV
In this example the maximum amount of concurrent bookings is 2. As you can see there are already 3 bookings. One of which has no end date specified.
Only viewing the existing bookings there are never more than 2 bookings at the time.
Now I would like to check whether a new booking is possible. I know the start date for every booking. For bookings with no specified end date, the end date will be null.
I'm trying to achieve this using Mongoose.
There is no existing code regarding this problem.
Looking at each example separately: The first one with no end date should fail, since between the 07th and 09th there would be 3 bookings at a time. The second one should be fine as there is only one existing booking on the 06th.

We will start by finding all intersections with the newDocument i'm only going to make one assumption which is that the "newBooking" endDate is typed date (if its null then we make it a very far futuristic date)
let newBookingStartDate = newBooking.startDate;
let newBookingEndDate = newBooking.endDate ? newBooking.endDate : new Date().setYear(3000);
Now for the query:
let results = await db.collection.aggregate([
{
$addFields: {
tmpEndDate: {
$cond: [
{$ne: ["$endDate", null]},
"$endDate",
newBookingEndDate
]
}
}
},
{
$match: {
$or: [
{
$and: [
{
startDate: {$lt: newBookingStartDate},
},
{
tmpEndDate: {$gt: newBookingStartDate}
},
]
},
{
$and: [
{
startDate: {$gte: newBookingStartDate},
},
{
startDate: {$lt: newBookingEndDate}
},
]
},
],
}
},
])
We match documents by diving into two cases:
document.startDate is lower than newDocument.startDate - in this case all we need to check if the document.endDate is greater then the newDocument.startDate, if it is then we have an intersection
document.startDate is greater or equal to newDocument.startDate - in this case we just need to check the document.startDate is less than the newDocument.endDate and again we'll get an intersection
Now we need to iterate over the documents we found and calculate intersections between them by running the same query:
for (let i = 0; i < results.length; i++) {
let doc = results[i];
let otherIds = results.map(val => val._id);
let docStartDate = doc.startDate;
let docEndDate = doc.endDate ? doc.endDate : new Date().setYear(3000);
let count = await db.collection.aggregate([
{
$match: {
_id: {$in: otherIds}
}
},
{
$addFields: {
tmpEndDate: {
$cond: [
{$ne: ["$endDate", null]},
"$endDate",
docEndDate
]
}
}
},
{
$match: {
$or: [
{
$and: [
{
startDate: {$lt: docStartDate},
},
{
tmpEndDate: {$gt: docStartDate}
},
]
},
{
$and: [
{
startDate: {$gte: docStartDate},
},
{
startDate: {$lt: docEndDate}
},
]
},
],
}
},
{
$count: "count"
}
])
if (count[0].count >= 3){
return false;
}
}
If any of the count results is 3 or greater (3 because i didn't remove the curr document ID from the array and it will always intersect with itself) return false as inserting a new document will set you over the threshold.

Related

MongoDB find documents using an array of objects as search argument

Is there a way to search a mongodb database using an array of objects as search arguments?
Lets say I have the following search preferences:
preferences = [{
product: "PRODUCT_A",
minqty: 5,
maxqty: 50
},
{
product: "PRODUCT_B",
minqty: 100,
maxqty: 500
}
]
In my database I have Jobs with the following structure:
{
jobName: "job name",
items: [{
product: "PRODUCT_A"
qty: 25
},
{
product: "PRODUCT_F"
qty: 300
}
]
}
I would like to query the database using preferences and returning any jobs that match at least one of the criteria's.
ATTEMPT 1:
I managed to use all my preferences as filters, but $match is cumulative, the way it's written it works like && in javascript. My goal is to have "match THIS || match THAT".
let pipeline = [];
preferences.map((item) => {
let search = {
product: item.product,
quantity: { $gte: item.minqty, $lte: item.maxqty },
};
return pipeline.push({
$match: {
items: {
$elemMatch: search,
},
},
});
});
const jobs = await Job.aggregate(pipeline);
ATTEMPS 2 (SUCCESS):
let search = [];
preferences.map((item, index) => {
let arguments = {
product: item.product,
quantity: { $gte: item.minqty, $lte: item.maxqty },
};
search.push(arguments);
});
let pipeline = [
{
$match: {
items: {
$elemMatch: {
$or: search,
},
},
},
},
];
const jobs = await Job.aggregate(pipeline);
Use aggregation
Denormalize items using $Unwind
Once denormalized, you can use simple match with $or
Use $lte and $gte
And update the question with your attempts of these or post a new one.

MongoDB aggregation $match with $or

Is there a way to search a mongodb database using an array of objects as search arguments?
Lets say I have the following search preferences:
preferences = [{
product: "PRODUCT_A",
minqty: 5,
maxqty: 50
},
{
product: "PRODUCT_B",
minqty: 100,
maxqty: 500
}
]
In my database I have Jobs with the following structure:
{
jobName: "job name",
items: [{
product: "PRODUCT_A"
qty: 25
},
{
product: "PRODUCT_F"
qty: 300
}
]
}
I would like to query the database using preferences and returning any jobs that match at least one of the criteria's.
ATTEMPT 1:
I managed to use all my preferences as filters, but $match is cumulative, the way it's written it works like && in javascript. My goal is to have "match THIS || match THAT".
let pipeline = [];
preferences.map((item) => {
let search = {
product: item.product,
quantity: { $gte: item.minqty, $lte: item.maxqty },
};
return pipeline.push({
$match: {
items: {
$elemMatch: search,
},
},
});
});
const jobs = await Job.aggregate(pipeline);
ATTEMPS 2 (SUCCESS):
let search = [];
preferences.map((item, index) => {
let arguments = {
product: item.product,
quantity: { $gte: item.minqty, $lte: item.maxqty },
};
search.push(arguments);
});
let pipeline = [
{
$match: {
items: {
$elemMatch: {
$or: search,
},
},
},
},
];
const jobs = await Job.aggregate(pipeline);
I think you can create your search object by reducing the preferences array and use the $or operator. When you map the preferences array it is returning an array that will perform and operation. you need an object like -
{
$or: [{product1, quantity1}, {product2, quantity2}]
}
I guess you got my point.

Use $lookup with a Conditional Join

provided I have following documents
User
{
uuid: string,
isActive: boolean,
lastLogin: datetime,
createdOn: datetime
}
Projects
{
id: string,
users: [
{
uuid: string,
otherInfo: ...
},
{... more users}
]
}
And I want to select all users that didn't login since 2 weeks and are inactive or since 5 weeks that don't have projects.
Now, the 2 weeks is working fine but I cannot seem to figure out how to do the "5 weeks and don't have projects" part
I came up with something like below but the last part does not work because $exists obviously is not a top level operator.
Anyone ever did anything like this?
Thanks!
return await this.collection
.aggregate([
{
$match: {
$and: [
{
$expr: {
$allElementsTrue: {
$map: {
input: [`$lastLogin`, `$createdOn`],
in: { $lt: [`$$this`, twoWeeksAgo] }
}
}
}
},
{
$or: [
{
isActive: false
},
{
$and: [
{
$expr: {
$allElementsTrue: {
$map: {
input: [`$lastLogin`, `$createdOn`],
in: { $lt: [`$$this`, fiveWeeksAgo] }
}
}
}
},
{
//No projects exists on this user
$exists: {
$lookup: {
from: _.get(Config, `env.collection.projects`),
let: {
currentUser: `$$ROOT`
},
pipeline: [
{
$project: {
_id: 0,
users: {
$filter: {
input: `$users`,
as: `user`,
cond: {
$eq: [`$$user.uuid`, `$currentUser.uuid`]
}
}
}
}
}
]
}
}
}
]
}
]
}
]
}
}
])
.toArray();
Not certain why you thought $expr was needed in the initial $match, but really:
const getResults = () => {
const now = Date.now();
const twoWeeksAgo = new Date(now - (1000 * 60 * 60 * 24 * 7 * 2 ));
const fiveWeeksAgo = new Date(now - (1000 * 60 * 60 * 24 * 7 * 5 ));
// as long a mongoDriverCollectionReference points to a "Collection" object
// for the "users" collection
return mongoDriverCollectionReference.aggregate([
// No $expr, since you can actually use an index. $expr cannot do that
{ "$match": {
"$or": [
// Active and "logged in"/created in the last 2 weeks
{
"isActive": true,
"$or": [
{ "lastLogin": { "$gte": twoWeeksAgo } },
{ "createdOn": { "$gte": twoWeeksAgo } }
]
},
// Also want those who...
// Not Active and "logged in"/created in the last 5 weeks
// we'll "tag" them later
{
"isActive": false,
"$or": [
{ "lastLogin": { "$gte": fiveWeeksAgo } },
{ "createdOn": { "$gte": fiveWeeksAgo } }
]
}
]
}},
// Now we do the "conditional" stuff, just to return a matching result or not
{ "$lookup": {
"from": _.get(Config, `env.collection.projects`), // there are a lot cleaner ways to register models than this
"let": {
"uuid": {
"$cond": {
"if": "$isActive", // this is boolean afterall
"then": null, // don't really want to match
"else": "$uuid" // Okay to match the 5 week results
}
}
},
"pipeline": [
// Nothing complex here as null will return nothing. Just do $in for the array
{ "$match": { "$in": [ "$$uuid", "$users.uuid" ] } },
// Don't really need the detail, so just reduce any matches to one result of [null]
{ "$group": { "_id": null } }
],
"as": "projects"
}},
// Now test if the $lookup returned something where it mattered
{ "$match": {
"$or": [
{ "active": true }, // remember we selected the active ones already
{
"projects.0": { "$exists": false } // So now we only need to know the "inactive" returned no array result.
}
]
}}
]).toArray(); // returns a Promise
};
It's pretty simple as calculated expressions via $expr are actually really bad and not what you want in a first pipeline stage. Also "not what you need" since createdOn and lastLogin really should not have been merged into an array for $allElementsTrue which would just be an AND condition, where you described logic would really mean OR. So the $or does just fine here.
So does the $or on the separation of conditions for the isActive of true/false. Again it's either "two weeks" OR "five weeks". And this certainly does not need $expr since standard inequality range matching works fine, and uses an "index".
Then you really just want to do the "conditional" things in the let for $lookup instead of your "does it exist" thinking. All you really need to know ( since the range selection of dates is actually already done ) is whether active is now true or false. Where it's active ( meaning by your logic you don't care about projects ) simply make the $$uuid used within the $match pipeline stage a null value so it will not match and the $lookup returns an empty array. Where false ( also already matching the date conditions from earlier ) then you use the actual value and "join" ( where there are projects of course ).
Then it's just a simple matter of keeping the active users, and then only testing the remaining false values for active to see if the "projects" array from the $lookup actually returned anything. If it did not, then they just don't have projects.
Probably should note here is since users is an "array" within the projects collection, you use $in for the $match condition against the single value to the array.
Note that for brevity we can use $group inside the inner pipeline to only return one result instead of possibly many matches to actual matched projects. You don't care about the content or the "count", but simply if one was returned or nothing. Again following the presented logic.
This gets you your desired results, and it does so in a manner that is efficient and actually uses indexes where available.
Also return await certainly does not do what you think it does, and in fact it's an ESLint warning message ( I suggest you enable ESLint in your project ) since it's not a smart thing to do. It does nothing really, as you would need to await getResults() ( as per the example naming ) anyway, as the await keyword is not "magic" but just a prettier way of writing then(). As well as hopefully being easier to understand, once you understand what async/await is really for syntactically that is.

Aggregation Timing Out

I am using aggregates to query my schema for counts over date ranges, my problem is i am not getting any response from the server (Times out everytime), other mongoose queries are working fine (find, save, etc.) and when i call aggregates it depends on the pipeline (when i only use match i get a response when i add unwind i don't get any).
Connection Code:
var promise = mongoose.connect('mongodb://<username>:<password>#<db>.mlab.com:<port>/<db-name>', {
useMongoClient: true,
replset: {
ha: true, // Make sure the high availability checks are on
haInterval: 5000 // Run every 5 seconds
}
});
promise.then(function(db){
console.log('DB Connected');
}).catch(function(e){
console.log('DB Not Connected');
console.errors(e.message);
process.exit(1);
});
Schema:
var ProspectSchema = new Schema({
contact_name: {
type: String,
required: true
},
company_name: {
type: String,
required: true
},
contact_info: {
type: Array,
required: true
},
description:{
type: String,
required: true
},
product:{
type: Schema.Types.ObjectId, ref: 'Product'
},
progression:{
type: String
},
creator:{
type: String
},
sales: {
type: Schema.Types.ObjectId,
ref: 'User'
},
technical_sales: {
type: Schema.Types.ObjectId,
ref: 'User'
},
actions: [{
type: {type: String},
description: {type: String},
date: {type: Date}
}],
sales_connect_id: {
type: String
},
date_created: {
type: Date,
default: Date.now
}
});
Aggregation code:
exports.getActionsIn = function(start_date, end_date) {
var start = new Date(start_date);
var end = new Date(end_date);
return Prospect.aggregate([
{
$match: {
// "actions": {
// $elemMatch: {
// "type": {
// "$exists": true
// }
// }
// }
"actions.date": {
$gte: start,
$lte: end
}
}
}
,{
$project: {
_id: 0,
actions: 1
}
}
,{
$unwind: "actions"
}
,{
$group: {
_id: "actions.date",
count: {
$sum: 1
}
}
}
// ,{
// $project: {
// _id: 0,
// date: {
// $dateToString: {
// format: "%d/%m/%Y",
// date: "actions.date"
// }
// }
// // ,
// // count : "$count"
// }
// }
]).exec();
}
Calling the Aggregation:
router.get('/test',function(req, res, next){
var start_date = req.query.start_date;
var end_date = req.query.end_date;
ProspectCont.getActionsIn(start_date,end_date).then(function(value, err){
if(err)console.log(err);
res.json(value);
});
})
My Main Problem is that i get no response at all, i can work with an error message the issue is i am not getting any so i don't know what is wrong.
Mongoose Version: 4.11.8
P.s. I tried multiple variations of the aggregation pipeline, so this isn't my first try, i have an aggregation working on the main prospects schema but not the actions sub-document
You have several problems here, mostly by missing concepts. Lazy readers can skip to the bottom for the full pipeline example, but the main body here is in the explanation of why things are done as they are.
You are trying to select on a date range. The very first thing to check on any long running operation is that you have a valid index. You might have one, or you might not. But you should issue: ( from the shell )
db.prospects.createIndex({ "actions.date": 1 })
Just to be sure. You probably really should add this to the schema definition so you know this should be deployed. So add to your defined schema:
ProspectSchema.index({ "actions.date": 1 })
When querying with a "range" on elements of an array, you need to understand that those are "multiple conditions" which you are expecting to match elements "between". Whilst you generally can get away with querying a "single property" of an array using "Dot Notation", you are missing that the application of [$gte][1] and $lte is like specifying the property several times with $and explicitly.
Whenever you have such "multiple conditions" you always mean to use $elemMatch. Without it, you are simply testing every value in the array to see if it is greater than or less than ( being some may be greater and some may be lesser ). The $elemMatch operator makes sure that "both" are applied to the same "element", and not just all array values as "Dot notation" exposes them:
{ "$match": {
"actions": {
"$elemMatch": { "date": { "$gte": start, "$lte: end } }
}
}}
That will now only match documents where the "array elements" fall between the specified date. Without it, you are selecting and processing a lot more data which is irrelevant to the selection.
Array Filtering: Marked in Bold because it's prominence cannot be ignored. Any initial $match works just like any "query" in that it's "job" is to "select documents" valid to the expression. This however does not have any effect on the contents of the array in the documents returned.
Whenever you have such a condition for document selection, you nearly always intend to "filter" such content from the array itself. This is a separate process, and really should be performed before any other operations that work with the content. Especially [$unwind][4].
So you really should add a $filter in either an $addFields or $project as is appropriate to your intend "immediately" following any document selection:
{ "$project": {
"_id": 0,
"actions": {
"$filter": {
"input": "$actions",
"as": "a",
"in": {
"$and": [
{ "$gte": [ "$$a.date", start ] },
{ "$lte": [ "$$a.date", end ] }
]
}
}
}
}}
Now the array content, which you already know "must" have contained at least one valid item due to the initial query conditions, is "reduced" down to only those entries that actually match the date range that you want. This removes a lot of overhead from later processing.
Note the different "logical variants" of $gte and $lte in use within the $filter condition. These evaluate to return a boolean for expressions that require them.
Grouping It's probably just as an attempt at getting a result, but the code you have does not really do anything with the dates in question. Since typical date values should be provided with millisecond precision, you general want to reduce them.
Commented code suggests usage of $dateToString within a $project. It is strongly recommended that you do not do that. If you intend such a reduction, then supply that expression directly to the grouping key within $group instead:
{ "$group": {
"_id": {
"$dateToString": {
"format": "%Y-%m-%d",
"date": "$actions.date"
}
},
"count": { "$sum": 1 }
}}
I personally don't like returning a "string" when a natural Date object serializes properly for me already. So I like to use the "math" approach to "round" dates instead:
{ "$group": {
"_id": {
"$add": [
{ "$subtract": [
{ "$subtract": [ "$actions.date", new Date(0) ] },
{ "$mod": [
{ "$subtract": [ "$actions.date", new Date(0) ] },
1000 * 60 * 60 * 24
]}
],
new Date(0)
]
},
"count": { "$sum": 1 }
}}
That returns a valid Date object "rounded" to the current day. Mileage may vary on preferred approaches, but it's the one I like. And it takes the least bytes to transfer.
The usage of Date(0) represents the "epoch date". So when you $subtract one BSON Date from another you end up with the milliseconds difference between the two as an integer. When $add an integer value to a BSON Date, you get a new BSON Date representing the sum of the milliseconds value between the two. This is the basis of converting to numeric, rounding to the nearest start of day, and then converting numeric back to a Date value.
By making that statement directly within the $group rather than $project, you are basically saving what actually gets interpreted as "go through all the data and return this calculated value, then go and do...". Much the same as working through a pile of objects, marking them with a pen first and then actually counting them as a separate step.
As a single pipeline stage it saves considerable resources as you do the accumulation at the same time as calculating the value to accumulate on. When you think it though much like the provided analogy, it just makes a lot of sense.
As a full pipeline example you would put the above together as:
Prospect.aggregate([
{ "$match": {
"actions": {
"$elemMatch": { "date": { "$gte": start, "$lte: end } }
}
}},
{ "$project": {
"_id": 0,
"actions": {
"$filter": {
"input": "$actions",
"as": "a",
"in": {
"$and": [
{ "$gte": [ "$$a.date", start ] },
{ "$lte": [ "$$a.date", end ] }
]
}
}
}
}},
{ "$unwind": "$actions" },
{ "$group": {
"_id": {
"$dateToString": {
"format": "%Y-%m-%d",
"date": "$actions.date"
}
},
"count": { "$sum": 1 }
}}
])
And honestly if after making sure an index is in place, and following that pipeline you still have timeout problems, then reduce the date selection down until you get a reasonable response time.
If it's still taking too long ( or the date reduction is not reasonable ) then your hardware simply is not up to the task. If you really have a lot of data then you have to be reasonable with expectations. So scale up or scale out, but those things are outside the scope of any question here.
As it stands those improvements should make a significant difference over any attempt shown so far. Mostly due to a few fundamental concepts that are being missed.

How to sum several months of data separately in a MongoDB collection

I'm trying to tally a field in a sub-array of a collection and I want to do it for each month. I had this working in Mongo 2.6 but recently upgrading to 3.0.12 has cause some erroneous results in the query. It almost seems like the sum is not getting reset for the several queries.
So currently I am doing twelve queries async and waiting for them all to complete. Again this was working in 2.6. My table structure is like this:
{
"_id" : ObjectId("<id>"),
"userId" : ObjectId("<id>"),
"accountId" : "1234567890",
"transactionId" : "22222222333",
"date" : ISODate("2016-09-08T04:00:00.000Z"),
"description" : "SUPERCOOL STORE",
"amount" : -123.45,
"allocations" : [
{
"jarId" : ObjectId("566faf1937af9ae11ef84bc4"),
"amount" : -100
},
{
"jarId" : ObjectId("566faf1937af9ae11ef84bc1"),
"amount" : -23.45
}
],
"reconciled" : true,
"tally" : true,
"split" : true
}
And this is my code:
var getTransactionAggregate = function (userId, month) {
var deferred = q.defer();
var nextMonth = moment(month).add(1, 'M');
Transactions.aggregate([
{$match: {
userId: userId,
tally: true,
date: {
$gte: month.toDate(),
$lt: nextMonth.toDate()
}
}},
{$project: { _id: 0, allocations: 1 } },
{$unwind: '$allocations'},
{$group: {_id: '$allocations.jarId', total: {$sum: '$allocations.amount'}}}
]).then(function(data) {
deferred.resolve(data);
})
.catch(function (err) {
logger.error(err);
deferred.reject(err);
});
return deferred.promise;
};
summaryReport = function (req, res) {
Jars.aggregate([
{ $match: {userId: new ObjectID(req.user._id)} },
{ $sort: {name: 1} }
])
.then(function (jars) {
var month = moment(moment().format('YYYY-MM') + '-01T00:00:00.000');
var i, j;
var promises = [];
for (i = 0; i < 12; i++) {
promises.push(getTransactionAggregate(new ObjectID(req.user._id), month));
month.add(-1, 'M');
}
q.allSettled(promises).then(function (data) {
for (i = 0; i < data.length; i++) {
// data[i].value here is returned incorrectly from the queries
........
});
});
};
So essentially what is happening is the first month includes the correct data but it appears that the sum continues to include data from all the previous months. If I break down the query the correct transactions are returned in the date range, and the unwind is working as well. Just when the groupBy step seems to be the culprit. The same logic worked fine before I upgraded Mongo to 3.0.12.
Is there a better way to execute this query in one shot or is doing the twelve queries the best way?
It seems to be a problem during the $match phase. Your date field has two expressions, and this scenario you need to use the $and operator, as specified in the docs:
MongoDB provides an implicit AND operation when specifying a comma
separated list of expressions. Using an explicit AND with the $and
operator is necessary when the same field or operator has to be
specified in multiple expressions.
So it becomes:
{$match: {
userId: userId,
tally: true,
$and: [
{ date: { $gte : month.toDate() } },
{ date: { $lt: nextMonth.toDate() } }
]
}}
It ended up being related to the match although not because of the $and case mentioned in the above answer. It had to do with the date matching, I'm guessing the moment object.toDate() does not return the same date object as when you use new Date(), although I thought they were the same.
Anyway the working logic looks like this:
Transactions.aggregate([
{$match: {
userId: userId,
tally: true,
$and: [
{ date: { $gt : new Date(month.toISOString()) } },
{ date: { $lt: new Date(nextMonth.toISOString()) } }
]
}},
{$unwind: '$allocations'},
{$group: {_id: '$allocations.jarId', total: {$sum: '$allocations.amount'}}}
])
Credit to Date query with ISODate in mongodb doesn't seem to work and #Leif for pointing me in the right direction.

Resources