I have a model of the following schema:
const mongoose = require('mongoose');
const livenessLogSchema = new mongoose.Schema({
probability: {
type: Number,
required: false,
},
timestamp: {
type: Date,
required: true,
}
}, {
timestamps: true
});
const LivenessLog = mongoose.model('LivenessLog', livenessLogSchema);
module.exports = LivenessLog;
I want to split all documents to 2 groups: those whose probability value is less than 0.001 and those whose value is greater than 0.001. Also, in each group, I want to count for each probabilty value - how many documents has the same value.
So basically if I had the following probabilities data: [0.00001, 0.000003, 0.000025, 0.000003, 0.9, 0.6, 0.6], I'd like to get as a result: { less: { 0.00001: 1, 0.000003: 2, 0.000025:1 }, greater: { 0.9: 1, 0.6: 2 }.
This is my current aggregate method:
const livenessProbilitiesData = await LivenessLog.aggregate([
{
$match: {
timestamp: {
$gte: moment(new Date(startDate)).tz('Asia/Jerusalem').startOf('day').toDate(),
$lte: moment(new Date(endDate)).tz('Asia/Jerusalem').endOf('day').toDate(),
}
}
},
{
$group: {
}
}
]);
Note that I use undeclared variables startDate, endDate. These are input I get to filter out unrelevant documents (by timestamp).
Related
I am trying to locate a certain value in a sub array using Mongoose.js with MongoDB. Below is my Mongoose schema.
const foobarSchema = new mongoose.Schema({
foo: {
type: Array,
required: true
},
comments: {
type: Array,
required: false
},
createdAt: { type: Date, required: true, default: Date.now }
});
The value I am trying to get is inside foo, so in foo I always have one array at place [0] which contains an object that is like the below
{
_id
code
reason
createdAt
}
I'd like to get the value for reason for all records created in the last 30 days. I've looked around on stack overflow and haven't found anything I could piece together. Below is my existing but non working code
const older_than = moment().subtract(30, 'days').toDate();
Foobar.find({ ...idk.. req.body.reason, createdAt: { $lte: older_than }})
edit add mock document
{
foo: [{
_id: 'abc123',
code: '7a',
reason: 'failure',
createdAt: mongo time code date now
}],
comments: []
}
curent code half working
const reason = req.params.reason
const sevenAgo = moment().subtract(7, 'days').toISOString()
Foo.aggregate([
{
$match: {
"foo.createdAt": {
$gte: sevenAgo
},
"foo.reason": {
reason
}
}
},
{
$project: {
reason: {
$arrayElemAt: [
"$foo.reason",
0
]
}
}
}
])
Currently returns blank array - no query failure - which is wrong it should return at least 1 document/record as that is what is in the DB that matches
expected mock data
[
{
code: 7a,
reason: failure
}
{
code: 7a,
reason:failure
}
]
I have been trying to use updatemany with mongoose. I want to update the values in database using an array of objects.
[
{
"variantId": "5e1760fbdfaf28038242d676",
"quantity": 5
},
{
"variantId": "5e17e67b73a34d53160c7252",
"quantity": 13
}
]
I want to use variantId as filter.
Model schema is:
let variantSchema = new mongoose.Schema({
variantName: String,
stocks: {
type: Number,
min: 0
},
regularPrice: {
type: Number,
required: true
},
salePrice: {
type: Number,
required: true
}
})
I want to filter the models using variantId and then decrease the stocks.
As you need to update multiple documents with multiple criteria then .updateMany() wouldn't work - it will work only if you need to update multiple documents with same value, Try this below query which will help you to get it done in one DB call :
const Mongoose = require("mongoose");
let variantSchema = new mongoose.Schema({
variantName: String,
stocks: {
type: Number,
min: 0
},
regularPrice: {
type: Number,
required: true
},
salePrice: {
type: Number,
required: true
}
})
const Variant = mongoose.model('variant', variantSchema, 'variant');
let input = [
{
"variantId": "5e1760fbdfaf28038242d676",
"quantity": 5
},
{
"variantId": "5e17e67b73a34d53160c7252",
"quantity": 13
}
]
let bulkArr = [];
for (const i of input) {
bulkArr.push({
updateOne: {
"filter": { "_id": Mongoose.Types.ObjectId(i.variantId) },
"update": { $inc: { "stocks": - i.quantity } }
}
})
}
Variant.bulkWrite(bulkArr)
Ref : MongoDB-bulkWrite
I don't think this can be done with a single Model.updateMany query. You will need to loop the array and use Model.update instead.
for (const { variantId, quantity } of objects) {
Model.update({ _id: variantId }, { $inc: { stocks: -quantity } });
}
To run this in a transaction (https://mongoosejs.com/docs/transactions.html), the code should look something like this (however I have not tried or tested this):
mongoose.startSession().then(async session => {
session.startTransaction();
for (const { variantId, quantity } of objects) {
await Model.update({ _id: variantId }, { $inc: { stocks: -quantity } }, { session });
}
await session.commitTransaction();
});
Context:
I am trying to upsert in bulk an array of data, with an additional computed field: 'status'.
Status should be either :
- 'New' for newly inserted docs;
- 'Removed' for docs present in DB, but inexistent in incoming dataset;
- a percentage explaining the evolution for the field price, comparing the value in DB to the one in incoming dataset.
Implementations:
data.model.ts
import { Document, model, Model, models, Schema } from 'mongoose';
import { IPertinentData } from './site.model';
const dataSchema: Schema = new Schema({
sourceId: { type: String, required: true },
name: { type: String, required: true },
price: { type: Number, required: true },
reference: { type: String, required: true },
lastModified: { type: Date, required: true },
status: { type: Schema.Types.Mixed, required: true }
});
export interface IData extends IPertinentData, Document {}
export const Data: Model<IData> = models.Data || model<IData>('Data', dataSchema);
data.service.ts
import { Data, IPertinentData } from '../models';
export class DataService {
static async test() {
// await Data.deleteMany({});
const data = [
{
sourceId: 'Y',
reference: `y0`,
name: 'y0',
price: 30
},
{
sourceId: 'Y',
reference: 'y1',
name: 'y1',
price: 30
}
];
return Data.bulkWrite(
data.map(function(d) {
let status = '';
// #ts-ignore
console.log('price', this);
// #ts-ignore
if (!this.price) status = 'New';
// #ts-ignore
else if (this.price !== d.price) {
// #ts-ignore
status = (d.price - this.price) / this.price;
}
return {
updateOne: {
filter: { sourceId: d.sourceId, reference: d.reference },
update: {
$set: {
// Set percentage value when current price is greater/lower than new price
// Set status to nothing when new and current prices match
status,
name: d.name,
price: d.price
},
$currentDate: {
lastModified: true
}
},
upsert: true
}
};
}
)
);
}
}
... then in my backend controller, i just call it with some route :
try {
const results = await DataService.test();
return new HttpResponseOK(results);
} catch (error) {
return new HttpResponseInternalServerError(error);
}
Problem:
I've tried lot of implementation syntaxes, but all failed either because of type casting, and unsupported syntax like the $ symbol, and restrictions due to the aggregation...
I feel like the above solution might be closest to a working scenario but i'm missing a way to grab the value of the price field BEFORE the actual computation of status and the replacement with updated value.
Here the value of this is undefined while it is supposed to point to current document.
Questions:
Am i using correct Mongoose way for a bulk update ?
if yes, how to get the field value ?
Environment:
NodeJS 13.x
Mongoose 5.8.1
MongoDB 4.2.1
EUREKA !
Finally found a working syntax, pfeeeew...
...
return Data.bulkWrite(
data.map(d => ({
updateOne: {
filter: { sourceId: d.sourceId, reference: d.reference },
update: [
{
$set: {
lastModified: Date.now(),
name: d.name,
status: {
$switch: {
branches: [
// Set status to 'New' for newly inserted docs
{
case: { $eq: [{ $type: '$price' }, 'missing'] },
then: 'New'
},
// Set percentage value when current price is greater/lower than new price
{
case: { $ne: ['$price', d.price] },
then: {
$divide: [{ $subtract: [d.price, '$price'] }, '$price']
}
}
],
// Set status to nothing when new and current prices match
default: ''
}
}
}
},
{
$set: { price: d.price }
}
],
upsert: true
}
}))
);
...
Explanations:
Several problems were blocking me :
the '$field_value_to_check' instead of this.field with undefined 'this' ...
the syntax with $ symbol seems to work only within an aggregation update, using update: [] even if there is only one single $set inside ...
the first condition used for the inserted docs in the upsert process needs to check for the existence of the field price. Only the syntax with BSON $type worked...
Hope it helps other devs in same scenario.
I am creating a list of scores for a user in mongoDB by adding a new score 1 at a time and sorting the list. I want to remove the lowest score when the list grows larger than 5 elements.
The reason for this is because I want to store the top 5 scores of the user.
What would be the best way to do this? Is there a way to make the whole thing an atomic operation?
My code is below. I'm using NodeJS with Mongoose and MongoDB.
const maxScoresToStore = 5
var scoreEntrySchema = new Schema({
score: Number,
when: { type: Date, default: Date.now }
})
var scoreSchema = new Schema({
_userid: { type: Schema.Types.ObjectId, ref: 'Users' },
username: {type: String, index:{unique: true}},
scores: [scoreEntrySchema]
})
const scoreModel = mongoose.model("Scores", scoreSchema)
exports.addUserScore = (uid, uname, score) => {
var query = {_userid:uid, username:uname},
update = { $push : {"scores" : {$each: [{"score": score}], $sort: {"score":-1}}} }, // sorts in descending order after pushing
options = { upsert: true, new: true, setDefaultsOnInsert: true };
scoreModel.findOneAndUpdate(query, update, options).then(
(result)=>{
if(result.scores.length > maxScoresToStore)
{
// ToDo:
// result.update({$pop: {"scores" : 1 }}) // pops the last element of the list
}
}
)
}
You can use $slice operator, And your query looks like:
let score = await scoreModel.findOneAndUpdate({ _userid: uid, username: uname },
{
$push: {
scores: {
$each: [{ score: score }],
$sort: { score: -1 },
$slice: maxScoresToStore
}
}
},
{
upsert: true,
new: true,
setDefaultsOnInsert: true,
multi: true
});
[DO VOTE TO THIS ANSWER, IF ITS HELPFUL TO YOU]
You can add slice option to your update option:
update = {
$push: {
scores: { $each: [{ score: score }], $sort: { score: -1 }, $slice: maxScoresToStore }
}
}
Here is the full method code written in async/await style:
exports.addUserScore = async (uid, uname, score) => {
const query = { _userid: uid, username: uname };
const update = {
$push: {
scores: {
$each: [{ score: score }],
$sort: { score: -1 },
$slice: maxScoresToStore
}
}
};
const options = {
upsert: true,
new: true,
setDefaultsOnInsert: true,
multi: true
};
try {
let score = await scoreModel.findOneAndUpdate(query, update, options);
if (!score) res.send(404).send("Score not found");
res.send("Everything is ok");
} catch (err) {
console.log(err);
res.status(500).send("Something went wrong");
}
};
I'm not certain If this would help, but it might work
scoreModel.update(
{ "scores.5": { "$exists": 1 } },
{ "$pop": { "scores": 1 } },
{ "multi": true }
)
As you are already sorting by descending, you can check if the array length is greater than 5 by using scores.5, If this returns true then you can pop the last element using $pop.
If $exists return false then it will skip the query. you can run this update after .then() and you won't have to use if condition.
But keep in mind $pop will only remove 1 element.
I got this query :
exports.search = (req, res) => {
let lat1 = req.body.lat;
let lon1 = req.body.lng;
let page = req.body.page || 1;
let perPage = req.body.perPage || 10;
let radius = req.body.radius || 100000; // This is not causing the issue, i can remove it and the issue is still here
var options = { page: page, limit: perPage, sortBy: { updatedDate: -1 } }
let match = {}
var aggregate = null;
if (lat1 && lon1) {
aggregate = Tutor.aggregate([
{
"$geoNear": {
"near": {
"type": "Point",
"coordinates": [lon1, lat1]
},
"distanceField": "distance", // this calculated distance will be compared in next section
"distanceMultiplier": 0.001,
"spherical": true,
"key": "loc",
"maxDistance": radius
}
},
{
$match: match
},
{ "$addFields": { "islt": { "$cond": [{ "$lt": ["$distance", "$range"] }, true, false] } } },
{ "$match": { "islt": true } },
{ "$project": { "islt": 0 } }
])
// .allowDiskUse(true);
} else {
aggregate = Tutor.aggregate([
{
$match: match
}
]);
}
Tutor
.aggregatePaginate(aggregate, options, function (err, result, pageCount, count) {
if (err) {
console.log(err)
return res.status(400).send(err);
}
else {
var opts = [
{ path: 'levels', select: 'name' },
{ path: 'subjects', select: 'name' },
{ path: 'assos', select: 'name' }
];
Tutor
.populate(result, opts)
.then(result2 => {
return res.send({
page: page,
perPage: perPage,
pageCount: pageCount,
documentCount: count,
tutors: result2
});
})
.catch(err => {
return res.status(400).send(err);
});
}
})
};
The query is supposed to retrieve all the tutors in a given range (which is a field from the tutor model, an integer in km, indicating how far the tutor is willing to move) around a certain location. (lat1, lon1).
The issue is that all the documents are not returned. After many tests, I have noticed that only tutors that are less than approximatively 7.5km away from the location are returned and not the others. Even if the tutor is 10km away and has a range of 15km, he won't be returned as he is farer than 7.5km.
I have tried switching location between two tutors (one that is returned and one that is not but should be) to see if this is the only thing causing the issue and it is. After I switched their location (lng and loc), the one that was returned before is no longer and vice versa.
I really don't get why this is happening.
Also, I know the result size is less than 16MB since I don't get all the results, even with allowDiskUse:true.
If you have any other idea about why I'm not getting all the results, don't hesitate !
Thank you !
PS : this is a part of the tutor model with the concerned fields (loc):
import mongoose from 'mongoose';
import validate from 'mongoose-validator';
import { User } from './user';
import mongooseAggregatePaginate from 'mongoose-aggregate-paginate';
var ObjectId = mongoose.Schema.Types.ObjectId;
var rangeValidator = [
validate({
validator: (v) => {
v.isInteger && v >= 0 && v <= 100;
},
message: '{VALUE} is a wrong value for range'
})
];
var tutorSchema = mongoose.Schema({
fullName: {
type: String,
trim: true,
minlength: [1, 'Full name can not be empty'],
required: [true, 'Full name is required']
},
location: {
address_components: [
{
long_name: String,
short_name: String,
types: String
}
],
description: String,
lat: Number,
lng: Number
},
loc: {
type: { type: String },
coordinates: []
},
});
tutorSchema.plugin(mongooseAggregatePaginate);
tutorSchema.index({ "loc": "2dsphere" });
var Tutor = User.discriminator('Tutor', tutorSchema);
module.exports = {
Tutor
};
The user model is using two indexes. The ID and this one ;
db['system.indexes'].find() Raw Output
{
"v": 2,
"key": {
"loc": "2dsphere"
},
"name": "loc_2dsphere",
"background": true,
"2dsphereIndexVersion": 3,
"ns": "verygoodprof.users"
}
I also have some similar kind of problem, in my case there is problem with limit
https://docs.mongodb.com/manual/reference/operator/aggregation/geoNear/
by default limit is 100 (Optional. The maximum number of documents to return. The default value is 100).
If you want you can increase the limit. Hope it help
You are using spherical: true, having the semantics different from planar geospacial queries.
Also, since you are using a GeoJSON object, instead of plain coordinates, the distance MUST be provided in meters (or in kilometers, since you are using the .001 multiplier)
When using spherical: true, it uses the $nearSphere query: https://docs.mongodb.com/manual/reference/operator/query/nearSphere/#op._S_nearSphere
With spherical: false, mongo uses the $near query: https://docs.mongodb.com/manual/reference/operator/query/near/#op._S_near
Since you are working with Lat/Lng, meaning planar coordinates, you should disable the spherical option.
The distance returned by $geoNear in case of "spherical": true is in radians so you need to change the "distanceMultiplier": 6371 to be equal to earth radius to get the distance in Km.