When Implementing a search filter on a document withh populated documents and array of object, How can I enable the search filter to checkout the populated document or array for the searched word?
What I have only checks through the surface fields
async findAllFeedbacks(user: any, payload: FilterDto) {
if (payload.fields) {
const query = await this.distributionChannelsDocumentModel
.find({
[payload.fields]: {
$regex: `${payload.searchFilter}`,
$options: 'i'
},
})
.clone()
.where({ business: user.business })
.populate({
path: 'feedbackId',
// match: { 'feedbackId[payload.fields]': { $regex: `${payload.searchFilter}` } }
})
return { query, total: query.length }
} else {
const query = await this.distributionChannelsDocumentModel
.find()
.clone()
.where({ business: user.business })
.populate('feedbackId')
return { query, total: query.length }
}
}
Here is a response sample:
{
"query": [
{
"_id": "62a8645641b312b3a04a7233",
"feedbackId": {
"_id": "62a8642b41b312b3a04a722a",
"title": "Monthly Feedback test",
"questions": [
{
"title": "Rate Our Services test",
"subtitle": "Customer retention is our priority",
"type": "SMILEY"
}
],
"thankYouMessage": "Thanks for dropping a feedback",
"allowComment": true,
"allowEmail": true,
"allowFullName": false,
"allowAttachment": false,
"status": "Active",
"business": "62a83f5ab4b33de7307e9cb8",
"user": "62a83f59b4b33de7307e9cb6",
"createdAt": "2022-06-14T10:34:19.709Z",
"updatedAt": "2022-06-14T10:34:19.709Z",
"__v": 0
},
"user": "62a83f59b4b33de7307e9cb6",
"distributionChannelType": [
"email"
],
"meta": [
{
"sender": "string",
"subject": "string",
"positioning": "string",
"timing": "string",
"brandColor": "string"
}
],
"createdAt": "2022-06-14T10:35:02.092Z",
"updatedAt": "2022-06-14T10:35:02.092Z",
"__v": 0
},
{
"_id": "62b1c87510fae7f3a54b13b2",
"feedbackId": {
"_id": "62ac55d0b28630c3a8c9b940",
"title": "Yearly Feedback",
"questions": [
{
"title": "Rate Our Services",
"subtitle": "We love to hear from you! How is your Jumia food experience",
"type": "SMILEY"
}
],
"thankYouMessage": "Thanks for dropping a feedback",
"allowComment": true,
"allowEmail": true,
"allowFullName": false,
"allowAttachment": false,
"status": "Active",
"business": "62ac434542452ccb513f2241",
"user": "62ac434542452ccb513f223f",
"createdAt": "2022-06-17T10:22:08.863Z",
"updatedAt": "2022-06-17T10:22:08.863Z",
"__v": 0
},
"user": "62ac434542452ccb513f223f",
"distributionChannelType": [
"sms"
],
"meta": [
{
"position": "bottom",
"timing": "",
"brandColor": "#444"
}
],
"createdAt": "2022-06-21T13:32:37.895Z",
"updatedAt": "2022-06-21T13:32:37.895Z",
"__v": 0
}
],
"total": 2
}
In the above response, feedbackId is the populated document and questions is the array I would also want to search through when using the search filter
Related
I have millions of documents in my es index.
I wanted to fetch the documents where the array field length greater than 0.
My docs looks like this
[
{
"primaryKey": "9c30d9e8-af04-4cc8-afcb-0c1311988c1e",
"language": "all",
"industry": [
"Accounting & auditing"
],
"text": "what's the status of my incident?",
"textId": "d0c70fc4-5e2a-4cab-a5f6-32339e6632dd",
"extractions": [],
"active": true,
"status": "active",
"createdAt": 1620208485092,
"updatedAt": 1620208485092,
"secondaryKey": "5db5f725-ec09-49da-9507-7bb2f94fd741"
},
{
"primaryKey": "9c30d9e8-af04-4cc8-afcb-0c1311988c1e",
"language": "all",
"industry": [
"Accounting"
],
"text": "What is the rating of my incident",
"textId": "4a53533f-293e-440c-aaa9-f7e5ae1436ca",
"extractions": [
{
"name": "Abinas Patra",
"role": "api-user",
"primaryKey": "ed12851d-c18d-4c92-8cc3-1782e41bc9d0"
},
{
"name": "Anil Patra",
"role": "ui-user",
"primaryKey": "933fad33-78b3-4779-a7bd-c62c6e02af75"
}
],
"active": true,
"status": "active",
"createdAt": 1620208485092,
"updatedAt": 1620208485092,
"secondaryKey": "5db5f725-ec09-49da-9507-7bb2f94fd741"
}
]
I am using elasticsearch nodejs client.
I tried in the below way
let dataCount = await esClient.count({
index: "indexName",
type: "docType",
body: {
query: {
bool: {
must: [
{
"script": {
"script": {
"inline": "doc['extractions'].values.length > 0",
"lang": "painless"
}
}
},
{
"match": {
"primaryKey": {
query: primaryKey,
"operator": "and"
}
}
},
{
"match": {
"language": {
query: language,
"operator": "and"
}
}
}
]
}
}
}
});
I get runtime parsing error everytime, i tried with exist field as well.
{"error":{"root_cause":[{"type":"script_exception","reason":"runtime error","script_stack":["org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:65)","org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:27)","doc[\'extractions\'].values.length > 1"," ^---- HERE"],
tried this as well
must_not:[
{
"script": {
"script": "_source.extractions.size() > 0"
}
}
]
Can anyone please help here.
thanks :)
I need to filter some users according to some fixed criteria. I have a user collection and a talent collection. The talent collection holds the reference to a master category collection.
What I need is to filter these users according to the category in the talent collection and some keys from the user collection.
For example I need to search for a user whose gender is 'male' and education 'BTech' and will have talents as a programmer and tester
my user collection is like,
{
"_id": "5f1939239bd35429ac9cd78f",
"isOtpVerified": "false",
"role": "user",
"adminApproved": 1,
"status": 0,
"languages": "Malayalam, Tamil, Telugu, Kannada",
"name": "Test user",
"email": "test#email.com",
"phone": "1234567890",
"otp": "480623",
"uid": 100015,
"bio": "Short description from user",
"dob": "1951-09-07T00:00:00.000Z",
"gender": "Male",
"education": "Btech",
"bodyType": "",
"complexion": "",
"height": "",
"weight": "",
"requests": [],
"location": {
"place": "place",
"state": "state",
"country": "country"
},
"image": {
"avatar": "5f1939239bd35429ac9cd78f_avatar.jpeg",
"fullsize": "5f1939239bd35429ac9cd78f_fullsize.png",
"head_shot": "5f1939239bd35429ac9cd78f_head_shot.jpeg",
"left_profile": "5f1939239bd35429ac9cd78f_left_profile.png",
"right_profile": "5f1939239bd35429ac9cd78f_right_profile.png"
},
"__v": 42,
"createdAt": "2020-07-23T07:15:47.387Z",
"updatedAt": "2020-08-18T18:54:22.272Z",
}
Talent collection
[
{
"_id": "5f38efef179aca47a0089667",
"userId": "5f1939239bd35429ac9cd78f",
"level": "5",
"chars": {
"type": "Fresher",
},
"category": "5f19357b50bcf9158c6be572",
"media": [],
"createdAt": "2020-08-16T08:35:59.692Z",
"updatedAt": "2020-08-16T08:35:59.692Z",
"__v": 0
},
{
"_id": "5f3b7e6f7e322948ace30a2c",
"userId": "5f1939239bd35429ac9cd78f",
"level": "3",
"chars": {
"type": "Fresher",
},
"category": "5f19359250bcf9158c6be573",
"media": [
{
"adminApproved": 0,
"status": 0,
"_id": "5f3c22573065f84a48e04a14",
"file": "id=5f1939239bd35429ac9cd78f&dir=test&img=5f1939239bd35429ac9cd78f_image_undefined.jpeg",
"description": "test",
"fileType": "image",
"caption": "test file"
},
{
"adminApproved": 0,
"status": 0,
"_id": "5f3c2d7a8c7f8336b0bfced2",
"file": "id=5f1939239bd35429ac9cd78f&dir=test&img=5f1939239bd35429ac9cd78f_image_1.jpeg",
"description": "this is a demo poster for testing",
"fileType": "image",
"caption": "A Test Poster"
}
],
"createdAt": "2020-08-18T07:08:31.532Z",
"updatedAt": "2020-08-18T19:35:22.899Z",
"__v": 2
}
]
And the category in the above document is a separate one populated to this. the category collection as,
[
{
"_id": "5f19359250bcf9158c6be573",
"status": true,
"title": "Testing",
"description": "Application tester",
"code": "test",
"characteristics": [],
"createdAt": "2020-07-23T07:00:34.221Z",
"updatedAt": "2020-07-23T07:00:34.221Z",
"__v": 0
},
{
"status": true,
"_id": "5f29829a705b4e648c28bc88",
"title": "Designer",
"description": "UI UX Designer",
"code": "uiux",
"createdAt": "2020-08-04T15:45:30.125Z",
"updatedAt": "2020-08-04T15:45:30.125Z",
"__v": 0
},
{
"_id": "5f19357b50bcf9158c6be572",
"status": true,
"title": "programming",
"description": "Java programmer",
"code": "program",
"createdAt": "2020-07-23T07:00:11.137Z",
"updatedAt": "2020-07-23T07:00:11.137Z",
"__v": 0
}
]
So my filter terms will be;
{
categories: ["5f19359250bcf9158c6be573", "5f19357b50bcf9158c6be572"],
minAge: 18,
maxAge: 25,
minHeight: 5,
maxHeight: 6,
minWeight: 50,
maxWeight: 80,
complexion: "white",
gender: "male",
}
And the expected result will be a user have both the above talents and followed conditions,
{
users: { ..User details.. },
medias: { ...medias from the matching talents.. }
}
If there are two collections you need to join them either by primary key or _id with foriegn fields and you can use $lookup with $match to filter down.
Documentation
You need to use $lookup with pipeline,
$match you condition for category match
$lookup to join users collection
$match conditions for users collections fields
$match exclude documents that don't found matching users of criteria passed in conditions
db.talents.aggregate([
{
$match: {
category: { $in: ["5f19359250bcf9158c6be573", "5f19357b50bcf9158c6be572"] }
}
},
{
$lookup: {
from: "users",
as: "users",
let: { userId: "$userId" },
pipeline: [
{
$match: {
$expr: {
$and: [
{ $eq: ["$$userId", "$_id"] },
{ $eq: ["$gender", "Male"] },
{ $eq: ["$education", "Btech"] }
// ... add you other match criteria here
]
}
}
}
]
}
},
{ $match: { users: { $ne: [] } } }
])
Playground
I'll be grateful for help with Mongoose. I have 3 tables: (Users)Table of users, (Animals)table of animals and table AnimalComments. So (AnimalComments)table reference users and animals.
const schemaComment = new mongoose.Schema({
userRef: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User'
},
animalRef: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Animal'
},
content: {
type: String
}
});
const schemaAnimal = new mongoose.Schema({
name: {
type: String
},
isCommentedByMe: {
type : Boolean,
default: false
},
commentCount: {
type : Number,
default: 0
}
});
What I want: I have animals. Users can add comment to animal. When user comment animal his comment is added to table AnimalComments where is stored userRef (userId), animalRef (animalId) and user comment text. Then in request response I want return all animals from table Animals but I need update property commentCount and isCommentedByMe based on value from table AnimalComments.
Response from table Animals:
{
"animals": [
{
"isCommentedByMe": false,
"commentCount": 0,
"name": "Jessica",
"userRef": {
"id": "5dc9bdf3dd5cae00177e184d"
},
"id": "5dcedd48368e9800176f2ef3"
}
]
}
Response from table Users:
{
"users": [
{
"name": "Jony Cash",
"id": "5dc9bdf3dd5cae00177e184d"
}
]
}
Response from table AnimalComments:
{
"comments": [
{
"userRef": "5dc9bdf3dd5cae00177e184d",
"animalRef": "5dcedd48368e9800176f2ef3",
"content": "Sample text"
}
]
}
I want result for exmaple:
{
"animals": [
{
"isCommentedByMe": true,
"commentCount": 4,
"name": "Jessica",
"userRef": {
"id": "5dc9bdf3dd5cae00177e184d"
},
"id": "5dcedd48368e9800176f2ef3"
}
]
}
You don't need to keep isCommentedByMe and commentCount fields in animal schema.
And you need to be able access comments from your animals. But in animal schema there is no field to make that connection. So we need to use virtual population.
So your animal schema must be like this:
const schemaAnimal = new mongoose.Schema(
{
name: {
type: String
}
},
{
toJSON: { virtuals: true },
toObject: { virtuals: true }
}
);
// Virtual populate
schemaAnimal.virtual("comments", {
ref: "Comment", //must be changed to the name you used for Comment model.
foreignField: "animalRef",
localField: "_id"
});
Now, we can use the following code to populate the comments.
router.get("/animals", async (req, res) => {
const animals = await Animal.find({}).populate("comments");
res.send(animals);
});
This will give you a result like this:
[
{
"_id": "5dd66c73069f88614c12b394",
"name": "Animal 1",
"__v": 0,
"comments": [
{
"_id": "5dd66cfd069f88614c12b39a",
"userRef": "5dd66b54c5195127ec5a1b82",
"animalRef": "5dd66c73069f88614c12b394",
"content": "User 1 - Animal 1",
"__v": 0
},
{
"_id": "5dd66d30069f88614c12b39d",
"userRef": "5dd66b71c5195127ec5a1b83",
"animalRef": "5dd66c73069f88614c12b394",
"content": "User 2 - Animal 1",
"__v": 0
}
],
"id": "5dd66c73069f88614c12b394"
},
{
"_id": "5dd66c7d069f88614c12b395",
"name": "Animal 2",
"__v": 0,
"comments": [
{
"_id": "5dd66d09069f88614c12b39b",
"userRef": "5dd66b54c5195127ec5a1b82",
"animalRef": "5dd66c7d069f88614c12b395",
"content": "User 1 - Animal 2",
"__v": 0
}
],
"id": "5dd66c7d069f88614c12b395"
},
{
"_id": "5dd66c88069f88614c12b396",
"name": "Animal 3",
"__v": 0,
"comments": [
{
"_id": "5dd66d46069f88614c12b39e",
"userRef": "5dd66b71c5195127ec5a1b83",
"animalRef": "5dd66c88069f88614c12b396",
"content": "User 2 - Animal 3",
"__v": 0
}
],
"id": "5dd66c88069f88614c12b396"
}
]
To convert this result to what you wanted, we can use map like this:
Please note that, you need to set loggedInUserId variable to logged in user's id.
router.get("/animals", async (req, res) => {
const loggedInUserId = "5dd66b54c5195127ec5a1b82";
const animals = await Animal.find({}).populate("comments");
const result = animals.map(animal => {
return {
id: animal._id,
name: animal.name,
isCommentedByMe:
animal.comments.filter(c => c.userRef.toString() === loggedInUserId)
.length > 0,
commentCount: animal.comments.length
};
});
res.send(result);
});
The result will be like this:
[
{
"id": "5dd66c73069f88614c12b394",
"name": "Animal 1",
"isCommentedByMe": true,
"commentCount": 2
},
{
"id": "5dd66c7d069f88614c12b395",
"name": "Animal 2",
"isCommentedByMe": true,
"commentCount": 1
},
{
"id": "5dd66c88069f88614c12b396",
"name": "Animal 3",
"isCommentedByMe": false,
"commentCount": 1
}
]
And the answer for the question in the comments is: (how to reference User)
const animals = await Animal.find({}).populate({
path: "comments",
model: Comment,
populate: [
{
path: "userRef",
model: User
}
]
});
This will give you the userRef like this:
[
{
"_id": "5dd66c73069f88614c12b394",
"name": "Animal 1",
"__v": 0,
"comments": [
{
"_id": "5dd66cfd069f88614c12b39a",
"userRef": {
"_id": "5dd66b54c5195127ec5a1b82",
"name": "User 1",
"__v": 0
},
"animalRef": "5dd66c73069f88614c12b394",
"content": "User 1 - Animal 1",
"__v": 0
},
{
"_id": "5dd66d30069f88614c12b39d",
"userRef": {
"_id": "5dd66b71c5195127ec5a1b83",
"name": "User 2",
"__v": 0
},
"animalRef": "5dd66c73069f88614c12b394",
"content": "User 2 - Animal 1",
"__v": 0
}
],
"id": "5dd66c73069f88614c12b394"
},
{
"_id": "5dd66c7d069f88614c12b395",
"name": "Animal 2",
"__v": 0,
"comments": [
{
"_id": "5dd66d09069f88614c12b39b",
"userRef": {
"_id": "5dd66b54c5195127ec5a1b82",
"name": "User 1",
"__v": 0
},
"animalRef": "5dd66c7d069f88614c12b395",
"content": "User 1 - Animal 2",
"__v": 0
}
],
"id": "5dd66c7d069f88614c12b395"
},
{
"_id": "5dd66c88069f88614c12b396",
"name": "Animal 3",
"__v": 0,
"comments": [
{
"_id": "5dd66d46069f88614c12b39e",
"userRef": {
"_id": "5dd66b71c5195127ec5a1b83",
"name": "User 2",
"__v": 0
},
"animalRef": "5dd66c88069f88614c12b396",
"content": "User 2 - Animal 3",
"__v": 0
}
],
"id": "5dd66c88069f88614c12b396"
}
]
I have 2 entities to merge: Customer and Feedback. Feedback contains an embedded array of upvotes (Upvote)
A customer is not able to upvote more than once for a specific feedback.
What I would like to achieve is - given a specific feedback id - get the complete list of customers with an additional virtual attribute that states whether he/she upvoted the given feedback.
Customer.aggregate(
[
{
$match: { company_id: new ObjectID(req.user.company_id) }
},
{
$lookup: {
from: 'feedbacks',
let: { 'c_id': '$_id' },
pipeline: [
{
$unwind: '$upvotes'
},
{
$match: { $expr: { $eq: ['$upvotes.customer_id._id', '$$c_id'] } }
}
],
as: 'upvotes'
}
}
],
function(err, customers) {
if (err) {
console.log(err);
res.status(400).send(err);
} else {
res.send({ customers });
}
}
);
To do that I have to look through the list of upvotes for that specific feedback and, then, join it with the customer table using the customer_id.
The above mentioned approach does not work. Any suggestion what I am doing wrong?
Sample data (Feedback)
{
"size": 0,
"points": 50,
"status": "open",
"potential": 0,
"real": 5000,
"_id": "5c3d033271ceb7edc37d156c",
"title": "Custom Invoice Templates",
"description": "Provide an editor to create custom invoices.",
"owner_id": {
"_id": "5c3b684f7cec8be977c2a465",
"email": "maurizio#acme.com"
},
"company_id": "5c3b684f7cec8be977c2a462",
"project_id": "5c3b68507cec8be977c2a468",
"upvotes": [
{
"_id": "5c3fa5b371ceb7edc37d159a",
"comments": "bbbb",
"priority": "should",
"customer_id": {
"size": 0,
"potential": 0,
"real": 5000,
"_id": "5c3b68507cec8be977c2a485",
"name": "Oyomia Ltd."
},
"owner_id": {
"_id": "5c3b684f7cec8be977c2a465",
"email": "maurizio#acme.com"
}
}
],
"updatedAt": "2019-01-16T21:44:19.215Z",
"createdAt": "2019-01-14T21:46:26.286Z",
"__v": 0
}
Sample data (Customer)
{
"size": 0,
"potential": 0,
"real": 5000,
"_id": "5c3b68507cec8be977c2a485",
"name": "Oyomia Ltd.",
"contact": {
"_id": "5c40f8de71ceb7edc37d15ab",
"name": "Nick Page",
"email": "np#oyoma.com"
},
"company_id": "5c3b684f7cec8be977c2a462",
"deals": [
{
"value": 5000,
"_id": "5c3b68507cec8be977c2a487",
"name": "Armour batch",
"status": "won",
"type": "non_recurring",
"updatedAt": "2019-01-13T16:33:20.870Z"
}
],
"__v": 0,
"updatedAt": "2019-01-17T21:51:26.877Z"
}
I'm getting below response from loopback when I filter with below api:
http://localhost:8001/myloopback/productDetails?q=product.productId==apple
I want to exclude the data which doesn't have property "productModel" - how can we achieve this in Loopback rest API. Is there any option to exclude property from the rest API query param? I'm new to Loopback - it would be really grateful if someone can help me to fix this
{
"myData": [{
"data": {
"product": {
"productId": "apple",
"productName": "iPhone",
"productModel": "6s"
},
"statusCode": "active",
"date": "2018-08-07T00:00:00.000Z"
},
"links": [{
"productUrl": "test"
},
{
"productImage": "test"
}
],
"info": {}
},
{
"data": {
"product": {
"productId": "apple",
"productName": "iPhone",
"productModel": "7"
},
"statusCode": "active",
"date": "2018-08-07T00:00:00.000Z"
},
"links": [{
"productUrl": "test"
},
{
"productImage": "test"
}
],
"info": {}
},
{
"data": {
"product": {
"productId": "apple",
"productName": "Macbook"
},
"statusCode": "active",
"date": "2018-08-07T00:00:00.000Z"
},
"links": [{
"productUrl": "test"
},
{
"productImage": "test"
}
],
"info": {}
}
],
"metadata": {
"count": 3,
"offset": 0
}
}
There is no such inbuilt function to do that but you can use multiple Fields filter to achieve that.