the problem I am facing is as follows:
Search value: 'cooking'
JSON object::
data: {
skills: {
items: [ { name: 'cooking' }, ... ]
}
}
Expected result: Should find all the "skill items" that contain 'cooking' inside their name, using TypeORM and Nest.js.
The current code does not support search on the backend, and I should implement this. I want to use TypeORM features, rather than handling it with JavaScript.
Current code: (returns data based on the userId)
const allItems = this.dataRepository.find({ where: [{ user: { id: userId } }] })
I investigated the PostgreSQL documentation regarding the PostgreSQL functions and even though I understand how to create a raw SQL query, I am struggling to convert this to the TypeORM equivalent.
Note: I researched many StackOverflow issues before creating this question, but do inform me If I missed the right one. I will be glad to investigate.
Can you help me figure out the way to query this with TypeORM?
UPDATE
Let's consider the simple raw query:
SELECT *
FROM table1 t
WHERE t.data->'skills' #> '{"items":[{ "name": "cooking"}]}';
This query will provide the result for any item within the items array that will match exact name - in this case, "cooking".
That's totally fine, and it can be executed as a raw request but it is certainly not easy to maintain in the future, nor to use pattern matching and wildcards (I couldn't find a solution to do that, If you know how to do it please share!). But, this solution is good enough when you have to work on the exact matches. I'll keep this question updated with the new findings.
use Like in Where clause:
servicePoint = await this.servicePointAddressRepository.find({
where: [{ ...isActive, name: Like("%"+key+"%"), serviceExecutive:{id: userId} },
{ ...isActive, servicePointId: Like("%"+key+"%")},
{ ...isActive, branchCode: Like("%"+key+"%")},
],
skip: (page - 1) * limit,
take: limit,
order: { updatedAt: "DESC" },
relations:["serviceExecutive","address"]
});
This may help you! I'm matching with key here.
I am searching if it exists, in mongodb or its nodejs driver, any method to merge/update a subdocument like this:
The sample document, for example in collection C:
{
subdocument: {
a: 1,
b: 2
}
}
The update query that I do NOT want to use:
db.C.updateOne(
{},
{
$set: {
"subdocument.b": 3
}
}
}
The update query that I want to use:
db.C.updateOne(
{},
{
$set: {
subdocument: {
b: 3
}
}
}
}
The resulting document I get when running such a query:
{
subdocument: {
b: 3
}
}
The resulting merged document I would like to get instead:
{
subdocument: {
a: 1,
b: 3
}
}
For the record, the reason I want this is because I'm trying to use interfaces to avoid as much as possible writing schema keywords inside strings, so I can have as many of them checked by the typescript compiler as possible. So i know the way to do this is to update "subdocument.b", but I am exactly trying to not do this, to avoid using a string.
Obviously, the subdocument is not merged but fully replaced using the standard update witout options. I would like to know if there is a way to do this natively using mongodb query language, aggregation framework, mongodb nodejs driver, or maybe something else? From what i could learn by myself, it seems unsupported, but maybe it is and someone can tell me how?
Thanks
Its very easy to do that if you use pipeline updates MongoDB >= 4.2
With pipeline updates all aggregate operators can be used, but only limited stages see the database command , drivers use that internally.
If you use pipeline updates you can use only aggregation operators, or query operators for the $match, only if you use $expr.The old update operators doesn't work inside a pipeline.
Test code here
db.collection.update({},
[
{
"$addFields": {
"subdocument": {
"$mergeObjects": [
"$subdocument",
{
"b": 3
}
]
}
}
}
])
Summary:
I am building my first large scale full stack application(MERN stack) that is trying to mimic a large scale clothing store. Each article of clothing has many 'tags' that represent its features, top/bottom/accessory/shoes/ect, and subcategories, for example on top there is shirt/outerwear/sweatshirt/etc, and sub-sub-categories within it, for example on shirt there is blouse/t-shirt/etc. Each article has tags for primary colors, hemline, pockets, technical features, the list goes on.
Main question:
how should I best organize the data in mongodb with mongoose schemas in order for it to be quickly searchable when I plan on having 50,000 or more articles? And genuinely curious, how do large clothing retailers typically design databases to be easily searchable by customers when items have so many identifying features?
Things I have tried or thought of:
On the mongoDB website there is a recommendation to use a tree structure with child references. here is the link: https://docs.mongodb.com/manual/tutorial/model-tree-structures-with-child-references/ I like this idea but I read here: https://developer.mongodb.com/article/mongodb-schema-design-best-practices/ that when storing over a few thousand pieces of data, using object ID references is no longer sufficient, and could create issues because of datalimits.
Further, each clothing item would fall into many different parts of the tree. For example it could be a blouse so it would be in the blouse "leaf" of the tree, and then if its blue, it would be in the blue "leaf" of the tree, and if it is sustainably sourced, it would fall into that "leaf" of the tree as well. Considering this, a tree like data structure seems not the right way to go. It would be storing the same ObjectID in many different leaves.
My other idea was to store the article information (description, price, and picture) seperate from the tagging/hierarchical information. Then each tagging object would have a ObjectID reference to the item. This way I could take advantage of the propogate method of mongoose if I wanted to collect that information.
I also created part of the large tree structure as a proof of concept for a design idea I had, and this is only for the front end right now, but this also creates bad searches cause they would look like taxonomy[0].options[0].options[0].options[0].title to get to 'blouse'. Which from my classes doesnt seem like a good way to make the code readable. This is only a snippet of a long long branching object. I was going to try to make this a mongoose schema. But its a lot of work and I wanna make sure that I do it well.
const taxonomy = [
{
title: 'Category',
selected: false,
options: [
{
title: 'top',
selected: false,
options: [
{
title: 'Shirt',
selected: false,
options: [
{
title: 'Blouse',
selected: false,
},
{
title: 'polo',
selected: false,
},
{
title: 'button down',
selected: false,
},
],
},
{
title: 'T-Shirt',
selected: false,
},
{
title: 'Sweater',
selected: false,
},
{
title: 'Sweatshirt and hoodie',
selected: false,
},
],
},
Moving forward:
I am not looking for a perfect answer, but I am sure that someone has tackled this issue before (all big businesses that sell lots of categorized products have) If someone could just point me in the right direction, for example, give me some terms to google, some articles to read, or some videos to watch, that would be great.
thank you for any direction you can provide.
MongoDB is a document based database. Each record in a collection is a document, and every document should be self-contained (it should contain all information that you need inside it).
The best practice would be to create one collection for each logical whole that you can think of. This is the best practice when you have documents with a lot of data, because it is scalable.
For example, you should create Collections for: Products, Subproducts, Categories, Items, Providers, Discounts...
Now, when you creating Schemas, instead of creating nested structure, you can just store a reference of one collection document as a property of another collection document.
NOTE: The maximum document size is 16 megabytes.
BAD PRACTICE
Let us first see what would be the bad practice. Consider this structure:
Product = {
"name": "Product_name",
"sub_products": [{
"sub_product_name": "Subpoduct_name_1",
"sub_product_description": "Description",
"items": [{
"item_name": "item_name_1",
"item_desciption": "Description",
"discounts": [{
"discount_name": "Discount_1",
"percentage": 25
}]
},
{
"item_name": "item_name_2",
"item_desciption": "Description",
"discounts": [{
"discount_name": "Discount_1",
"percentage": 25
},
{
"discount_name": "Discount_2",
"percentage": 50
}]
},
]
},
...
]
}
Here product document has sub_products property which is an array of sub_products. Each sub_product has items, and each item has discounts. As you can see, because of this nested structure, the maximum document size would be quickly exceeded.
GOOD PRACTICE
Consider this structure:
Product = {
"name": "Product_name",
"sub_products": [
'sub_product_1_id',
'sub_product_2_id',
'sub_product_3_id',
'sub_product_4_id',
'sub_product_5_id',
...
]
}
Subproduct = {
"id": "sub_product_1_id",
"sub_product_name": "Subroduct_name",
"sub_product_description": "Description",
"items": [
'item_1_id',
'item_2_id',
'item_3_id',
'item_4_id',
'item_5_id',
...
]
}
Item = {
"id": "item_1_id",
"item_name": "item_name_1",
"item_desciption": "Description",
"items": [
'discount_1_id',
'discount_2_id',
'discount_3_id',
'discount_4_id',
'discount_5_id',
...
]
}
Discount = {
"id": "discount_1_id",
"discount_name": "Discount_1",
"percentage": 25
}
Now, you have collection for each logical whole and you are just storing a reference of one collection document as a property of another collection document.
Now you can use one of the best features of the Mongoose that is called population. If you store a reference of one collection document as a property of another collection document, when performing querying of the database, Mongoose will replace references with the actual documents.
I hope I don't miss something obvious, but I haven't seen anywhere a syntax like that.
I'm working for an online courses provider where the user subscribes by topic.
A course can have multiple topics
User must have subscribed for all topics of that course to have access to it
I know how to make a request like { topics: { $nin : ...allTheTopicsUserHaveNoAccessTo } } but that forces me to get all "not allowed" topics first.
Is there a way to make that kind of request in one call ?
User model:
const userModel = {
allowedTopics: [ 'math', 'physics' ],
}
Course model:
const courseModel = {
topics: [ 'math', 'physics', 'biology' ], // user need to have subscribed to those three to see it
}
setIsSubset does it in a single command and can be used in a regular find with $expr operator:
db.course.find({
$expr: {
$setIsSubset: [
"$topics",
user.allowedTopics
]
}
})
https://mongoplayground.net/p/JfSiSrboOuj
It still won't let you benefit from multikey indexes though.
Yes it can be done.
Aggregation with $setDifference followed by a match for empty array will accomplish that.
However, such an aggregation pipeline would not be able to make use of an index, and would have to fetch from disk and examine every course in the entire catalog for every single run.
I am trying to write a lookup which returns an array from a document and skipping some fields:
{
"id": 10000,
"schedule": [
{
"day": 0,
"flight": "AF198",
"utc": "10:13:00"
},
{
"day": 0,
"flight": "AF547",
"utc": "19:14:00"
},
...
]
}
I would like to get all schedule items but only the flight properties. I want to get something like this:
[
{
"flight: "AF198"
},
{
"flight: "AF547"
},
...
]
bucket.lookupIn(key).get("schedule.flight") doesn’t work. I tried "schedule[].flight", "schedule.$.flight" It seems I always need to know the index.
I saw that this is possible with N1QL.
Couchbase - SELECT a subset of fields from array of objects
Do you guys know how to do this with the Subdocument API? Sorry if it is a trivial question. I just cannot find an example on
https://developer.couchbase.com/documentation/server/current/sdk/subdocument-operations.html
Couchbase Subdocument requires the full path, it does not support expansion. In this case it needs to know the index of the array. There are a few other options:
If every path is known, then you could chain all of the subdocument gets. A total of 16 paths can be got at once:
bucket.lookupIn(key).get("schedule[0].flight").get(schedule[1].flight")
Get the parent object and filter on the application side:
bucket.lookupIn(key).get("schedule")
As mentioned in the question, use N1QL.