nodejs - mongodb $lte and $gte not working with $lookup - node.js

I need to get record by price range but the query is working fine in shell but not in nodejs. Here is query:
db.model.aggregate([
{ $match: { make_id: make_id } },
{
$lookup: {
from: 'variants',
localField: '_id',
foreignField: 'model_id',
as: 'variant_list'
}
},
{
$unwind: {
path: '$variant_list',
preserveNullAndEmptyArrays: false
}
},
{$match: {status:1, $and:[{'variant_list.price': {$lte:Number(max_price)}}]}},
{$project: { make_id: 1, model_name: 1, variant_list: '$variant_list'}}
])
I have checked the price in database. it is stored like NumberInt(300000) and in my query i have parsed in integer type like => max_price to Number(max_price).
and it gives me all record which are less than and equal to 300000 but it also gives the record which are less than 400000.
But If i'm not using aggregate framework then it is giving me perfect result:
db.model.find({status:1, $and:[{'variant_list.price': {$lte:Number(max_price)}}]})
I dont know why $lte not working perfectly with aggregate and $lookup.
Please Help!!
Updated Sample Collection and Documents
MakeId = a123;
Models:
-----------------------------
mid | makeid | model |
-----------------------------
m1 | a123 | MDLi1 |
m2 | a123 | MDLi2 |
m3 | a123 | MDLi3 |
m4 | a123 | MDLi4 |
m5 | a123 | MDLi5 |
-----------------------------
Variants:
----------------------------------------
vid | modelid | variants | price |
----------------------------------------
v1 | m1 | VARi1 | 150000 |
v2 | m1 | VARi2 | 250000 |
v3 | m1 | VARi3 | 300000 |
v4 | m1 | VARi4 | 350000 |
v5 | m1 | VARi5 | 380000 |
v6 | m2 | VARi6 | 250000 |
v7 | m2 | VARi7 | 280000 |
v8 | m2 | VARi8 | 380000 |
v9 | m2 | VARi9 | 400000 |
v10 | m2 | VARi10 | 450000 |
----------------------------------------
I need to get model detail and variants list by variant price like price range should be between 0 to 300000
My Desire result is :
{
"data": [
{
"makeid": "a123",
"modelid": "m1",
"variant_list": [
{
"variant_id": "v1",
"variant_name": "VARi1",
"price": 150000
},
{
"variant_id": "v2",
"variant_name": "VARi2",
"price": 250000
},
{
"variant_id": "v3",
"variant_name": "VARi3",
"price": 300000
}
]
},
{
"makeid": "a123",
"modelid": "m2",
"variant_list": [
{
"variant_id": "v6",
"variant_name": "VARi6",
"price": 250000
},
{
"variant_id": "v7",
"variant_name": "VARi7",
"price": 280000
}
]
}
]
}

Related

Create data frame from json step exaction history aws

I have got step function execution history in JSON format
[{
"timestamp": "2022-07-18T13:03:03.346000+00:00",
"type": "ExecutionFailed",
"id": 3,
"previousEventId": 2,
"executionFailedEventDetails": {
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'Workflow Choice state' (entered at the event id #2). Invalid path '$.contributor_id': The choice state's condition path references an invalid value."
}
}, {
"timestamp": "2022-07-18T13:03:03.306000+00:00",
"type": "ChoiceStateEntered",
"id": 2,
"previousEventId": 0,
"stateEnteredEventDetails": {
"name": "Workflow Choice state",
"input": "{\n \"Comment\": \"Insert your JSON here\"\n}",
"inputDetails": {
"truncated": false
}
}
}, {
"timestamp": "2022-07-18T13:03:03.252000+00:00",
"type": "ExecutionStarted",
"id": 1,
"previousEventId": 0,
"executionStartedEventDetails": {
"input": "{\n \"Comment\": \"Insert your JSON here\"\n}",
"inputDetails": {
"truncated": false
},
"roleArn": "arn:aws:iam::asdfg:role/step-all"
}
}]
We want to create a view like below
The issue is i am not able to create executionFailedEventDetails ,stateEnteredEventDetails,executionStartedEventDetails as new row .
It comes in first row only .
Step column is the name in the stateEnteredEventDetails
This is what i am doing
import json
import pandas as pd
from tabulate import tabulate
raw = r"""[{
"timestamp": "2022-07-18T13:03:03.346000+00:00",
"type": "ExecutionFailed",
"id": 3,
"previousEventId": 2,
"executionFailedEventDetails": {
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'Workflow Choice state' (entered at the event id #2). Invalid path '$.contributor_id': The choice state's condition path references an invalid value."
}
}, {
"timestamp": "2022-07-18T13:03:03.306000+00:00",
"type": "ChoiceStateEntered",
"id": 2,
"previousEventId": 0,
"stateEnteredEventDetails": {
"name": "Workflow Choice state",
"input": "{\n \"Comment\": \"Insert your JSON here\"\n}",
"inputDetails": {
"truncated": false
}
}
}, {
"timestamp": "2022-07-18T13:03:03.252000+00:00",
"type": "ExecutionStarted",
"id": 1,
"previousEventId": 0,
"executionStartedEventDetails": {
"input": "{\n \"Comment\": \"Insert your JSON here\"\n}",
"inputDetails": {
"truncated": false
},
"roleArn": "arn:aws:iam::asdfg:role/step-all"
}
}]"""
data = json.loads(raw, strict=False)
data = pd.json_normalize(data)
# print(data.to_csv(), index=False)
print(tabulate(data, headers='keys', tablefmt='psql'))
data.to_csv('file.csv',encoding='utf-8', index=False)
and the output is
+----+----------------------------------+--------------------+------+-------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------+----------------------------------------+---------------------------------------------------+----------------------------------------+-------------------------------------------------------+----------------------------------------+
| | timestamp | type | id | previousEventId | executionFailedEventDetails.error | executionFailedEventDetails.cause | stateEnteredEventDetails.name | stateEnteredEventDetails.input | stateEnteredEventDetails.inputDetails.truncated | executionStartedEventDetails.input | executionStartedEventDetails.inputDetails.truncated | executionStartedEventDetails.roleArn |
|----+----------------------------------+--------------------+------+-------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------+----------------------------------------+---------------------------------------------------+----------------------------------------+-------------------------------------------------------+----------------------------------------|
| 0 | 2022-07-18T13:03:03.346000+00:00 | ExecutionFailed | 3 | 2 | States.Runtime | An error occurred while executing the state 'Workflow Choice state' (entered at the event id #2). Invalid path '$.contributor_id': The choice state's condition path references an invalid value. | nan | nan | nan | nan | nan | nan |
| 1 | 2022-07-18T13:03:03.306000+00:00 | ChoiceStateEntered | 2 | 0 | nan | nan | Workflow Choice state | { | 0 | nan | nan | nan |
| | | | | | | | | "Comment": "Insert your JSON here" | | | | |
| | | | | | | | | } | | | | |
| 2 | 2022-07-18T13:03:03.252000+00:00 | ExecutionStarted | 1 | 0 | nan | nan | nan | nan | nan | { | 0 | arn:aws:iam::asdfg:role/step-all |
| | | | | | | | | | | "Comment": "Insert your JSON here" | | |
| | | | | | | | | | | } | | |
+----+----------------------------------+--------------------+------+-------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------+----------------------------------------+---------------------------------------------------+----------------------------------------+-------------------------------------------------------+----------------------------------------+
The Details column 5th column is dynamic like all columns currently i have given example of only 3 event but it can go up to any number .
Final Expected Output
Given file.json:
[{
"timestamp": "2022-07-18T13:03:03.346000+00:00",
"type": "ExecutionFailed",
"id": 3,
"previousEventId": 2,
"executionFailedEventDetails": {
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'Workflow Choice state' (entered at the event id #2). Invalid path '$.contributor_id': The choice state's condition path references an invalid value."
}
}, {
"timestamp": "2022-07-18T13:03:03.306000+00:00",
"type": "ChoiceStateEntered",
"id": 2,
"previousEventId": 0,
"stateEnteredEventDetails": {
"name": "Workflow Choice state",
"input": "{\n \"Comment\": \"Insert your JSON here\"\n}",
"inputDetails": {
"truncated": false
}
}
}, {
"timestamp": "2022-07-18T13:03:03.252000+00:00",
"type": "ExecutionStarted",
"id": 1,
"previousEventId": 0,
"executionStartedEventDetails": {
"input": "{\n \"Comment\": \"Insert your JSON here\"\n}",
"inputDetails": {
"truncated": false
},
"roleArn": "arn:aws:iam::asdfg:role/step-all"
}
}]
Doing
import pandas as pd
df = pd.read_json('file.json')
df = df.melt(['timestamp', 'type', 'id', 'previousEventId'], var_name='step', value_name='details').dropna()
print(df.to_markdown(index=False))
Output (Markdown was just the easiest for me to show all):
| timestamp | type | id | previousEventId | step | details |
|:---------------------------------|:-------------------|-----:|------------------:|:-----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2022-07-18 13:03:03.346000+00:00 | ExecutionFailed | 3 | 2 | executionFailedEventDetails | {'error': 'States.Runtime', 'cause': "An error occurred while executing the state 'Workflow Choice state' (entered at the event id #2). Invalid path '$.contributor_id': The choice state's condition path references an invalid value."} |
| 2022-07-18 13:03:03.306000+00:00 | ChoiceStateEntered | 2 | 0 | stateEnteredEventDetails | {'name': 'Workflow Choice state', 'input': '{\n "Comment": "Insert your JSON here"\n}', 'inputDetails': {'truncated': False}} |
| 2022-07-18 13:03:03.252000+00:00 | ExecutionStarted | 1 | 0 | executionStartedEventDetails | {'input': '{\n "Comment": "Insert your JSON here"\n}', 'inputDetails': {'truncated': False}, 'roleArn': 'arn:aws:iam::asdfg:role/step-all'} |

How to get each comments of different document as separate in mongoDB?

I am creating a social web using Node js and mongoDB. I have three tables as events, projects and reactions(like, comment).
Every event has many projects and every project has many reactions.
events
|----|---------|-------------|
| id | title | description |
|----|---------|-------------|
| e1 | event 1 | |
| e2 | event 2 | descc |
| e3 | event 3 | |
|----|---------|-------------|
projects
|----|-----------|-------------|----------|
| id | project | description | event_id |
|----|-----------|-------------|----------|
| p1 | project 1 | sample | e1 |
| p2 | project 2 | sample | e1 |
| p3 | project 3 | sample | e1 |
| p4 | project 1 | sample | e2 |
|----|-----------|-------------|----------|
reactions
|----|----------|------------|----------|--------|-------|
| id | user_id | project_id | event_id | type | value |
|----|----------|------------|----------|--------|-------|
| 1 | user1 | p1 | e1 | comment| nice |
| 2 | user1 | p1 | e1 | comment| wow |
| 3 | user2 | p1 | e1 | like | 1 |
| 4 | user2 | p1 | e2 | comment| good |
| 5 | user1 | p2 | e1 | comment| nice |
| 6 | user3 | p1 | e1 | comment| nice |
| 7 | user4 | p1 | e1 | like | 1 |
| 8 | user4 | p1 | e1 | comment| bad |
| 9 | user5 | p2 | e1 | comment| enjoy |
| 10 | user3 | p2 | e1 | comment| happy |
|----|----------|------------|----------|--------|-------|
my code
try
{
const id = req.params.id;
var ObjectId = require('mongodb').ObjectID;
if(!ObjectId.isValid(id))
{
return res.status(503).json("Invalid ID");
}
var aggregate = Event.aggregate();
aggregate.match({"_id":ObjectId(id)})
.lookup({ from: "pojects", localField: "_id", foreignField: "event_id",as: "project_doc"})
.lookup({from: "user_reactins", localField: "_id", foreignField: "event_id", as: "reactions"})
.lookup({from: "user_reactins", localField: "project_doc._id", foreignField: "project_id", as: "pro_reactions"})
.addFields({"project_doc.totalComments": "$pro_reactions"})
.project({_id:"$_id",title:"$title", project_doc:"$project_doc",
total_like_event: {
"$size": {
$filter: {
input: "$reactions",
as: "s",
cond: {
$eq: ["$$s.key", "like"]
},
}
}
},
total_comment_event: {
"$size": {
$filter: {
input: "$reactions",
as: "s",
cond: {
$eq: ["$$s.key", "comment"]
},
}
}
}
})
var options = {}
Event.aggregatePaginate(aggregate, options, function(err, results, pageCount, count)
{
return res.status(200).json({data:results[0],pro_status:statusResult,eve_status:eveResult,eve_cat:categoryResult});
})
}
catch (err)
{
return res.status(500).json(err);
}
this is my end point http://url/api/events/e1
If I used this end point, I want all comments for the given event(e1) by categorized by project as follows
{
"data": {
"_id": "e1",
"title": "event 1",
"project_doc": [
{
"_id": "p1",
"title": "project 1",
"description": "sample",
"totalComments": [
{
"_id": "1",
"user_id": "user1",
"programme_id": "p1",
"key": "comment",
"value": "nice",
},
{
"_id": "2",
"user_id": "user1",
"programme_id": "p1",
"key": "comment",
"value": "wow",
},
{
"_id": "6",
"user_id": "user3",
"programme_id": "p1",
"key": "comment",
"value": "nice",
},
{
"_id": "1",
"user_id": "user1",
"programme_id": "p1",
"key": "comment",
"value": "nice",
}
]
},
],
"total_view_event": 0,
"total_like_event": 1,
"total_comment_event": 4
}
}

How to retrieve multiple record-sets using 'npm mssql' in nodejs?

I have store-procedure which returns multiple results but when I retrieve it from npm mssql it returns the only first result.
in my T-SQL script:
CREATE PROCEDURE usp_myStoreProcedure #param1 varchar(3),#param2 varchar(3)
AS
BEGIN
select * from firstTable where name=#param1;
select * from secondTable where name=#param2;
END
when run this :
result1:
| Name | Subject | Mark|
|----------------------|
| Alice| Maths | 96 |
result2:
| Name | Subject | Mark|
|----------------------|
| Bob | Science | 93 |
in my nodejs using npm mssql package
let conn = await mssql.connect(config);
let output= await conn
.request()
.input("param1", mssql.VarChar(10), "Alcie")
.input("param2", mssql.VarChar(10), "Bob")
.execute("usp_myStoreProcedure");
mssql.close();
console.log(output);
current result:
{
"recordsets":
[
[
{
"Name": "Alice",
"Subject":"Maths"
"Mark": 96
}
],
[]
],
"recordset":
[
{
"Name": "Alice",
"Subject":"Maths"
"Mark": 96
}
],
"output": {},
"rowsAffected": [1,0],
"returnValue": 0
}
below result2 missing in the output:
| Name | Subject | Mark|
|----------------------|
| Bob | Science | 93 |

"query" count posts for each category expressjs mongoose

i am new at MERN stack
i tried to write this query but i can't and search a lot on google without any solution
i have to table one for posts and the second for categories
table 1 posts
--------------------------
|id | title | category |
--------------------------
| 1 | title1 | 1 |
| 2 | title2 | 2 |
| 3 | title3 | 1 |
| 4 | title4 | 1 |
=================
table 2 categories
---------------
|id | name |
---------------
| 1 | cat1 |
| 2 | cat2 |
====================
the results i want is this
---------------------------------
|id | name | number of posts |
---------------------------------
| 1 | cat1 | 3 |
| 2 | cat2 | 1 |
if it's can a help i write this query in mysql
SELECT categories.*,COUNT(posts.id) AS np FROM `categories` JOIN materials ON (categories.id = posts.category) GROUP BY categories.id
and thank you
You can try below aggregation in mongodb 3.6 and above
db.collection.aggregate([
{ "$lookup": {
"from": "posts",
"let": { "id", "$id" },
"pipeline": [
{ "$match": { "$expr": { "$eq": ["$category", "$$id"] }}},
{ "$count": "count" }
],
"as": "count"
}},
{ "$project": {
"name": 1,
"numberOfPosts": { "$arrayElemAt": ["$count.count", 0] }
}}
])

Elasticsearch doesn't give output I expect

My search query is:
query: {
match: {
name: "le sul"
}
},
I expect to see the output as:
.-------------------------------------------------.
| ID | Name | Score |
|-------|-----------------------------|-----------|
| 9 | le sultan | ... |
| 467 | le sultan | ... |
| 23742 | LE DUONG | 1.1602057 |
| 11767 | LE VICTORIA | 0.9554229 |
| 11758 | LE CANONNIER | 0.9554229 |
| 23762 | PHA LE XANH | 0.9281646 |
| 15795 | LE SURCOUF HOTEL & SPA | 0.9281646 |
| 33066 | LE CORAL HIDEAWAY BEYOND | 0.8695703 |
| 11761 | LE MERIDIEN MAURITIUS | 0.8682439 |
| 11871 | LE RELAX HOTEL & RESTAURANT | 0.8682439 |
'-------------------------------------------------'
But what I see is:
.-------------------------------------------------.
| ID | Name | Score |
|-------|-----------------------------|-----------|
| 23742 | LE DUONG | 1.1602057 |
| 9 | le sultan | 1.0869629 | <----
| 11767 | LE VICTORIA | 0.9554229 |
| 11758 | LE CANONNIER | 0.9554229 |
| 467 | le sultan | 0.9554229 | <----
| 23762 | PHA LE XANH | 0.9281646 |
| 15795 | LE SURCOUF HOTEL & SPA | 0.9281646 |
| 33066 | LE CORAL HIDEAWAY BEYOND | 0.8695703 |
| 11761 | LE MERIDIEN MAURITIUS | 0.8682439 |
| 11871 | LE RELAX HOTEL & RESTAURANT | 0.8682439 |
'-------------------------------------------------'
As you can see, "le sultan" not the first element of the result set.
Where am I going wrong?
Your query result is not match because elasticsearch search via _score.
In your case you want to search in an analyzed search and get result in not analyzed.
So you should put your mapping like given below
Put your_index_name
{
"mappings": {
"your_type_name": {
"properties": {
"name": {
"type": "string",
"analyzer": "english",
"fields": {
"your_temporary_sort_filed_name": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}
And then
GET /your_index_name/your_type_name/_search
{
"sort": [
{
"name.your_temporary_sort_filed_name":{
"order": "desc"
}
}
],
"query": {
"match": {
"name": "le sul"
}
}
}
If you want to get le sultan as use following query:
{
"query": {
"query_string": {
"default_field": "name",
"query": "le sul*"
}
}
}

Resources