Elasticsearch and express.js - Find and update nested property in document - node.js

Successfully I found how to implement adding an element to nested property list, but currently I'm struggling with Elasticsearch mechanisms of updating document properties. I'm trying to update using Axios as my main library of wrapping of HTTP client. An example of saved documents in Elasticsearch:
{
"took": 4,
"timed_out": false,
"_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 },
"hits": {
"total": { "value": 2, "relation": "eq" },
"max_score": 1.0,
"hits": [
{
"_index": "my-index-01",
"_type": "_doc",
"_id": "-8_MhnsBRfQBiuhMDxjT",
"_score": 1.0,
"_source": {
"code": "ABC",
"description": "Test",
"items": [{ "key": "XYZ", "value": "123" }]
}
},
{ ... },
{ ... },
{ ... }
]
}
}
My main goal is to filter from _source element by code field and get from items table element by key and then change the value.
I tried with this code but the value is not changing and I'm getting en 400 response from my docker Elasticsearch container:
const input = {
key: "XYZ",
value: "1234"
}
this.axios.post(
`/my-index-01/_update_by_query`,
{
script: {
lang: 'painless',
source: `if (ctx._source.items['items.key'].value.key == params.key) {
doc['items.key'].value.value == params.value;
} `,
params: input
},
query: {
match: {
code: "ABC",
},
}
},
{
headers: {
"Content-Type": "application/json",
},
}
);

I found a solution to this problem. I want to share it here as answer, maybe someday someone will struggle with the same issue:
this.axios.post(
`/my-index-01/_update_by_query`,
{
script: {
lang: 'painless',
source: `for (def item : ctx._source.items) {
if (item.key == params.key) {
item.value = params.value;
}
} `,
params: input
},
query: {
match: {
code: "ABC",
},
}
},
{
headers: {
"Content-Type": "application/json",
},
}
);

Related

Filter nested result inside a nested object with elasticsearch

I'm trying to filter a nested object and sort by the result, however, I tried some things without success, I'll leave my initial attempt and it works partially, it just filters according to what I have in my search variable, but all the results come of this nested object as it is inside the 'root' which is another nested object
Elastic version: 7.13.0 with NodeJS
using #elastic/elasticsearch official package from npm
let params: RequestParams.Search = {
index: index,
body: {
size: 30,
query: {
bool: {
must: [
{
nested: {
path: "profile",
query: {
bool: {
must: [
{
match: {
"profile.id": profileId,
},
},
],
},
},
},
},
],
filter: [
{
nested: {
path: "profile.following",
ignore_unmapped: true,
query: {
query_string: {
fields: [
"profile.following.name",
"profile.following.username",
],
query: searchWord + "*",
},
},
},
},
],
},
},
},
};
I need it to be this specific 'profile.id' that is passed by parameter in the function, so the result is only 1 profile with N people that it follows
the document is mapped as follows, I left only the fields relevant to the question:
{
"mappings": {
"_doc": {
"properties": {
"id": {
"type": "integer"
},
"phone": {
"type": "text"
},
"profile": {
"type": "nested",
"properties": {
"id": {
"type": "integer"
},
"username": {
"type": "text"
},
"following": {
"type": "nested",
"properties": {
"id": {
"type": "integer"
},
"isAwaitingApproval": {
"type": "boolean"
},
"name": {
"type": "text"
},
"profilePicURL": {
"type": "text"
},
"username": {
"type": "text"
}
}
}
}
}
}
}
}
}
an example of a current result is:
with the following parameters (profileId:141, searchWord: "para" )
{
"res": [
{
"profilePicURL": "localimage",
"name": "donor donor",
"id": 140,
"username": "victorTesteElastic2",
"isAwaitingApproval": false
},
{
"profilePicURL": "localimage",
"name": "para ser seguido",
"id": 142,
"username": "victorprivate",
"isAwaitingApproval": true
}
]
}
the desired result is:
{
"res": [
{
"profilePicURL": "localimage",
"name": "para ser seguido",
"id": 142,
"username": "victorprivate",
"isAwaitingApproval": true
}
]
}
with some more research I got what I needed, I'll leave the answer here in case anyone needs it too
let params: RequestParams.Search = {
index: index,
body: {
size: 30,
query: {
bool: {
must: [
{
nested: {
path: "profile",
query: {
bool: {
must: [
{
match: {
"profile.id": profileId,
},
},
],
},
},
},
},
{
nested: {
path: "profile",
inner_hits: {
name: "profile",
},
query: {
nested: {
path: "profile.following",
inner_hits: {
name: "following",
},
ignore_unmapped: true,
query: {
query_string: {
fields: [
"profile.following.name",
"profile.following.username",
],
query: searchWord + "*",
},
},
},
},
},
},
],
},
},
},
};
I basically put in must what was in the filter, mapped the nested object from above, in this case the profile, and put the tag inner_hits for profile and inner_hits for followings, that's the only way it worked
the answer I need was returned here:
body.hits.hits[0].inner_hits.profile.hits.hits[0].inner_hits.following.hits.hits
below is an example of the answer:
{
"res": [
{
"_index": "donor",
"_type": "_doc",
"_id": "P3VWNnsB4coAEhD-F3fF",
"_nested": {
"field": "profile",
"offset": 0,
"_nested": {
"field": "following",
"offset": 0
}
},
"_score": 1,
"_source": {
"profilePicURL": "localimage",
"name": "donor donor",
"id": 140,
"username": "victorTesteElastic2",
"isAwaitingApproval": false
}
},
{
"_index": "donor",
"_type": "_doc",
"_id": "P3VWNnsB4coAEhD-F3fF",
"_nested": {
"field": "profile",
"offset": 0,
"_nested": {
"field": "following",
"offset": 1
}
},
"_score": 1,
"_source": {
"profilePicURL": "localimage",
"name": "para ser seguido",
"id": 142,
"username": "victorprivate",
"isAwaitingApproval": true
}
}
]
}
the filtered data I really need that have been matched in must is in this array, where I need to iterate and look at _source which is the data that is indexed

GraphQL Query - cannot see nested elements in JSON respone

if anyone can hazard a guess or where to look it would be greatly appreciated.
I can get nested data when I run using graphgl API, however, from my node program it only shows top-level items - does not display the nested elements for the customer and lineitem object.
I am using Koa middle where, with promise response:
router.get('/orders/', async (ctx) => {
const auth = prepareAuth(ctx);
await getOrders(auth).then(response => ctx.body = response.data.data.orders);
console.log(ctx.body.edges)
However from the console it has (customer null and 'object':
[
{
node: {
createdAt: '2020-02-24T12:53:20Z',
customer: null,
name: '#1001',
lineItems: [Object]
}
},
{
node: {
createdAt: '2020-02-24T12:53:50Z',
customer: null,
name: '#1002',
lineItems: [Object]
}
},
{
node: {
createdAt: '2020-03-10T21:11:04Z',
customer: null,
name: '#1003',
lineItems: [Object]
}
}
]
when i use the GraphQL API directly the query works fine and I get full response:
{
"data": {
"orders": {
"edges": [
{
"node": {
"createdAt": "2020-02-24T12:53:20Z",
"customer": {
"displayName": "franko girl"
},
"name": "#1001",
"lineItems": {
"edges": [
{
"node": {
"name": "dance mat red",
"quantity": 4
}
}
]
}
}
},
{
"node": {
"createdAt": "2020-02-24T12:53:50Z",
"customer": {
"displayName": "franko man"
},
"name": "#1002",
"lineItems": {
"edges": [
{
"node": {
"name": "dance mat black",
"quantity": 2
}
}
]
}
}
},
{
"node": {
"createdAt": "2020-03-10T21:11:04Z",
"customer": {
"displayName": "franko man"
},
"name": "#1003",
"lineItems": {
"edges": [
{
"node": {
"name": "dance mat black",
"quantity": 1
}
},
{
"node": {
"name": "dance mat red",
"quantity": 1
}
}
]
}
}
}
]
}
},
Okay, so finally figured this out, for anyone else who stumbles accross this problem, you need to the convert the json object to a string using built in javascript function: JSON.stringify()
from W3schools.com
var obj = { name: "John", age: 30, city: "New York" };
var myJSON = JSON.stringify(obj);

ElasticSearch NodeJS - Aggregation term return more than one source property

I need to get a unique list of things, with some of the properties that are attached. As of now this just returns a unique list of names, yet if I wanted to include the id of the aggregates doc's, what do I do?
I'm using the elasticsearch npm module with the .search() method
Any help would be greatly appreciated.
params.body.aggs = {
uniqueCoolThings: {
terms: {
field: 'cool_thing.name.keyword'
}
}
}
This will return a list of { key, doc_count } I want { key, id, doc_count }
That works! Thank you Technocrat Sid!
So what if my docs looks like this
{ cool_things: [{ name, id }, { name, id }] }
How would I find the id of the one I'm currently in the hit. For example this is the working query.
params.body.aggs = {
uniqueCoolThings: {
terms: {
field: 'cool_things.name.keyword'
},
aggs: {
value: {
top_hits: {
size: 1,
_source: {
includes: ['cool_things.id']
}
}
}
}
}
}
}
Yet this will return
...hits._source: {
uniqueCoolThings: [
{
"id": 500
},
{
"id": 501
}
]
} ...
I'm wondering how to do a where condition so that it will only return the ID that matches the unique cool_things.name.keyword it is currently on.
At most you can use top hits aggregation as a sub aggregation which keeps the track of the aggregated documents.
Example:
A similar terms aggregation query:
"aggs": {
"uniqueCoolThings": {
"terms": {
"field": "cool_thing.name.keyword"
}
}
}
will return the following results:
"aggregations": {
"uniqueCoolThings": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "XYZ",
"doc_count": 2
},
{
"key": "ABC",
"doc_count": 1
}
]
}
}
And if you add top hits aggregation as a sub aggregation to the above query:
"aggs": {
"uniqueCoolThings": {
"terms": {
"field": "cool_thing.name.keyword"
},
"aggs": {
"value": {
"top_hits": {
"_source": "false"
}
}
}
}
}
You'll get the following result:
"aggregations": {
"uniqueCoolThings": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "XYZ",
"doc_count": 2,
"value": {
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "product",
"_type": "_doc",
"_id": "BqGhPGgBOkyOnpPCsRPX",
"_score": 1,
"_source": {}
},
{
"_index": "product",
"_type": "_doc",
"_id": "BaGhPGgBOkyOnpPCfxOx",
"_score": 1,
"_source": {}
}
]
}
}
}
....
.... excluding output for brevity !!
Notice in the above result you have the aggregated documents _id(value.hits.hits._id) within your terms bucket.
Not sure of the syntax but something like this should work for you:
params.body.aggs = {
uniqueCoolThings: {
terms: {
field: 'cool_thing.name.keyword'
},
aggs: {
value: {
top_hits: {
_source: 'false'
}
}
}
}
}

How to calculate total for each token in Elasticsearch

I have a request into Elastic
{
"query":{
"bool":{
"must":[
{
"query_string":{
"query":"something1 OR something2 OR something3",
"default_operator":"OR"
}
}
],
"filter":{
"range":{
"time":{
"gte":date
}
}
}
}
}
}
I wanna calculate count for each token in all documents using elastic search in one request, for example:
something1: 26 documents
something2: 12 documents
something3: 1 documents
Assuming that the tokens are not akin to enumerations (i.e. constrained set of specific values, like state names, which would make a terms aggregation your best bet with the right mapping), I think the closest thing to what you want would be to use filters aggregation:
POST your-index/_search
{
"query":{
"bool":{
"must":[
{
"query_string":{
"query":"something1 OR something2 OR something3",
"default_operator":"OR"
}
}
],
"filter":{
"range":{
"time":{
"gte":date
}
}
}
}
},
"aggs": {
"token_doc_counts": {
"filters" : {
"filters" : {
"something1" : {
"bool": {
"must": { "query_string" : { "query" : "something1" } },
"filter": { "range": { "time": { "gte": date } } }
}
},
"something2" : {
"bool": {
"must": { "query_string" : { "query" : "something2" } },
"filter": { "range": { "time": { "gte": date } } }
}
},
"something3" : {
"bool": {
"must": { "query_string" : { "query" : "something3" } },
"filter": { "range": { "time": { "gte": date } } }
}
}
}
}
}
}
}
The response would look something like:
{
"took": 9,
"timed_out": false,
"_shards": ...,
"hits": ...,
"aggregations": {
"token_doc_counts": {
"buckets": {
"something1": {
"doc_count": 1
},
"something2": {
"doc_count": 2
},
"something3": {
"doc_count": 3
}
}
}
}
}
You can split your query into filters aggregation of three filters. For reference look here: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filters-aggregation.html
What you would need to do, is to create a Copy_To field and have the mapping as shown below.
Depending on the fields that your query_string queries, you need to include some or all of the fields with copy_to field.
By default query_string searches all the fields, so you may need to specify copy_to for all the fields as shown in below mapping, where for sake of simplicity, I've created only three fields, title, field_2 and a third field content which would act as copied to field.
Mapping
PUT <your_index_name>
{
"mappings": {
"mydocs": {
"properties": {
"title": {
"type": "text",
"copy_to": "content"
},
"field_2": {
"type": "text",
"copy_to": "content"
},
"content": {
"type": "text",
"fielddata": true
}
}
}
}
}
Sample Documents
POST <your_index_name>/mydocs/1
{
"title": "something1",
"field_2": "something2"
}
POST <your_index_name>/mydocs/2
{
"title": "something2",
"field_2": "something3"
}
Query:
You'd get the required document counts for the each and every token using the below aggregation query and I've made use of Terms Aggregation:
POST <your_index_name>/_search
{
"size": 0,
"query": {
"query_string": {
"query": "something1 OR something2 OR something3"
}
},
"aggs": {
"myaggs": {
"terms": {
"field": "content",
"include" : ["something1","something2","something3"]
}
}
}
}
Query Response:
{
"took": 7,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0,
"hits": []
},
"aggregations": {
"myaggs": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "something2",
"doc_count": 2
},
{
"key": "something1",
"doc_count": 1
},
{
"key": "something3",
"doc_count": 1
}
]
}
}
}
Let me know if it helps!

NodeJs-ElasticSearch Bulk API error handling

I can't find any documentation on what happens if Elastic Bulk API fails on one or more of the actions. For example, for the following request, let's say there is already a document with id "3", so "create" should fail- does this fail all of the other actions?
{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_type" : "type1", "_id" : "2" } }
{ "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_type" : "type1", "_index" : "index1"} }
{ "doc" : {"field2" : "value2"} }
I'm using nodejs elastic module.
No failures in one action does not affect the others .
From the documentation of elasticsearch bulk api :
The response to a bulk action is a large JSON structure with the
individual results of each action that was performed. The failure of a
single action does not affect the remaining actions.
In the response from elasticsearch client there is status in response corresponding to each action to determine if it was a failure or not
Example:
client.bulk({
body: [
// action description
{ index: { _index: 'test', _type: 'test', _id: 1 } },
// the document to index
{ title: 'foo' },
// action description
{ update: { _index: 'test', _type: 'test', _id: 332 } },
// the document to update
{ doc: { title: 'foo' } },
// action description
{ delete: { _index: 'test', _type: 'test', _id: 33 } },
// no document needed for this delete
]
}, function (err, resp) {
if(resp.errors) {
console.log(JSON.stringify(resp, null, '\t'));
}
});
Response:
{
"took": 13,
"errors": true,
"items": [
{
"index": {
"_index": "test",
"_type": "test",
"_id": "1",
"_version": 20,
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 200
}
},
{
"update": {
"_index": "test",
"_type": "test",
"_id": "332",
"status": 404,
"error": {
"type": "document_missing_exception",
"reason": "[test][332]: document missing",
"shard": "-1",
"index": "test"
}
}
},
{
"delete": {
"_index": "test",
"_type": "test",
"_id": "33",
"_version": 2,
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 404,
"found": false
}
}
]
}

Resources