Can anyone let me know how can I transform this input json I have tried with below transformation but didn't worked.
I have tried to used filter function and groupBy function, but for Multiple output it is failing the testCase.
tried with this method
`
%dw 2.0
output application/json
---
payload.Bd map (val,index) ->{
"d23": val,
"lt":(payload.output2 filter(payload.Bd contains val) map(Value) ->
{
Val34: Value.PId
}
)
}
input -
[
{
"Val34": "968",
"d23": "Y1"
},
{
"Val34": "958",
"d23": "Y2"
},
{
"Val34": "951",
"d23": "Y2"
}
]
expected output -
[
{
"d23": "Y1",
"lt": [
{
"Val34": "968"
}
]
},
{
"d23": "Y2",
"lt": [
{
"Val34": "958"
},
{
"Val34": "951"
}
]
}
]
`
Group by d23 values and map all the rest:
%dw 2.0
output application/json
---
payload groupBy ((item, index) -> item.d23) pluck {
d23: $[0].d23,
lt: ($.Val34 map (l,indOfl) -> {Val34: l})
}
You can try first groupingBy 2nd key value pair.. I assumed "d23": "Y1", "d23": "Y3" will be in position index[1] .
Further you can map on grouped elements and access index[0]
%dw 2.0
output application/json
---
(payload groupBy ((item, index) -> item[1])) pluck $ map{
($[0][&1]),
"lt": ($ map(it,in)->(it[&0]))
}
Sample Input
[
{
"Val34": "968",
"d23": "Y1"
},
{
"Val34": "998",
"d23": "Y3"
},
{
"Val34": "988",
"d23": "Y1"
},
{
"Val34": "958",
"d23": "Y2"
},
{
"Val34": "951",
"d23": "Y2"
}
]
Output
[
{
"d23": "Y1",
"lt": [
{
"Val34": "968"
},
{
"Val34": "988"
}
]
},
{
"d23": "Y3",
"lt": [
{
"Val34": "998"
}
]
},
{
"d23": "Y2",
"lt": [
{
"Val34": "958"
},
{
"Val34": "951"
}
]
}
]
Related
My document is like shown below
{
WORKFLOWS: [
{
ID: 1,
STARTEDBY: "BOBU",
ACTIVE: true,
HISTORY: [
{
STATS: 21,
DATE_TIME: "2020-03-19T00:16:39.612Z",
USER: "VICTORIA",
},
{
STATS: 20,
DATE_TIME: "2020-03-19T00:16:40.190Z",
USER: "VICTORIA",
REMINDER: [
"2020-03-19T00:13:39.709Z",
"2020-03-19T00:16:39.612Z",
"2020-03-19T00:16:40.190Z",
],
},
],
},
],
}
What I am trying to achieve is to find the latest history object and create REMINDER property if not existing and push the current date-time as a string
Since I am going to call this from node, i can send the current time from node as a parameter
Regards
Bobu
Well, we need to do it in two steps.
Explanation
We need to find the recent DATE_TIME with $addFields.
var MAX_DT = "1970-01-01T00:00:00:00.000Z";
for(var dt in WORKFLOWS[*].HISTORY[*].DATE_TIME){
MAX_DT = MAX_DT > dt ? MAX_DT : dt
}
In the next step, we iterate over WORKFLOWS and HISTORY arrays with $map operator and add current date with $$NOW (Standalone) or $$CLUSTER_TIME (Cluster).
$dateToString allows specify timezone to customize HH:MM result.
If you want to persist the result, just add as the last step $out operator:
{$out:"collection"}
Note: It will override entire collection with aggregation result.
db.collection.aggregate([
{
$addFields: {
"MAX_DT": {
$reduce: {
input: {
$reduce: {
input: "$WORKFLOWS.HISTORY.DATE_TIME",
initialValue: [],
in: {
$concatArrays: [ "$$value", "$$this" ]
}
}
},
initialValue: "1970-01-01T00:00:00:00.000Z",
in: {
$cond: [
{
$gt: [ "$$value", "$$this" ]
},
"$$value",
"$$this"
]
}
}
}
}
},
{
$addFields: {
"WORKFLOWS": {
$map: {
input: "$WORKFLOWS",
as: "wf",
in: {
$mergeObjects: [
"$$wf",
{
HISTORY: {
$map: {
input: "$$wf.HISTORY",
as: "his",
in: {
$cond: [
{
$eq: [ "$$his.DATE_TIME", "$MAX_DT" ]
},
{
$mergeObjects: [
"$$his",
{
REMINDER: {
$concatArrays: [
{
$ifNull: [ "$$his.REMINDER", [] ]
},
[
{
$dateToString: {
date: "$$NOW",
timezone: "+02:00"
}
}
]
]
}
}
]
},
"$$his"
]
}
}
}
}
]
}
}
}
}
}
])
MongoPlayground
The json format is like that:
[
[
{},
{
"Country": "Japan",
"cityName": "tokyo",
"onto": [
{
"level1": "one",
"articles": [
null,
{
"id": "114506604",
"name": "bunya3",
"abc": [
{
"filename": "attachmentsfilename3",
"size": 3
}
],
"image": {}
}
]
}
]
}
],
[
{}
]
]
We can see few null, {} and [{}]. How can we remove it ? By the way I am using node js. I have tried by nnjson
nnjson.removeNull(obj_summary);
But not works object without key.
If we assume that your data is always going to be an array, we can map over it and remove empty arrays and objects from the first level:
const data = [
[
{},
{
Country: 'Japan',
cityName: 'tokyo',
onto: [
{
level1: 'one',
articles: [
null,
{
id: '114506604',
name: 'bunya3',
abc: [
{
filename: 'attachmentsfilename3',
size: 3
}
],
image: {}
}
]
}
]
}
],
[{}]
]
function clean(input) {
return input
.map(item => {
// remove empty arrays
if (Array.isArray(item) && item.length === 0) {
return null
}
// Remove empty objects
if (item instanceof Object && Object.keys(item).length === 0) {
return null
}
return item
})
.filter(item => item)
}
console.log(clean(data))
I found the solution.
To remove null I used:
let retSummary = JSON.parse(stringifySummary, (k, v) => Array.isArray(v) ?
v.filter(e => e !== null) : v);
To remove {} I used
var newArray = parObj.filter(value => Object.keys(value).length !== 0);
Case
I have store the location coordinates in MongoDB collection with 2dsphere index. I want to get the records which included into the specific bounds on map. I have one coordinate point which is fixed with latitude and longitude. I have used the method $geoWithin with $box in MongoDB aggregation. I am able to get the data properly.
Problem
I need to add distance field in each record from the one fixed latitude and longitude I have.
{
"$match": {
"location": {
"$geoWithin": {
"$box": [
[
0.024719919885622943,
51.54643953472475
],
[
-0.1589577534542408,
51.47239969138267
]
]
}
}
}
}
Now I have coordinates array-like [-0.0649729793707321, 51.50160291888072]
I need distance for each record from the above point. How can I find this distance field with MongoDB aggregation?
sample document for the database record
{
"_id" : ObjectId("5e342ac2e0dcfa65273751ea"),
"location" : {
"type" : "Point",
"coordinates" : [
-0.123507,
51.5083228
]
},
"status" : 1,
"price" : {
"value" : 27,
"type" : "day",
"base" : 3.38
},
"name" : "Initial Space with Wormup",
"venuePolicy" : "Venue Policies",
"updatedAt" : ISODate("2020-02-20T12:28:16.024Z"),
"createdAt" : ISODate("2020-01-31T13:25:22.600Z"),
"dimension" : {
"width" : 100
},
"description" : "<p>Hi, </p><p>please find attached document for testing purpose",
}
This one should work:
db.collection.aggregate([
{
"$match": {
"location": {
"$geoWithin": {
"$box": [
[0.024719919885622943, 51.54643953472475],
[-0.1589577534542408, 51.47239969138267]
]
}
}
}
},
{ $set: { lon: -0.0649729793707321 } },
{ $set: { lat: 51.50160291888072 } },
{
$set: {
distance: {
$let: {
vars: {
dlon: { $degreesToRadians: { $subtract: [{ $arrayElemAt: ["$location.coordinates", 0] }, "$lon"] } },
dlat: { $degreesToRadians: { $subtract: [{ $arrayElemAt: ["$location.coordinates", 1] }, "$lat"] } },
lat1: { $degreesToRadians: { $arrayElemAt: ["$location.coordinates", 1] } },
lat2: { $degreesToRadians: "$lat" }
},
in: {
// Haversine formula: sin²(dLat / 2) + sin²(dLon / 2) * cos(lat1) * cos(lat2);
$add: [
{ $pow: [{ $sin: { $divide: ["$$dlat", 2] } }, 2] },
{ $multiply: [{ $pow: [{ $sin: { $divide: ["$$dlon", 2] } }, 2] }, { $cos: "$$lat1" }, { $cos: "$$lat2" }] }
]
}
}
}
}
},
{
$set: {
distance: {
// Distance in Meters given by "6372.8 * 1000"
$multiply: [6372.8, 1000, 2, { $asin: { $sqrt: "$distance" } }]
}
}
},
])
Just a note, $box is only used for Legacy Coordinate Pairs. For proper query (i.e. proper use of 2dsphere index) you should use $geometry:
db.collection.createIndex({ location: "2dsphere" })
db.collection.find(
{
"location": {
"$geoWithin": {
"$geometry": {
type: "Polygon",
coordinates: [[
[-0.1589577534542408, 51.47239969138267],
[-0.1589577534542408, 51.54643953472475],
[0.024719919885622943, 51.54643953472475],
[0.024719919885622943, 51.47239969138267],
[-0.1589577534542408, 51.47239969138267]
]]
}
}
}
}
).explain().queryPlanner.winningPlan.inputStage.indexName
--> location_2dsphere // which means index is used
db.collection.find(
{
"location": {
"$geoWithin": {
"$box": [
[0.024719919885622943, 51.54643953472475],
[-0.1589577534542408, 51.47239969138267]
]
}
}
}
).explain().queryPlanner.winningPlan.stage
--> COLLSCAN // which means full collection scan
Or proper use of Legacy Coordinates:
db.collection.createIndex({ "location.coordinates": "2d" })
db.collection.find(
{
"location.coordinates": { // <- note this difference
"$geoWithin": {
"$box": [
[0.024719919885622943, 51.54643953472475],
[-0.1589577534542408, 51.47239969138267]
]
}
}
}
).explain().queryPlanner.winningPlan.inputStage.indexName
--> location.coordinates_2d
I am searching for objects in Elasticsearch 6.3.
I have 3 objects:
{
identifier: 01,
lineCodes: [],
},
{
identifier: 02,
lineCodes: [
{
link: "brussels",
name: "br"
},
{
link: "antwerp",
name: "an"
},
],
},
{
identifier: 03,
lineCodes: [
{
link: "ghent",
name: "gh"
},
],
}
My lineCodes schema is:
{
"lineCodes": {
"properties": {
"__typename": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"link": {
"type": "keyword"
},
"name": {
"type": "keyword"
}
}
}
}
Can you help me with Query DSL's that find all objects with:
Brussels in lineCodes.link or empty lineCodes
All empty lineCodes
Ghent in lineCodes.link or empty lineCodes
query 1 should result in object 01 and 02
query 2 should result in object 01
query 3 should result in object 01 and 03
I tried this query, but it gives me not the empty LineCodes objects
let query = {
size: 1000,
index: 'catalog',
body: {
query: {
bool: {
must: [
{
bool: {
should: [
{ term: { 'lineCodes.link': lineCode } },
{ terms: { 'lineCodes': [] } },
],
},
}
],
}
}
}
};
You can use the exists query ( doc here ) and negate it with a must_not clause in a bool query
Could you try :
{
size: 1000,
index: 'catalog',
body: {
query: {
bool: {
must: [
{
bool: {
should: [
{ term: { 'lineCodes.link': lineCode } },
{
bool: {
must_not: {
exists: {
field: 'lineCodes'
}
}
}
}
],
},
}
],
}
}
}
}
NB: As you use a term query for linecode.link watch out for case and accents!
Hope it helps !
I use ELasticsearch and Meteor.js with npm/elasticsearch plugin to use ES with Meteor.
Here is my query:
search = {
"body": {
"aggs": {
"traces": {
"filter": {
"and": [
{
"range": {
"date": {
"gte": "2015-01-31T23:00:00.000Z",
"lte": "2015-10-02T21:59:59.000Z"
}
}
},
{
"geo_bounding_box": {
"loc.coordinates": {
"top_left": {
"lat": 51.767839887322154,
"lon": -9.404296875
},
"bottom_right": {
"lat": 40.96330795307353,
"lon": 14.326171874999998
}
}
}
},
{
"or": [
{
"and": [
{
"term": {
"geoip": false
}
},
{
"term": {
"trackerId": "RG-000000010-1"
}
}
]
}
]
}
]
},
"aggs": {
"trackerId": {
"terms": {
"field": "trackerId",
"size": 0
},
"aggs": {
"heatmap": {
"geohash_grid": {
"field": "loc.coordinates",
"precision": 6
}
}
}
}
}
}
}
}
}
I send it with esClient.search(search) and receive the excepted answer but ES throws this error:
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:735)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:560)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:532)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:294)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.query.QueryParsingException: [.marvel-2015.10.02]
**failed to find geo_point field [loc.coordinates]**
at org.elasticsearch.index.query.GeoBoundingBoxFilterParser.parse(GeoBoundingBoxFilterParser.java:173)
at org.elasticsearch.index.query.QueryParseContext.executeFilterParser(QueryParseContext.java:368)
at org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:349)
at org.elasticsearch.index.query.AndFilterParser.parse(AndFilterParser.java:65)
at org.elasticsearch.index.query.QueryParseContext.executeFilterParser(QueryParseContext.java:368)
at org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:349)
at org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:295)
at org.elasticsearch.search.aggregations.bucket.filter.FilterParser.parse(FilterParser.java:42)
at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:148)
at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:78)
at org.elasticsearch.search.aggregations.AggregationParseElement.parse(AggregationParseElement.java:60)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:719)
Apparently it does not find the geo_point field type.
Here is the revelant part of my mapping:
{
'trace': {
'properties': {
'loc': {
'type': 'object',
'properties': {
'type': {
'type': 'string'
},
'coordinates':{
'type': 'geo_point',
'geohash':true,
'geohash_prefix':true,
'lat_lon':true,
'fielddata' : {
'format' : 'compressed',
'precision' : '1cm'
}
}
}
}
...
So why the geo_point type is not found ?
You need to send your query to the index that contains the documents of type trace. If you send the query to the root endpoint / (default if not specified) then all indices will be queried. In your case, it fails because the .marvel-2015.10.02 index doesn't have any geo_point field named loc.coordinates.
So your call needs to look like this:
var search = {
index: "your_index", <--- add this
body: {...}
};
esClient.search(search);