I am trying to index data that look like the following :
var json = {
"kwg": {
"kwg0List": [
{
"lemma": "bilingue",
"categories": [
"terme"
],
"occList": [
{
"startTimeSec": 537.1,
"endTimeSec": 537.46,
"value": "bilingue"
},
{
"startTimeSec": 563.2,
"endTimeSec": 563.55,
"value": "bilingue"
}
]
}
]
}
}
Everything works fine. Now, let's say, for whatever reason, that the one of the startTimeSec fields is equal to 10. It's interpreted as a long and not as a double anymore.
I would get the following error : mapper_parsing_exception, telling me that I should have a double and not a long.
Now my question is : is there a way to "force" the long to be cast to a double when indexing, or is previously checking that the data is correctly formatted the only way of doing it?
Trace :
{
"took": 1112,
"errors": true,
"items": [
{
"create": {
"_index": "library",
"_type": "relevance",
"_id": "AViRhRJ-_Tb2laJ1W4JH",
"status": 400,
"error": {
"type": "mapper_parsing_exception",
"reason": "failed to parse",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "mapper [kwg.kwg0List.occList.endTimeSec] of different type, current_type [double], merged_type [long]"
}
}
}
}
]
}
Related
I am using logstash 5.6
In my document, I have a subfield "[emailHeaders][reingested-on]", and another field called [attributes], which contains several subfields [string], [double], each of which are arrays. :
{
"emailHeaders": {
"reingested-on": ["1613986076000"]
},
"attributes": {
"string": [
{
"name": "attributeString1",
"value": "attributeStringValue1"
},
{
"name": "attributeString2",
"value": "attributeStringValue2"
}
],
"double": [
{
"name": "attributeDouble1",
"value": 1.0
}
]
}
}
If the element [emailHeaders][reingested-on] is present in the document, I want to copy 1613986076000 (ie. the first element of [emailHeaders][reingested-on]) into [attributes][date] as follows:
{
"emailHeaders": {
"reingested-on": ["1613986076000"]
},
"attributes": {
"string": [
{
"name": "attributeString1",
"value": "attributeStringValue1"
},
{
"name": "attributeString2",
"value": "attributeStringValue2"
}
],
"double": [
{
"name": "attributeDouble1",
"value": 1.0
}
],
"date": [
{
"name": "Reingested on",
"value": 1613986076000
}
]
}
}
Note that if [attributes][date] already exists, and already contains an array of name/value pairs, I want my new object to be appended to the array.
Also, note that [attributes][date] is an array of objects which contain a date in their [value] attribute, as per the mapping of my ElasticSearch index:
...
"attributes": {
"properties": {
...
"date": {
"type": "nested",
"properties": {
"id": {"type": "keyword"},
"name": {"type": "keyword"},
"value": {"type": "date"}
}
},
...
}
},
...
I tried the following logstash configuration, with no success:
filter {
# See https://stackoverflow.com/questions/30309096/logstash-check-if-field-exists : this is supposed to allow to "test" if [#metadata][reingested-on] exists
mutate {
add_field => { "[#metadata][reingested-on]" => "None" }
copy => { "[emailHeaders][reingested-on][0]" => "[#metadata][reingested-on]" }
}
if [#metadata][reingested-on] != "None" {
# See https://stackoverflow.com/questions/36127961/append-array-of-json-logstash-elasticsearch: I create a temporary [error] field, and I try to append it to [attributes][date]
mutate {
add_field => { "[error][name]" => "Reingested on" }
add_field => { "[error][value]" => "[#metadata][reingested-on]" }
}
mutate {
merge => {"[attributes][date]" => "[error]"}
}
}
}
But what I get is:
{
"emailHeaders": {
"reingested-on": ["1613986076000"]
},
"error": {
"name": "Reingested on",
"value": "[#metadata][reingested-on]"
},
"attributes": {
"string": [
{
"name": "attributeString1",
"value": "attributeStringValue1"
},
{
"name": "attributeString2",
"value": "attributeStringValue2"
}
],
"double": [
{
"name": "attributeDouble1",
"value": 1.0
}
]
}
}
My temporary [error] object is created, but its value is wrong: it should be 1613986076000 instead of [#metadata][reingested-on]
Also, it is not appended to the array [attribute][date]. In this example, this array does not exist, so I want it to be created with my temporary object as first element, as per the expected result above.
This is a problem i have working in Excels Power Query.
I have this query saved in a variable named "content" which is passed to the call Web.Contents.
I can not run the query, i get "Token Comma expected" error. Can somebody tell what that is about?
`let
content = "{
"query": [
{
"code": "Region",
"selection": {
"filter": "vs:RegionKommun07",
"values": [
"1283"
]
}
},
{
"code": "Sysselsattning",
"selection": {
"filter": "item",
"values": [
"FÖRV"
]
}
},
{
"code": "Alder",
"selection": {
"filter": "item",
"values": [
"30-34"
]
}
},
{
"code": "Kon",
"selection": {
"filter": "item",
"values": [
"1"
]
}
},
{
"code": "Tid",
"selection": {
"filter": "item",
"values": [
"2015"
]
}
}
],
"response": {
"format": "px"
}
}",
Source = Json.Document(Web.Contents("http://api.scb.se/OV0104/v1/doris/sv/ssd/START/AM/AM0207/AM0207H/BefSyssAldKonK", [Content=Text.ToBinary(content)]))
in
Source`
If you want " inside a quoted string then you need to double them up like "" to escape them.
let
content = "{
""query"": [
{
""code"": ""Region"",
""selection"": {
""filter"": ""vs:RegionKommun07"",
""values"": [
""1283""
]
}
},
...
...
}"
See page 21 here: http://download.microsoft.com/download/8/1/A/81A62C9B-04D5-4B6D-B162-D28E4D848552/Power%20Query%20M%20Formula%20Language%20Specification%20(July%202019).pdf
To include quotes in a text value, the quote mark is repeated, as
follows: "The ""quoted"" text" // The "quoted" text
I am new to ElasticSearch and looking for bool query to pass it to get the data from elasticsearch in spark scala code.
Here is my query:
Get all records for the eventName = "XXXXXX" and date between("1438367180542","1738367180542")
Could you please help me to write the elasticsearch query. Below is the one I tried but its giving error.
GET _search
{
"query": {
"bool": {
"must": [
{
"range": {
"date": {
"gte": "1438367180542",
"lte": "1738367180542"
}
}
}
],
"term": {
"eventName.keyword": "XXXXXXX"
}
}
}
}
Here is the error message:
{
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "[bool] query does not support [term]",
"line": 15,
"col": 19
}
],
"type": "parsing_exception",
"reason": "[bool] query does not support [term]",
"line": 15,
"col": 19
},
"status": 400
}
You're almost there! As you can see your range query is surrounded by curly braces, but your term query isn't and is out of the must array. Simply add those and move it to the must array and it will work. Even better use filter instead of must since you only have filters (i.e. you're not interested in scoring here)
GET _search
{
"query": {
"bool": {
"filter": [
{
"range": {
"date": {
"gte": "1438367180542",
"lte": "1738367180542"
}
}
},
{
"term": {
"eventName.keyword": "XXXXXXX"
}
}
]
}
}
}
Viewing documents on per weekday classification?
My data is in a format like this:
{"text": "hi","created_at": "2016-02-21T18:30:36.000Z"}
For this I am using a dateConversion.groovy script and kept in the scripts folder in ES 5.1.1.
Date date = new Date(doc[date_field].value);
java.text.SimpleDateFormat format = new java.text.SimpleDateFormat(format);
format.format(date)
When I executed the following code in ES PLUGIN:
"aggs": {
"byDays": {
"terms": {
"script": {
"lang": "groovy",
"file": "dateConversion",
"params": {
"date_field": "created_at",
"format": "EEEEEE"
}
}
}
} ``
I am getting an exception like this:
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Unable to find on disk file script [dateConversion] using lang [groovy]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "testindex-stats",
"node": "vVhZxH7pQ7CO3qpbYm_uew",
"reason": {
"type": "illegal_argument_exception",
"reason": "Unable to find on disk file script [dateConversion] using lang [groovy]"
}
}
],
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Unable to find on disk file script [dateConversion] using lang [groovy]"
}
},
"status": 400
}
To use a script in an aggregation the script value is not a string but another object. I think you also need to specify lang alongside file.
"aggs": {
"byDays": {
"terms": {
"script": {
"lang": "groovy",
"file": "dateConversion",
"params": {
"date_field": "created_at",
"format": "EEEEEE"
}
}
}
}
}
Some parts of my code need some modifications
{
"aggs": {
"byDays": {
"terms": {
"script":{
"file":"test",
"params": {
"date_field": "created_at",
"format": "EEEEEE"
}
}
}
}
}
}
And also my test.groovy code too
Date date = new Date(doc[date_field].value);
date.format(format);
Cloudant is returning error message:
{"error":"invalid_key","reason":"Invalid key use-index for this request."}
whenever I try to query against an index with the combination operator, "$or".
A sample of what my documents look like is:
{
"_id": "28f240f1bcc2fbd9e1e5174af6905349",
"_rev": "1-fb9a9150acbecd105f1616aff88c26a8",
"type": "Feature",
"properties": {
"PageName": "A8",
"PageNumber": 1,
"Lat": 43.051523,
"Long": -71.498852
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
-71.49978935969642,
43.0508382914137
],
[
-71.49978564033566,
43.052210148524
],
[
-71.49791499857444,
43.05220740550381
],
[
-71.49791875962663,
43.05083554852429
],
[
-71.49978935969642,
43.0508382914137
]
]
]
}
}
The index that I created is for field "properties.PageName", which works fine when I'm just querying for one document, but as soon as I try for multiple ones, I would receive the error response as quoted in the beginning.
If it helps any, here is the call:
POST https://xyz.cloudant.com/db/_find
request body:
{
"selector": {
"$or": [
{ "properties.PageName": "A8" },
{ "properties.PageName": "M30" },
{ "properties.PageName": "AH30" }
]
},
"use-index": "pagename-index"
}
In order to perform an $or query you need to create a text (full text) index, rather than a json index. For example, I just created the following index:
{
"index": {
"fields": [
{"name": "properties.PageName", "type": "string"}
]
},
"type": "text"
}
I was then be able to perform the following query:
{
"selector": {
"$or": [
{ "properties.PageName": "A8" },
{ "properties.PageName": "M30" },
{ "properties.PageName": "AH30" }
]
}
}