Kibana: Search within text for string - search

I have A log message in Kibana that contains this:
org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:597)
Actual search that isn't returning results: log_message: "hibernate3"
If I search for "hibernate3" this message will not appear. I am using an Elasticsearch template and have indexed the field, but also want to be able to do case-insensitive full-text searching. Is this possible?
Template that is in use:
{
"template": "filebeat-*",
"mappings": {
"mainProgram": {
"properties": {
"#timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"#version": {
"type": "text"
},
"beat": {
"properties": {
"hostname": {
"type": "text"
},
"name": {
"type": "text"
}
}
},
"class_method": {
"type": "text",
"fielddata": "true",
"index": "true"
},
"class_name": {
"type": "text",
"fielddata": "true"
},
"clientip": {
"type": "ip",
"index": "not_analyzed"
},
"count": {
"type": "long"
},
"host": {
"type": "text",
"index": "not_analyzed"
},
"input_type": {
"type": "text",
"index": "not_analyzed"
},
"log_level": {
"type": "text",
"fielddata": "true",
"index": "true"
},
"log_message": {
"type": "text",
"index": "true"
},
"log_timestamp": {
"type": "text"
},
"log_ts": {
"type": "long",
"index": "not_analyzed"
},
"message": {
"type": "text"
},
"offset": {
"type": "long",
"index": "not_analyzed"
},
"query_params": {
"type": "text",
"index": "true"
},
"sessionid": {
"type": "text",
"index": "true"
},
"source": {
"type": "text",
"index": "not_analyzed"
},
"tags": {
"type": "text"
},
"thread": {
"type": "text",
"index": "true"
},
"type": {
"type": "text"
},
"user_account_combo": {
"type": "text",
"index": "true"
},
"version": {
"type": "text"
}
}
},
"access": {
"properties": {
"#timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"#version": {
"type": "text"
},
"beat": {
"properties": {
"hostname": {
"type": "text"
},
"name": {
"type": "text"
}
}
},
"clientip": {
"type": "ip",
"index": "not_analyzed"
},
"count": {
"type": "long",
"index": "not_analyzed"
},
"host": {
"type": "text",
"index": "true"
},
"input_type": {
"type": "text",
"index": "not_analyzed"
},
"log_timestamp": {
"type": "text"
},
"log_ts": {
"type": "long",
"index": "not_analyzed"
},
"message": {
"type": "text"
},
"offset": {
"type": "long",
"index": "not_analyzed"
},
"query_params": {
"type": "text",
"index": "true"
},
"response_time": {
"type": "long"
},
"sessionid": {
"type": "text",
"index": "true"
},
"source": {
"type": "text",
"index": "not_analyzed"
},
"statuscode": {
"type": "long"
},
"tags": {
"type": "text"
},
"thread": {
"type": "text",
"index": "true"
},
"type": {
"type": "text",
"index": "true"
},
"uripath": {
"type": "text",
"index": "true"
},
"user_account_combo": {
"type": "text",
"index": "true"
},
"verb": {
"type": "text",
"index": "true"
}
}
}
}
}

message: *.hibernate3.*
also works (please note, that no quotes are needed for that)

According to your scenario, what you're looking for is an analyzed type string which would first analyze the string and then index it. A quote from the doc.
In other words, index this field as full text.
Thus make sure that, you have your mapping of the necessary fields properly so that you'll be able to do a full-text search on the docs.
Assuming that, in Kibana if the log line is under the field message, you could simply search for the word by:
message:"hibernate3"
You might also want to refer this, to identify the variance between Term Based and Full-Text.
EDIT
Have the mapping of the field log_message as such:
"log_message": {
"type": "string", <- to make it analyzed
"index": "true"
}
Also try doing a wildcard search as such:
{"wildcard":{"log_message":"*.hibernate3.*"}}

With Kibana 6.4.1 I used the "%" as wildcard.
message: %hibernate3%

For me it was because I was using the ".keyword".
My key was called "message" and I had "message" and "message.keyword" available.
Full text search isn't working on ".keyword".
Not working :
message.keyword : hello
Working :
message : hello

Related

How to insert array value into table using logic app

when a http request is received. i need to insert the array value into a table. in my case the array is response required.
I used these things: when a http request is received and i used parse JSON and i used for each loop then inside the for-each i used insert entity but it's throwing an error. if anybody knows how to implement let me know the answer.
i used expression for RRT as : body('Parse_JSON')['ResponseRequired'][0]['ResponseRequiredType']
json schema
{
"$schema": "http://json-schema.org/draft-04/schema#",
"properties": {
"AssetErrorCode": {
"type": "string"
},
"AssetErrorDesc": {
"type": "string"
},
"AssetId": {
"type": "integer"
},
"CustomerId": {
"type": "integer"
},
"ResponseRequired": {
"items": [
{
"properties": {
"ResponseRequiredAdditionalData": {
"type": "string"
},
"ResponseRequiredAddress": {
"type": "string"
},
"ResponseRequiredFrequency": {
"type": "string"
},
"ResponseRequiredType": {
"type": "integer"
}
},
"required": [
"ResponseRequiredType",
"ResponseRequiredFrequency",
"ResponseRequiredAddress",
"ResponseRequiredAdditionalData"
],
"type": "object"
},
{
"properties": {
"ResponseRequiredAdditionalData": {
"type": "string"
},
"ResponseRequiredAddress": {
"type": "string"
},
"ResponseRequiredFrequency": {
"type": "string"
},
"ResponseRequiredType": {
"type": "integer"
}
},
"required": [
"ResponseRequiredType",
"ResponseRequiredFrequency",
"ResponseRequiredAddress",
"ResponseRequiredAdditionalData"
],
"type": "object"
},
{
"properties": {
"ResponseRequiredAdditionalData": {
"type": "string"
},
"ResponseRequiredAddress": {
"type": "string"
},
"ResponseRequiredFrequency": {
"type": "string"
},
"ResponseRequiredType": {
"type": "integer"
}
},
"required": [
"ResponseRequiredType",
"ResponseRequiredFrequency",
"ResponseRequiredAddress",
"ResponseRequiredAdditionalData"
],
"type": "object"
}
],
"type": "array"
},
"ServiceKey": {
"type": "string"
}
},
"required": [
"CustomerId",
"ServiceKey",
"AssetId",
"AssetErrorCode",
"AssetErrorDesc",
"ResponseRequired"
],
"type": "object"
}
used this expression : items('For_each')?['ResponseRequiredAddress']
solved the issue

Azure budget alert schema template

I am setting up some Azure budget alerts to call a Logic App webhook to perform an action.
In the budget I have specified alert conditions to fire an action group at 25%, 50% and 75% of budget. The action group has an action to call the Logic App webhook using the common alert schema.
I have a "When a HTTP request is received" Logic App set up with the simple alert payload and a step to process the request.
In this processing step I would like to have access to details of the budget that triggered the alert (budget name, % of budget etc) but the sample schema does not contain that information:
{
"properties": {
"data": {
"properties": {
"alertContext": {
"properties": {
"condition": {
"properties": {
"allOf": {
"items": {
"properties": {
"dimensions": {
"items": {
"properties": {
"name": {
"type": "string"
},
"value": {
"type": "string"
}
},
"required": [
"name",
"value"
],
"type": "object"
},
"type": "array"
},
"metricName": {
"type": "string"
},
"metricNamespace": {
"type": "string"
},
"metricValue": {
"type": "number"
},
"operator": {
"type": "string"
},
"threshold": {
"type": "string"
},
"timeAggregation": {
"type": "string"
}
},
"required": [
"metricName",
"metricNamespace",
"operator",
"threshold",
"timeAggregation",
"dimensions",
"metricValue"
],
"type": "object"
},
"type": "array"
},
"windowSize": {
"type": "string"
}
},
"type": "object"
},
"conditionType": {
"type": "string"
},
"properties": {}
},
"type": "object"
},
"essentials": {
"properties": {
"alertContextVersion": {
"type": "string"
},
"alertId": {
"type": "string"
},
"alertRule": {
"type": "string"
},
"alertTargetIDs": {
"items": {
"type": "string"
},
"type": "array"
},
"description": {
"type": "string"
},
"essentialsVersion": {
"type": "string"
},
"firedDateTime": {
"type": "string"
},
"monitorCondition": {
"type": "string"
},
"monitoringService": {
"type": "string"
},
"originAlertId": {
"type": "string"
},
"resolvedDateTime": {
"type": "string"
},
"severity": {
"type": "string"
},
"signalType": {
"type": "string"
}
},
"type": "object"
}
},
"type": "object"
},
"schemaId": {
"type": "string"
}
},
"type": "object"
}
Is there somewhere that has a schema template with all of the possible fields for a budget alert? So that my Logic App can use those Budget fields as dynamic content in subsequent steps.
Thanks
I created a logic app which puts the input json into a blob storage. Added this Logic app as Webhook in Budget alert action group.
I received following message. This looks like the schema for budget alerts.
{
"schemaId": "AIP Budget Notification",
"data": {
"SubscriptionName": "",
"SubscriptionId": "",
"EnrollmentNumber": "",
"DepartmentName": "",
"AccountName": "",
"BillingAccountId": "",
"BillingProfileId": "",
"InvoiceSectionId": "",
"ResourceGroup": "",
"SpendingAmount": "",
"BudgetStartDate": "",
"Budget": "",
"Unit": "",
"BudgetCreator": "",
"BudgetName": "",
"BudgetType": "",
"NotificationThresholdAmount": ""
}
}
Looks like Microsoft did mention about this schema in their documentation but in a slightly hidden manner (look for the json in the below article)
https://learn.microsoft.com/en-us/azure/billing/billing-cost-management-budget-scenario#create-an-azure-logic-app-for-orchestration

No result returned from the nest c# elasticsearch query

I am indexing an attachment field. The POST query in sense returns expected resultset.
My query is
POST /mydocs/_search
{
"query" : {
"bool" : {
"must" : [
{ "match" : { "file.content":"abc"} },
{ "match":{"otherDetails":"asd"}},
{ "match" : { "filePermissionInfo.accountValue" : "xyz"} }
]
}
}
}
I need to convert it to a c# Nest code. I tried converting it, but its not returning any result,even it contains data. If I remove the
m.Match(mt1 => mt1.Field(f1 => f1.File.Coontent).Query(queryTerm))
from the below experssion, it returns a result set. Is there any problem with the attachement field?
client.Search<IndexDocument>(s => s
.Index("mydocs")
.Query(q => q
.Bool(b => b
.Must(m =>
m.Match(mt1 => mt1.Field(f1 => f1.File.Coontent).Query(queryTerm)) &&
m.Match(mt2 => mt2.Field(f2 => f2.FilePermissionInfo.First().SecurityIdValue).Query(accountName)) &&
m.Match(mt3 => mt3.Field(f3 => f3.OtherDetails).Query(other))
)))
);
My mapping is
{
"mydocs": {
"mappings": {
"indexdocument": {
"properties": {
"docLocation": {
"type": "string",
"index": "not_analyzed",
"store": true
},
"documentType": {
"type": "string",
"store": true
},
"file": {
"type": "attachment",
"fields": {
"content": {
"type": "string",
"term_vector": "with_positions_offsets",
"analyzer": "full"
},
"author": {
"type": "string"
},
"title": {
"type": "string",
"term_vector": "with_positions_offsets",
"analyzer": "full"
},
"name": {
"type": "string"
},
"date": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"keywords": {
"type": "string"
},
"content_type": {
"type": "string"
},
"content_length": {
"type": "integer"
},
"language": {
"type": "string"
}
}
},
"filePermissionInfo": {
"properties": {
"fileSystemRights": {
"type": "string",
"store": true
},
"securityIdValue": {
"type": "string",
"store": true
}
}
},
"id": {
"type": "double",
"store": true
},
"lastModifiedDate": {
"type": "date",
"store": true,
"format": "strict_date_optional_time||epoch_millis"
},
"otherDetails": {
"type": "string"
},
"title": {
"type": "string",
"store": true,
"term_vector": "with_positions_offsets"
}
}
}
}
}
}
It looks like the query hasn't been translated to NEST correctly. In the query you have
"filePermissionInfo.accountValue"
but in the NEST query, you only have
f2 => f2.FilePermissionInfo
which would result only in filePermissionInfo. You need to change this to
f2 => f2.FilePermissionInfo.AccountValue

mapper_parsing_exception in new elasticsearch 2.1.1 version

Problem : I have created mapping and its working fine in elasticsearch
1.7.1 but after updating to 2.1.1 it will give me exception
EXCEPTION
response: '{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason"
:"analyzer on field [_all] must be set when search_analyzer is set"}],"type":"ma
pper_parsing_exception","reason":"Failed to parse mapping [movie]: analyzer on f
ield [_all] must be set when search_analyzer is set","caused_by":{"type":"mapper
_parsing_exception","reason":"analyzer on field [_all] must be set when search_a
nalyzer is set"}},"status":400}',
toString: [Function],
toJSON: [Function] }
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"analysis": {
"filter": {
"nGram_filter": {
"type": "nGram",
"min_gram": 2,
"max_gram": 20,
"token_chars": [
"letter",
"digit",
"punctuation",
"symbol"
]
}
},
"analyzer": {
"nGram_analyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
"lowercase",
"asciifolding",
"nGram_filter"
]
},
"whitespace_analyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
"lowercase",
"asciifolding"
]
}
}
}
},
"mappings": {
"movie": {
"_all": {
"index_analyzer": "nGram_analyzer",
"search_analyzer": "whitespace_analyzer"
},
"properties": {
"movieName": {
"type": "string",
"index": "not_analyzed"
},
"movieYear": {
"type": "double"
},
"imageUrl": {
"type": "string"
},
"genre": {
"type": "string"
},
"director": {
"type": "string"
},
"producer": {
"type": "string"
},
"cast": {
"type": "String"
},
"writer": {
"type": "string"
},
"synopsis": {
"type": "string"
},
"rating": {
"type": "double"
},
"price": {
"type": "double"
},
"format": {
"type": "string"
},
"offer": {
"type": "double"
},
"offerString": {
"type": "string"
},
"language": {
"type": "string"
}
}
}
}
}
The error is quite clear if you ask me, you need to specify analyzer for _all in your movie mapping. Setting index_analyzer was removed in Elasticsearch 2.0.
"_all": {
"analyzer": "nGram_analyzer",
"search_analyzer": "whitespace_analyzer"
},

hasMany relation loopback

My models.json:
My models.json
"masterTrip": {
"options": {
"relations": {
"trips": {
"type": "hasMany",
"model": "trip",
"foreignKey": "masterTripId"
}
}
}
},
"trip": {
"options": {
"relations": {
"masterTrips": {
"type": "belongsTo",
"model": "masterTrip",
"foreignKey": "masterTripId"
}
}
}
},
But I do not get the relation between the trip and mastertrip. can anybody explain?
There are a couple things that might be the problem. Here is what comes to mind:
You should only need the foreign key on the model that belongsTo the other one. (In this case, that would be trip.)
Have you actually created trips underneath masterTrip? (Either in the code itself or via the API?) I know this sounds silly, but the context wasn't clear enough for me to tell if you had created the sample data or not.
It sounds like you might actually be getting data when you do a GET on /masterTrip/1/trips Is that right? If so , then that sounds like the correct behavior.
I'm still relatively new to LoopBack myself, but I'm not sure that filter[include]=belongsToRelationName is the correct way to get the data you want. Technically, you are just looking for the associated array of hasMany data, right? In this case, trips that belongTo masterTrip. The RESTful way to get that would be masterTrip/{id}/trips
Hope one of those helps.
Your "belongs to" relation name is not singular. It should be singular.
When you making " belongs to " relation name is singular and for hasMany your relation name in plural. Please see the official documentation for more details -
See this working example below -
{
"name": "Booking",
"base": "PersistedModel",
"strict": false,
"idInjection": true,
"properties": {
"myuserId": {
"type": "number"
},
"orgin": {
"type": "string"
},
"orgin_lat": {
"type": "string"
},
"orgin_lon": {
"type": "string"
},
"destination": {
"type": "string"
},
"dest_lat": {
"type": "string"
},
"dest_lon": {
"type": "string"
},
"parcel_type": {
"type": "string"
},
"volume": {
"type": "string"
},
"weight": {
"type": "string"
},
"price": {
"type": "string"
},
"receiver_phn": {
"type": "string"
},
"payment_mode": {
"type": "string"
},
"booking_status": {
"type": "string"
},
"lang": {
"type": "string"
},
"booking_no": {
"type": "string"
},
"cancel_reason": {
"type": "string"
},
"cancel_by": {
"type": "string"
},
"booking_date": {
"type": "string"
},
"plan_later": {
"type": "string"
},
"plan_date": {
"type": "string"
},
"created": {
"type": "string"
},
"modified": {
"type": "string"
}
},
"validations": [],
"relations": {
"biddings": {
"type": "hasMany",
"model": "Bidding",
"foreignKey": "bookingId"
}
},
"acls": [],
"methods": {}
}
{
"name": "Bidding",
"base": "PersistedModel",
"strict": false,
"idInjection": true,
"options": {
"validateUpsert": true
},
"properties": {
"myuserId": {
"type": "number"
},
"bookingId": {
"type": "number"
},
"booking_no": {
"type": "string"
},
"price": {
"type": "string"
},
"message": {
"type": "string"
},
"bid_date": {
"type": "string"
},
"bid_time": {
"type": "string"
},
"bid_status": {
"type": "string"
},
"lang": {
"type": "string"
},
"rate_driver": {
"type": "number"
},
"created": {
"type": "string"
},
"modified": {
"type": "string"
}
},
"validations": [],
"relations": {
"booking": {
"type": "belongsTo",
"model": "Booking",
"foreignKey": "bookingId"
},
"myuser": {
"type": "belongsTo",
"model": "Myuser",
"foreignKey": "myuserId"
}
},
"acls": [],
"methods": {}
}

Resources