How do I add a custom field in my logstash-forwarder configuration file,such that it can be accessed by logstash?
As documented in README.md, the "files" array in the configuration file contains not only a "paths" array but also a "fields" hash where you can specify additional fields to add to each message coming from the specified files. Example:
{
"network": { ... },
"files": [
"paths": [
"/var/log/syslog"
],
"fields": {
"type": "syslog"
}
]
}
Related
I'm using the Graph API to make a global search in my sharepoint website, and I need to retrieve some specific fields. I didn't find any documentation that specifies the available fields that I can use for the fields property on my payload, only a documentation for specific document library search.
I have to use the global search because my search needs to access all the document libraries on my sharepoint web site.
The field that I wanted to get from the request is the version of the document in the list. I could add this field in sharepoint, and my view is displaying the version values, but the request does not take this value. I'm using this request below:
Endpoint: https://graph.microsoft.com/v1.0/search/query
Payload:
"requests": [
{
"entityTypes": [
"listItem"
],
"query": {
"queryString": ""
},
"fields": [
"title",
"_UIVersionString"
]
}
]
}
Response:
{
"value": [
{
"searchTerms": [],
"hitsContainers": [
{
"hits": [
{
"hitId": "83C63693-C621-4CFE-B4F7-A36B68AEB421",
"rank": 1,
"summary": "...",
"resource": {
"#odata.type": "#microsoft.graph.listItem",
"fields": {
"title": "Calc.22090615231879"
}
}
},
],
"total": 1,
"moreResultsAvailable": false
}
]
}
],
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#Collection(microsoft.graph.searchResponse)"
}
The name of the field I'm using in the payload that corresponds to the version is _UIVersionString, where I got it from the specific list search request, using the https://graph.microsoft.com/v1.0/sites/{site-id}/lists/{list-id}/items?$expand=fields endpoint. But sadly, the version is not appearing on my search result.
Is there some documentation I could use to see a list of available fields for this request? I'm trying to find it in MS GraphAPI documentation, but it looks be a big real encyclopedia.
Do you know the name of the field that corresponds to the version?
Thanks a lot!
Other information:
Sharepoint Version: Sharepoint Web (Online)
Type of the lists: Document Library
Lists version configuration:
- Require content approval for submitted items?: No
- Create a version each time you edit a file in this document library?: Create Major Versions
- Require documents to be checked out before they can be edited?: No
Since Graph is using there managed properties from sharepoint you can try "UIVersionStringOWSTEXT".
{
"requests": [
{
"entityTypes": [
"listItem"
],
"query": {
"queryString": "test"
},
"fields": [
"title",
"UIVersionStringOWSTEXT"
]
}
]
}
The results look like
{
"value": [
{
"searchTerms": [
"test"
],
"hitsContainers": [
{
"hits": [
{
"hitId": "GUID",
"rank": 1,
"summary": "test",
"resource": {
"#odata.type": "#microsoft.graph.listItem",
"fields": {
"title": "test",
"uiVersionStringOWSTEXT": "1.0"
}
}
}
],
"total": 1,
"moreResultsAvailable": false
}
]
}
],
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#Collection(microsoft.graph.searchResponse)"
}
In my tenant this property was already created. I guess it is by default. If your search schema does not have this managed property you can create it and name it like you wish. Map it to the crawled property "ows_q_TEXT__UIVersionString" and make sure your managed property is set to "retrievable".
I use the Azure Search indexer to index documents from a MongoDB CosmosDB which contains objects with fields named _id.
As Azure Search does not allow underscores at the beginning of a field name in the index, I want to create a field mapping.
JSON structure in Cosmos --> structure in index
{
"id": "test"
"name": "test",
"productLine": {
"_id": "123", --> "id": "123"
"name": "test"
}
}
The documentation has exactly this scenario as an example but only for a top level field.
"fieldMappings" : [ { "sourceFieldName" : "_id", "targetFieldName" : "id" } ]}
I tried the following:
"fieldMappings" : [ { "sourceFieldName" : "productLine/_id", "targetFieldName" : "productLine/id" } ] }
that results in an error stating:
Value is not accepted. Valid values: "doc_id", "name", "productName".
What is the correct way to create a mapping for a target field that is a subfield?
It's not possible to directly map subfields. You can get around this by adding a Skillset with a Shaper cognitive skill to the indexer, and an output field mapping.
You will also want to attach a Cognitive Services resource to the skillset. The shaper skill doesn't get billed, but attaching a Cognitive Services resource allows you to process more than 20 documents per day.
Shaper skill
{
"#odata.type": "#Microsoft.Skills.Util.ShaperSkill",
"context": "/document",
"inputs": [
{
"name": "id",
"source": "/document/productLine/_id"
},
{
"name": "name",
"source": "/document/productLine/name"
}
],
"outputs": [
{
"name": "output",
"targetName": "renamedProductLine"
}
]
}
Indexer skillset and output field mapping
"skillsetName": <skillsetName>,
"outputFieldMappings": [
{
"sourceFieldName": "/document/renamedProductLine",
"targetFieldName": "productLine"
}
]
trying to add custom skill in the skillset and map it in the index
here is in detail
I'm using the azure Named Entity Recognition in my skillset as
{
"#odata.type": "#Microsoft.Skills.Text.MergeSkill",
"description": "Merge text content with image tags",
"insertPreTag": " ",
"context": "/document",
"inputs": [
{
"name": "text",
"source": "/document/fullTextAndCaptions"
},
{
"name": "itemsToInsert",
"source": "/document/normalized_images/*/Tags/*/name"
}
],
"outputs": [
{
"name": "mergedText",
"targetName": "finalText"
}
]
}
and in the indexer as
{
"sourceFieldName": "/document/finalText/pages/*/entities/*/value",
"targetFieldName": "entities"
},
{
"sourceFieldName": "/document/finalText/pages/*/locations/*",
"targetFieldName": "locations"
},
and it works 100% now I want to add the Distinct custom skill from https://github.com/Azure-Samples/azure-search-power-skills/tree/master/Text/Distinct
I did publish the function and when I go to test it manually it works as expected.
however overall its not working in skillset. I want it to take the location and filter it and output the distinct only in it's own field in the search index.
I'm having a really hard time to configure the skillset and indexer to get it to work.
any help please?
You'll need to add the distinct custom skill like this, assuming you want to dedup over the whole document
{
"skills": [
...
{
"#odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
"description": "Distinct skill",
"uri": "<https://distinct-skill>",
"context": "/document",
"inputs": [
{
"name": "locations",
"source": /document/finalText/pages/*/locations/*"
}
],
"outputs": [
{
"name": "distinct",
"targetName": "distinctLocations"
}
]
}
...
]
}
and an output field mapping to put it into the index.
{
"sourceFieldName": "/document/distinctLocations",
"targetFieldName": "distinctLocations"
}
See https://learn.microsoft.com/en-us/azure/search/cognitive-search-custom-skill-interface#consuming-custom-skills-from-skillset for adding a custom skill.
The skill inputs for the custom skill must be configured to point to the data you want to disambiguate. In this case, you didn't really need to modify the code, all you had to do was have an input with name 'words' and source '/document/finalText/pages//locations/'.
With a standard configuration of ELK stack (deviant/docker-elk)
and the template http://localhost:9200/logstash-alimentaris/_mapping/?pretty=true set to:
{
"string_fields": {
"mapping": {
"fielddata": {},
"index": "analyzed",
"omit_norms": true,
"type": "string",
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string",
"doc_values": true
}
}
},
"match": "*",
"match_mapping_type": "string"
}
}
In Kibana all raw fields are empty. What are some possibilities to investigate what hinders Elasticsearch to fill the raw fields?
One possibility is to create a custom template: Change default mapping of string to "not analyzed" in Elasticsearch
But, it is documented that the raw index works out of the box, and it would be in many cases better to stick with the default configuration. What are possible solutions, hints?
The logic in Kibana is to not show the raw variables in the discover pane. They show up only if one selects "show missing fields", and then they are represented as empty.
In the visualizations one can and should use the .raw fields in most of the aggregations.
I have added more than one path to logstash forwarder config file. But only the first one works. It is not sending logs for the second path.
My config file.
"network": {
"servers": [ "x.x.x.x:5000" ],
"ssl certificate": "./logstash-forwarder.crt",
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15
},
"files": [
{
"paths": [
"/opt/app1/application.log"
],
"fields": { "type": "app1" }
},
{
"paths":[
"/opt/app2/application.log"
],
"fields":{"type": "app2"}
}
]
}
I know I can add them together in the first path block and it will work, but I want to add two different types, which I guess can't be done in one path block.
This is resolved. Actually the issue is that my second log file is not updated for more than a day. So, logstash-forwarder skipped it.
2015/08/07 04:55:14.029097 Skipping file (older than dead time of 24h0m0s):