i create REST API for the power bi dataset.
It is my config file to transfer data from the power bi service to elasticsearch :
input {
http_poller {
urls => {
powerbi_data => {
method => get
url => "https://api.powerbi.com/../../../dataset_id"
headers => {
Authorization => "Bearer tocken_id"
}
}
}
schedule => { every => "5m" }
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "powerbi"
}
}
Related
I am using Logstash to update by query existing Elasticsearch documents with an additional field that contains aggregate values extracted from Potgresql table.
I use elastichsearch output to load one index using document_id and http output to update another index that have different document_id but receving errors:
[2023-02-08T17:58:12,086][ERROR][logstash.outputs.http ][main][b64f19821b11ee0df1bd165920785876cd6c5fab079e27d39bb7ee19a3d642a4] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://localhost:9200/medico/_update_by_query", :event=>#LogStash::Event:0x19a14c08}
This is my pipeline configuration:
input {
jdbc {
# Postgres jdbc connection string to our database, mydb
jdbc_connection_string => "jdbc:postgresql://handel:5432/mydb"
statement_filepath => "D:\ProgrammiUnsupported\logstash-7.15.2\config\nota_sede.sql"
}
}
filter {
aggregate {
task_id => "%{idCso}"
code => "
map['idCso'] = event.get('idCso')
map['noteSede'] ||= []
map['noteSede'] << {
'id' => event.get('idNota'),
'tipo' => event.get('tipoNota'),
'descrizione' => event.get('descrizione'),
'data' => event.get('data'),
'dataInizio' => event.get('dataInizio'),
'dataFine' => event.get('dataFine')
}
event.cancel()"
push_previous_map_as_event => true
timeout => 60
timeout_tags => ['_aggregatetimeout']
}
}
}
output {
stdout { codec => rubydebug { metadata => true } }
# this works
elasticsearch {
hosts => "https://localhost:9200"
document_id => "STRUTTURA_%{idCso}"
index => "struttura"
action => "update"
user => "user"
password => "password"
ssl => true
cacert => "/usr/share/logstash/config/ca.crt"
}
http {
url => "http://localhost:9200/medico/_update_by_query"
user => "elastic"
password => "changeme"
http_method => "post"
format => "message"
content_type => "application/json"
message => '{
"query":{
"term":{
"idCso":"%{idCso}"
}
},
"script":{
"source":"ctx._source.noteSede=params.noteSede",
"lang":"painless",
"params":{
"noteSede":"%{noteSede}"
}
}
}
}'
}
}
The stdout output show me the sended docs to output like this:
{
"query" => {
"term" => {
"idCso" => "859119"
}
},
"script" => {
"source" => "ctx._source.noteSede=params.noteSede",
"lang" => "painless",
"params" => {
"noteSede" => "{dataFine=null, dataInizio=2020-02-13, descrizione=?, tipo=DB, id=6390644, data=2020-02-13 12:26:58.409},{dataFine=null, dataInizio=2020-02-13, descrizione=?, tipo=DE, id=6390645, data=2020-02-13 12:26:58.41}"
}
}
}
}
How could I set noteSede array field into message to _update_by_query ?
I am new to Logstash manipulations and I have no idea how to do the below.
I have a sample data as below:
Column:Type
Incident Response P3
Incident Resolution L1.5 P2
...
I want to extract the word 'Response' and 'Resolution' into a new column 'SLA type'
Im looking for something very alike to the below SQL statement:
case when Type like '%Resolution%' then Resolution
when Type like '%Response%' then Response
end as SLA_Type
How do i manipulate this in Logstash?
Below is my conf. I'm using an API input.
input {
http_poller {
urls => {
snowinc => {
url => "https://service-now.com"
user => "your_user"
password => "yourpassword"
headers => {Accept => "application/json"}
}
}
request_timeout => 60
metadata_target => "http_poller_metadata"
schedule => { cron => "* * * * * UTC"}
codec => "json"
}
}
filter
{
json {source => "result" }
split{ field => ["result"] }
date {
match => ["[result][sys_created_on]","yyyy-MM-dd HH:mm:ss"]
target => "sys_created_on"
}
}
output {
elasticsearch {
hosts => ["yourelastuicIP"]
index => "incidentsnow"
action=>update
document_id => "%{[result][number]}"
doc_as_upsert =>true
}
stdout { codec => rubydebug }
}
The output for the API json url looks like the below:
{"result":[
{
"made_sla":"true",
"Type":"incident resolution p3",
"sys_updated_on":"2019-12-23 05:00:00",
"number":"INC0010275",
"category":"Network"} ,
{
"made_sla":"true",
"Type":"incident resolution l1.5 p4",
"sys_updated_on":"2019-12-24 07:00:00",
"number":"INC0010567",
"category":"DB"}]}
You can use the following filter block in your pipeline to add a new field if a word is present in another field.
if "response" in [Type] {
mutate {
add_field => { "SLA_Type" => "Response" }
}
}
if "resolution" in [Type] {
mutate {
add_field => { "SLA_Type" => "Resolution" }
}
}
If the word response is present in the field Type a new field named SLA_Type with the value Response will be added to your document, the same in will happen with resolution.
I have fields like A_Latitude, A_Longitude, B_Latitude and B_Longitude. I would like to make use of this data and create Maps in Kibana. The problem is data is getting into elasticsearch, but the gejson columns created in Logstash filter not gettin recognized and data is not being fed into geo_point1 and geo_point2.
Hence, first created a geo_point mapping in Kibana dev tools as follows,
PUT cc-test
{
"mappings": {
"properties": {
"geo_point1":{
"type": "geo_point"
},
"geo_point2":{
"type": "geo_point"
}
}
}
}
I have configured my logstash config file the following way,
input {
jdbc {
# Postgres jdbc connection string to our database, mydb
jdbc_connection_string => "some string"
# The user we wish to execute our statement as
jdbc_user => "User"
jdbc_password => "Password"
# The path to our downloaded jdbc driver
jdbc_driver_library => "/apps/ELK/logstash/driver/ngdbc-2.4.56.jar"
jdbc_driver_class => "com.sap.db.jdbc.Driver"
# our query
#jdbc_validate_connection => true
#schedule => "* * * * *"
#record_last_run => true
# last_run_metadata_path => "login.txt"
statement => "SELECT
inputdata.A_LATITUDE, inpudata.A_LONGITUDE, inputdata.B_LATITUDE,
inputdata.B_LONGITUDE, outputdata.BANDWIDTH, inputdata.SEQUENCEID,
inputdata.REQUESTTIMESTAMP
FROM inputdata, outputdata
WHERE
inputdata.SEQUENCEID = outputdata.SEQUENCEID
AND inputdata.REQUEST_TIMESTAMP >= '2019-01-01 00:00:00'
AND inputdata.SEQUENCEID IS NOT NULL
AND inputdata.SEQUENCEID NOT IN ('N/A')
ORDER BY inputdata.SEQUENCEID DESC "
# jdbc_paging_enabled => "true"
# jdbc_page_size => "10000"
}
}
filter {
mutate {
convert => { "A_LONGITUDE" => "float" }
convert => { "A_LATITUDE" => "float" }
convert => { "B_LONGITUDE" => "float" }
convert => { "B_LATITUDE" => "float" }
}
mutate {
rename => {
"A_LONGITUDE" => "[geo_point1][lon]"
"A_LATITUDE" => "[geo_point1][lat]"
}
}
mutate {
rename => {
"B_LONGITUDE" => "[geo_point2][lon]"
"B_LATITUDE" => "[geo_point2][lat]"
}
}
}
output {
elasticsearch {
hosts => ["http://some server"]
index => "cc-test"
#document_type => "system_logs"
user => "Username"
password => "Password"
}
stdout { codec => rubydebug }
}
Don't understand what is wrong with the Filter part and why data is not getting into the columns geo_point1 and geo_point2!!
Somebody please help :pray::pray::pray:
I want to sent to multiple destinations on logstash, here it is my configuration:
output {
elasticsearch {
hosts => "10.10.10.7:9200"
index => "ubuntu18"
}
kafka {
bootstrap_server => "10.10.10.6:9092"
codec => json
topic_id => "beats"
}
}
But it is not working, any idea?
It seems that your kafka output configuration is wrong, it is bootstrap_servers not bootstrap_server.
output {
elasticsearch {
hosts => "10.10.10.7:9200"
index => "ubuntu18"
}
kafka {
bootstrap_servers => "10.10.10.6:9092"
codec => json
topic_id => "beats"
}
}
We are storing our data types in each of its database in couchdb. What sort of format will the config file have to import data from multiple databases? Or do I need to have multiple config files for importing data from each database to an index. Will appreciate any help.
Thanks.
We use a single config file for multiple databases.
It's not perfect, but functional for now.
Currently looks like:
input {
couchdb_changes {
sequence_path => "db1.seq"
db => "db1"
host => "xxx.xxx.xxx.xxx"
username => "xxx"
password => "xxx"
add_field => {
"organization" => "db1"
}
}
couchdb_changes {
sequence_path => "db2.seq"
db => "db2"
host => "xxx.xxx.xxx.xxx"
username => "xxx"
password => "xxx"
add_field => {
"organization" => "db2"
}
}
}
filter {
mutate {
remove_field => [ "_attachments" ]
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
cluster => "cluster0"
host => ["xxx.xxx.xxx.xxx", "xxx.xxx.xxx.xxx", "xxx.xxx.xxx.xxx"]
protocol => "http"
index => "%{[organization]}"
document_id => "%{[#metadata][_id]}"
}
}