How to get logs[data] generated by running application using Logstash Container - logstash

I am new to logstash and docker.
I am using logstash container to retrieve logs, generated by other application. As i am able to run container successful but i am not getting the output in my file "finalOuput.log" [ both the file finalOuput.log & pipeline.conf is in container]
1->
My Docker File Content:
FROM logstash
copy pipeline.conf /var/app/
copy finalOuput.log /var/app/
CMD ["-f", "/var/app/pipeline.conf"]
2-> My pipeline.config file contains
input {
file {
path => "/predix/Documents/workspace-sts-3.8.1.RELEASE/sendRohit/9aug/application-logs/application-log.log"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
output {
file {
path => "/finalOuput.log"
codec => line { format => "custom format: %{message}"}
}
}

Now i am getting the data. Need to put the pipeline.conf in the root directory of container.
FROM logstash
run mkdir rvol
copy third-pipeline.conf /
copy outputdata.log /rvol/
copy inputData.log /rvol/
CMD ["-f", "/third-pipeline.conf"]
And then fire the command: Below command is single command
docker run -d --name=container_Name -v '/predix/Documents/rvol/inputData.log':'/rvol/inputData.log' -v '/predix/Documents/rvol/outputdata.log':'/rvol/outputdata.log' image_Name

Related

logstash custom patterns don´t get resolved

I´m trying to setup an environment for grok debugging and made this with a docker.
Everything works fine, until logstash tries to resolve a custom pattern.
Here is my environment
I start the docker with
docker run -it --name logstash_debug -v
/home/cloud/docker-elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
-v /home/cloud/docker-elk/logstash/pipeline/:/usr/share/logstash/pipeline/
-v /home/cloud/docker-elk/logstash/patterns/:/usr/share/logstash/patterns
docker.elastic.co/logstash/logstash:7.2.0
As I said, logstash starts up, loads the pipeline (debug.conf)
input { stdin {} }
filter {
grok {
patterns_dir => ["/usr/share/logstash/patterns"]
match => ["message", "%{YEAR1} \[%{LOGLEVEL:loglvl}\] %{GREEDYDATA:message}"]
}
date {
match => ["customer_time", "${YEAR1}"]
target => "#timestamp"
}
}
output { stdout { codec => rubydebug } }
and gives me this error:
Cannot evaluate ${YEAR1}. Replacement variable YEAR1 is not
defined in a Logstash secret store or as an Environment entry and
there is no default value given.
the patterns_dir contains a file "dateformats" which contains (stripped it down to a minimum)
YEAR1 %{YEAR}
the logstash debug output gives me this:
[DEBUG][logstash.filters.grok ] config
LogStash::Filters::Grok/#patterns_dir =
["/usr/share/logstash/patterns"]
[DEBUG][logstash.filters.grok ] config
LogStash::Filters::Grok/#match = {"message"=>"%{YEAR1}
\[%{LOGLEVEL:loglvl}\] %{GREEDYDATA:message}"}
.....
[DEBUG][logstash.filters.grok ] config
LogStash::Filters::Grok/#patterns_files_glob = "*"
Normally logstash should be able to grab this file (I even started the docker with --user 0 to be sure that I have no permission problem) but it somehow can´t.
Anyone can me give a hint to what´s going on ?
Thanks and cheers,
Wurzelseppi

Logstash doesnt read from configured input file

I am trying to configure my Logstash to read from a specified log file. When I configure it to read from stdin it works as expected, my input results in a message from Logstash and displays in my Kibana UI.
$ cat /tmp/logstash-stdin.conf
input {
stdin {}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
$./logstash -f /tmp/logstash-stdin.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
The stdin plugin is now waiting for input:
hellloooo
{
"#version" => "1",
"host" => "myhost.com",
"#timestamp" => 2017-11-17T16:05:41.595Z,
"message" => "hellloooo"
}
However, when I run Logstash with a file input I get no indication that the file is loaded into Logstash, and it does not show in Kibana.
$ cat /tmp/logstash-simple.conf
input {
file {
path => "/tmp/test_log.txt"
type => "syslog"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
$ ./logstash -f /tmp/logstash-simple.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Any suggestions of how I can troubleshoot why my Logstash is not ingesting the configured file?
By default the file input plugin starts reading at the end of the file, so only lines added after Logstash starts will be processed. To read all existing lines upon startup add the option "start_position" => "beginning" to the configuration, as explained in documentation.

puppet not able to run shell script from master to client

I am new to puppet. I want to run a shell script call crfs.sh located under /myscripts on a RHEL linux puppet master server.
How do I execute this script on a client or target server?
What you want can be solved using the file and the exec modules of puppet.
class mymodule::myclass {
file { 'my_bash_script':
ensure => 'file',
source => 'puppet:///modules/mymodule/my_bash_script.sh',
path => '/usr/local/bin/my_bash_script.sh',
owner => 'root'
group => 'root'
mode => '0744', # Use 0700 if it is sensitive
notify => Exec['run_my_script'],
}
exec { 'run_my_script':
command => '/usr/local/bin/my_bash_script.sh',
refreshonly => true,
}
}

Logstash service doesnt use config

Logstash service doesn't want to read my conf file in /etc/logstash/conf.d/.
Service is running all the time.
For example:
...
output {
stdout { codec => rubydebug }
file { path => "/home/vagrant/LOGSTASH_TEST.txt" }
}
And nothing is added to the file.
But when I'm running bin/logstash -f /etc/logstash/conf.d/myconfig.conf — everything is fine.
/etc/init.d/logstash is looking good:
LS_CONF_DIR=/etc/logstash/conf.d
...
program=/opt/logstash/bin/logstash
args="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"
What could be the problem?
Try changing /etc/init.d/logstash to:
LS_CONF_DIR=/etc/logstash/conf.d
LS_CONF_FILE=myconfig.conf
...
program=/opt/logstash/bin/logstash
args="agent -f ${LS_CONF_DIR}/${LS_CONF_FILE} -l ${LS_LOG_FILE} ${LS_OPTS}"

Unable to push data from file to elastic search

I am trying to read a son file data and visualize it in Kibana.The following is my stack.
read json file --> logstash --> elastic search -> Kibana (UI)
I tried the following simple configuration and it works fine till it reaches kibana.
input { stdin { } }
output {
elasticsearch { host => localhost }
}
When I tried to read the data from file and push it to elastic.I am not able to see the output .
input {
stdin {
type => "stdin-type"
}
file {
type => "jsonlog"
# Wildcards work, here :)
path => [ "/Users/path/logstash-1.5.0/sample.json" ]
codec => json
}
}
output {
stdout { }
elasticsearch { embedded => true }
}
Output : It says "logstash started".But I could not see the results in elastic nor the stdout
Jun 10, 2015 4:32:10 PM org.elasticsearch.node.internal.InternalNode start
INFO: [logstash-MacBook-Pro.local-12298-9782] started
Logstash startup completed
Software Version :
Logstash -> 1.5.0
Elasticsearch -> 1.5.2
Thanks in advance !

Resources