Logstash configtest - logstash

I ran service logstash configtest but error given was:
logstash: unrecognized service
I was able to run logstash service individually but not with "configtest". In etc/logstash/conf.d/ I created logstash.conf file where consist of code as present below:-
Additional info:-
service logstash status
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled)
Active: active (running) since Mon 2016-12-26 12:40:58 PST; 6s ago
Main PID: 3512 (java)
CGroup: /system.slice/logstash.service
└─3512 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX...
Dec 26 12:40:58 Mr systemd[1]: Started logstash.
Service while running with configtest:-
root#Mr:/# service logstash configtest
logstash: unrecognized service
I am running this on debian8 machine, hope i will get a good solution from you guys.
# This input block will listen on port 10514 for logs to come in.
# host should be an IP on the Logstash server.
# codec => "json" indicates that we expect the lines we're receiving to be in JSON format
# type => "rsyslog" is an optional identifier to help identify messaging streams in the pipeline.
input {
udp {
host => "logstash_private_ip"
port => 10514
codec => "json"
type => "rsyslog"
}
}
# This is an empty filter block. You can later add other filters here to further process
# your log lines
filter { }
# This output block will send all events of type "rsyslog" to Elasticsearch at the configured
# host and port into daily indices of the pattern, "rsyslog-YYYY.MM.DD"
output {
if [type] == "rsyslog" {
elasticsearch {
hosts => [ "elasticsearch_private_ip:9200" ]
}
}
}

for old logstash
/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/
Later, it became installed in /usr/share/logstash so try either
/usr/share/logstash/bin/logstash --configtest -f <the config file/folder>
Or if running version 5.1+ use --config.test_and_exit
/usr/share/logstash/bin/logstash --config.test_and_exit -f <the config file/folder>

I had the same problem and it helped me a lot:
If you are running Logstash version 5, the following command to test the configuration will give you an error:
sudo /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/
The right command to test it is:
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t
I'm running ELK + filebeat since an Ubuntu Server 16.04 and my result was:
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK
Sources:
https://www.elastic.co/guide/en/logstash/current/running-logstash.html; https://github.com/elastic/logstash/issues/6161

for logstash 5.1 its
/usr/share/logstash/bin/logstash --config.test_and_exit -f logstash.yml

Using the official logstash Docker image to validate a local file:
docker run -it -v /etc/logstash:/etc/logstash logstash /usr/share/logstash/bin/logstash -t -f /etc/logstash/logstash.conf
This assumes your config file is locally in /etc/logstash, then mounts that folder into the container, under the same path. Then the binary can find the config file inside the container.
There might be a better way to run that command, this worked for me.

If you want to test your configs with a docker container logstash 6.x
docker run -it -v $PWD:/etc/logstash/conf.d --entrypoint "bin/logstash" logstash "--config.test_and_exit"

You should do this on CentOS version 7:
/etc/rc.d/init.d/logstash configtest /etc/logstash/conf.d/test.conf

On Centos 7, you can also use the following command:
/usr/share/logstash/bin/logstash --path.settings /etc/logstash/
--config.test_and_exit /etc/logstash/conf.d/logstash.conf

Related

How to running logstash on background as a services

I have logstash configuration and running well when I running this on terminal
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/receiver.conf --path.settings /etc/logstash
The problem is, I want to logstash running on background as a service. But its not work when I tried to running with
systemctl start logstash
This is configuration on my /etc/logstash/logstash.yml
path.data: /var/lib/test
path.config: /etc/logstash/conf.d/receiver.conf
path.logs: /var/log/logstash
This is configuration on my /etc/logstash/pipeline.yml
- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
I'm using Linux 10 to running this logstash. Is there any changes I have to do with the services or I missing something?

Logstash: how to reset everything

I'm new to Logstash and Elasticsearch.
After some tests I want to restart and make a new index.
However, I'm getting this:
root#imageoffice:/etc/logstash/conf.d# sudo -Hu root /usr/share/logstash/bin/logstash --path.settings /etc/logstash/
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-04-22T17:00:04,910][FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[2019-04-22T17:00:04,965][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
How can I simple reset everything and run the sudo -Hu root /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ command succesfull?
The error message tells you that there's already a logstash running on your machine. If you don't want the old one running, shut it down or kill it.
Typically, logstash is managed through a startup script (systemd, etc).
Removed the .lock file from bin/data and the error was gone.

Filebeat command not found

I am running ELK in a docker container at localhost, I am trying to start filebeat (not through docker container) but I am receiving the error below.
/etc/init.d/filebeat: command not found
Has anyone had similar issues or any idea how to resolve this?
Also they have mentioned:
If you use an init.d script to start Filebeat on deb or rpm, you can’t specify command line flags (see Command Line Options). To specify flags, start Filebeat in the foreground.
But don't understood. I have checked a lot but nothing helped. I am running this in ubuntu 16.10
Sounds like you want to start filebeat on the host but you don't have it installed
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html
Since you are using a Linux distribution with systemd, have you tried starting filebeat as a systemd service? Check the status:
systemctl status -l filebeat
If that returns info about the service, then filebeat is set up to be a systemd service, not an init.d spawn. Enable it to have it persist after reboot with:
sudo systemctl enable filebeat
And start it with:
sudo systemctl start filebeat

How to start a Docker container with cloud-config file for CoreOS?

I'm trying to configure my CoreOS server with Terraform, using cloud-config file for CoreOS. I am currently trying to set up a Mongo database in a Docker container.
Here is my config file:
write_files:
- path: "/home/core/keyfile"
permissions: "0600"
owner: "999"
content: |
hUoQVrERB0*** <here is my key for MongoDB>
coreos:
units:
- name: "dockerstart.service"
command: "start"
content: |
[Unit]
Description=Start
Author=Me
[Service]
Restart=always
ExecStart=/usr/bin/docker run --name mongo -v /home/core:/opt --add-host node1.example.com:127.0.0.1 -p 27017:27017 -d mongo:2.6.5 --smallfiles --keyFile /opt/keyfile --replSet "rs0"
ExecStop=/usr/bin/docker rm -f mongo
I am not sure how to use coreOS units (when I ssh into the server, the docker container is not running, so the config file is not correct). According to CoreOS Validator, my file is valid. Also, I am not sure if that is the simplest way to deploy a MongoDB server.
How to properly use CoreOS units ? Any thoughts on a way to improve how deploy a Mongo Database ?
Any help, comments, suggestions are appreciated !
I finally found the solution.
Actually running docker run with -d option daemonizes the command. So, when systemd founds out that this action runs in the background, it considers that Docker is crashing.
Here is journalctl -u dockerstart.service result on server :
docker[1237]: ace3978442a729420ecb87af224bd146ec6ac7912c5cc452570735f4a3be3a79
docker[1297]: mongo
systemd[1]: dockerstart.service: Service hold-off time over, scheduling restart.
systemd[1]: Stopped Start.
systemd[1]: Started Start.
Here you can clearly see that systemd stops and restarts the Start service.
So the solution for this might be removing -d from the docker run command.
If help you in the future, you can use a Container Linux Config file format for setup the initial config for CoreOS:
I published an example that create an ignition config based on a Container Linux Config file applied to Terraform in: https://github.com/joariasl/terraform-ansible-docker-swarm-coreos-aws/tree/feature/coreos-etcd
About this: https://coreos.com/os/docs/latest/provisioning.html

Why `systemctl list-unit-files` command output does not contain logstash

I Installed ELK (Elasticsearch, Logstash and Kibana) with YUM onto our centos based system and set them as start at boot time. Then When I tried command systemctl list-unit-files, the output only contains Kibana and Elasticsearch without Logstash at all. But Logstash is indeed listed in command chkconfig --list
According to my understanding, it means that Logstash is a SysV service, whereas Kibana and Elasticsearch are systemd services. Is it so? If so why is Logstash different from Elasticsearch and Kibana as a SysV service other than systemd service?

Resources