I am using Filebeat 6.4.2, Logstash 6.3.1 and want to combine all logs files on the filebeat input path . Logs don't have any specific pattern to start with or end with.
Logs don't have any specific pattern. I want to capture all combined logs to Logstash together in bunch of max lines specified.
I tried with multiple RegEx in the pattern sections, it's not working. Problem is logs does'nt come in any specific pattern.
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/application.log
fields:
type: admin
tags: admin
fields_under_root: true
multiline.pattern: '.'
multiline.negate: true
multiline.match: after
multiline.max_lines: 1000
output.logstash:
# The Logstash hosts
hosts: ["xxx.20.x.xxx:5043"]
I want to combine all the multiline logs together as per max_lines configuration’s .
You can specify a pattern that would not be found in your logs like
'^HeLlO$€(^_^)€$bYe'
and it should do the trick.
Related
Having already a defined custom rsyslog configuration like:
:msg, regex, "myappname", /appl/logs/myappname.log
How can I prevent the logs being written both on /var/log/messages and /appl/logs/myappname.log?
Figured this out as I should just add:
:msg, regex, "myappname" ~
as the 2nd line
I am trying to create a Filebeat configuration file after creating a instance with Terraform:
resource "local_file" "greylogIP" {
content = <<EOF
filebeat.inputs:
- type: log
enabled: false
paths:
- /var/log/*.log
filebeat.config.modules:
path: '$'{path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
output.logstash:
hosts: ["${aws_instance.web.public_ip}:5014"]
EOF
filename = "filebeat.conf"
}
where I need to pass the ${aws_instance.web.public_ip} variable to allocate the dynamic IP but some how Terraform is also trying to interpolate path: '$'{path.config}/modules.d/*.yml which is a part of the Filebeat configuration and throws an error.
How can I pass the path: '$'{path.config}/modules.d/*.yml as a string instead?
You need to escape a literal dollar ($) with a double dollar ($$).
The interpolation documentation covers this:
You can escape interpolation with double dollar signs: $${foo} will be rendered as a literal ${foo}.
There's some further mention of it in the template docs:
Important: Template variables in an inline template (such as consul_address above) must be escaped with a double-$. Unescaped interpolations will be processed by Terraform normally prior to executing the template.
I want to uncomment a line in file sshd_config by using Ansible and I have the following working configuration:
- name: Uncomment line from /etc/ssh/sshd_config
lineinfile:
dest: /etc/ssh/sshd_config
regexp: '^#AuthorizedKeysFile'
line: 'AuthorizedKeysFile .ssh/authorized_keys'
However this config only works if the line starts by #AuthorizedKeysFile, but it won't work if the line starts by # AuthorizedKeysFile or # AuthorizedKeysFile (spaces between # and the words).
How can I configure the regexp so it won't take into account any number of spaces after '#'?
I've tried to add another lineinfile option with a space after '#', but this is not a good solution:
- name: Uncomment line from /etc/ssh/sshd_config
lineinfile:
dest: /etc/ssh/sshd_config
regexp: '# AuthorizedKeysFile'
line: 'AuthorizedKeysFile .ssh/authorized_keys'
If you need zero or more white spaces after the '#' character, the following should suffice:
- name: Uncomment line from /etc/ssh/sshd_config
lineinfile:
dest: /etc/ssh/sshd_config
regexp: '^#\s*AuthorizedKeysFile.*$'
line: 'AuthorizedKeysFile .ssh/authorized_keys'
The modification to your original code is the addition of the \s* and the .*$ in the regex.
Explanation:
\s - matches whitespace (spaces, tabs, line breaks and form feeds)
* - specifies that the expression to it's left (\s) can have zero or more instances in a match
.* - matches zero or more of any character
$ - matches the end of the line
Firstly, you are using the wrong language. With Ansible, you don't tell it what to do, but define the desired state. So it shouldn't be Uncomment line form /etc/ssh/sshd_config, but Ensure AuthorizedKeysFile is set to .ssh/authorized_keys.
Secondly, it doesn't matter what the initial state is (if the line is commented, or not). You must specify a single, unique string that identifies the line.
With sshd_config this is possible as the AuthorizedKeysFile directive occurs only once in the file. With other configuration files this might be more difficult.
- name: Ensure AuthorizedKeysFile is set to .ssh/authorized_keys
lineinfile:
dest: /etc/ssh/sshd_config
regexp: AuthorizedKeysFile
line: 'AuthorizedKeysFile .ssh/authorized_keys'
It will match any line containing AuthorizedKeysFile string (no matter if it's commented or not, or how many spaces are there) and ensure the full line is:
AuthorizedKeysFile .ssh/authorized_keys
If the line were different, Ansible will report "changed" state.
On the second run, Ansible will find the AuthorizedKeysFile again and discover the line is already in the desired state, so it will end the task with "ok" state.
One caveat with the above task is that if any of the lines contains a comment such as a real, intentional comment (for example an explanation in English containing the string AuthorizedKeysFile), Ansible will replace that line with the value specified in line.
I should caveat this with #techraf's point that 99% of the time a full template of a configuration file is almost always better.
Times I have done lineinfile include weird and wonderful configuration files that are managed by some other process, or laziness for config I don't fully understand yet and may vary by distro/version and I don't want to maintain all the variants... yet.
Go forth and learn more Ansible... it is great because you can keep iterating on it from raw bash shell commands right up to best practice.
lineinfile module
Still good to see how best to configuration manage one or two settings just a little better with this:
tasks:
- name: Apply sshd_config settings
lineinfile:
path: /etc/ssh/sshd_config
# might be commented out, whitespace between key and value
regexp: '^#?\s*{{ item.key }}\s'
line: "{{ item.key }} {{ item.value }}"
validate: '/usr/sbin/sshd -T -f %s'
with_items:
- key: MaxSessions
value: 30
- key: AuthorizedKeysFile
value: .ssh/authorized_keys
notify: restart sshd
handlers:
- name: restart sshd
service:
name: sshd
state: restarted
validate don't make the change if the change is invalid
notify/handlers the correct way to restart once only at the end
with_items (soon to become loop) if you have multiple settings
^#? the setting might be commented out - see the other answer
\s*{{ item.key }}\s will not match other settings (i.e. SettingA cannot match NotSettingA or SettingAThisIsNot)
Still might clobber a comment like # AuthorizedKeysFile - is a setting which we have to live with because there could be a setting like AuthorizedKeysFile /some/path # is a setting... re-read the caveat.
template module
- name: Configure sshd
template:
src: sshd_config.j2
dest: /etc/ssh/sshd_config
owner: root
group: root
mode: "0644"
validate: '/usr/sbin/sshd -T -f %s'
notify: restart sshd
handlers:
- name: restart sshd
service:
name: sshd
state: restarted
multiple distro support
And if you are not being lazy about supporting all your distros see this tip
- name: configure ssh
template: src={{ item }} dest={{ SSH_CONFIG }} backup=yes
with_first_found:
- "{{ ansible_distribution }}-{{ ansible_distribution_major_version }}.sshd_config.j2"
- "{{ ansible_distribution }}.sshd_config.j2"
https://ansible-tips-and-tricks.readthedocs.io/en/latest/modifying-files/modifying-files/
(needs to be updated to a loop using the first_found lookup)
Is it possible to achieve the same goal with replace module.
https://docs.ansible.com/ansible/latest/modules/replace_module.html
- name: Uncomment line from /etc/ssh/sshd_config
replace:
path: /etc/ssh/sshd_config
regexp: '^\s*#+AuthorizedKeysFile.*$'
replace: 'AuthorizedKeysFile .ssh/authorized_keys'
If you want to simply uncomment a line without setting the value, you can use replace with backreferences, eg (with a handy loop):
- name: Enable sshd AuthorizedKeysFile
replace:
path: /etc/ssh/sshd_config
# Remove comment and first space from matching lines
regexp: '^#\s?(\s*){{ item }}(.+)$'
replace: '\1{{ item }}\2'
loop:
- 'AuthorizedKeysFile'
This will only remove the first space after the #, and so retain any original indenting. It will also retain anything after the key (eg the default setting, and any following comments)
Thanks to the other helpful answers that provided a solid starting point.
I have done such configuration for specific log files for filebeat - files has cp1250 encoding:
-
document_type: collector
encoding: cp1250
ignore_older: 672h
log_type: log
max_bytes: 134217728
paths:
- \\someserver\collector\*
I have done such output configuration:
output:
logstash:
compression_level: 0
hosts:
- localhost:5045
What will be encoding of filebeat on logstash output?
I assume that it can works like this but I think it now works like that:
file(cp1250) -> filebeat(utf-8) -> output(utf-8) -> logstash(utf-8) -> gralog(utf-8).
What is filebeat output encoding really?
You seemed to have asked 2 different questions.
What is filebeat output encoding really?
The event + some metadata it added.
What will be encoding of filebeat on logstash output? / What is default encoding for filebeat to logstash?
Filebeat uses its special plain encoding to read and process your text if no encoding is specified.
From the docs
The plain encoding is special, because it does not validate or transform any input.
I've a logstash instance, version 2.3.1 which isn't running using the command
sudo service logstash start
Whenever I run this command, it returns logstash started and after a few moments when I check the status, I find that logstash isn't running. Although, when I start the logstash from opt to get output on the terminal, it runs without any error.
Note that logstash.err and logstash.stdout files are empty and logstash.log file isn't anywhere to be found. I've also set LS_GROUP to adm in init.d which caused the same issue on another instance, but even that doesn't seem to work now. Any help would be appreciated!
On an Ubuntu system, this behavior can be seen by logstash. To get around it, you can change the logstash user group in /etc/init.d/logstash to adm which stands for admin and you're good to go.
This is normal behaviour of Logstash.
Can you test if your Logstash instance is working correctly?
Windows:
Go to your bin folder of logstash
and type logstash
Linux:
Enter this command in the prompt (bin folder of your logstash instance)
/opt/logstash/bin/logstash
Both:
If you get No command given ... you're logstash instance has the correct setup.
You can always run your Logstash instance with this command
logstash -e 'input { stdin { } } output { stdout {} }'
After this you can enter some text values and they will output to your console.
If this all works you can be sure that your Logstash instance is running correctly.
You may ask yourself why is this? This is because Logstash waits to start untill it gets a config to run with or another option.
If you want to start Logstash automatically on startup. You need to use this command.
sudo update-rc.d logstash defaults 96 9
Actually,you should read the guide of logstash.In the "getting started section",The official documentation has the corret way for you to start a logstash work.
First,you should write a configure file such as "std.conf",look like this:
input {
stdin {
}
}
output{
stdout{
codec=>rubydebug
}
}
Then,start your logstash:
bin/logstash -f conf/std.conf
If you want this work can run in the background(such as get some log files into elasticsearch),you may also need add "&" in the end of the command,like this:
bin/logstash -f conf/getlog.conf &
with this file(std.conf) and this command,your logstash will start up and if you type any word in you terminal,it will print out in the terminal,like this:
{
"message" => "hello",
"#version" => "1",
"#timestamp" => "2016-08-06T19:47:36.543Z",
"host" => "bag"
}
Now,you have got the normal operation of logstah,you may need more information,from there:The official documentation of logstash
Try this,and keep going,it`s easy for you~