I can't figure it out how to write log files using fluentd, any help is welcome.
P.S.: I know that my config file is probably full of redundancies, it's because I was trying many things.
I'm executing using the Td-Agent prompt with the following command:
fluentd -c etc\td-agent\td-agent.conff
<source>
#type forward
bind 0.0.0.0
port 24230
</source>
<match **>
#type stdout
</match>
<match **>
#type file
path logs
add_path_suffix true
path_suffix ".txt"
flush_interval 1
flush_mode.immediate
flush_at_shutdown
compress text
append true
<buffer>
#type file
path logb/logs.*.txt
</buffer>
</match>
Use out_copy plugin to use multiple output plugins.
Related
I have a service that reads the logs from STDOUT for further analysis. It seems like there has been struggle with writing spark logs to STDOUT, by default, log4j sends any kind of log to STDERR.
Is there a way to change this behavior?
What changes need to be made specifically to move logs from STDERR to STDOUT?
Here's what my log4j file looks like:
log4j.rootLogger=INFO, FILE
log4j.appender.FILE=org.apache.log4j.RollingFileAppender
log4j.appender.FILE.File=stderr
log4j.appender.FILE.ImmediateFlush=true
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.conversionPattern=....
log4j.appender.FILE.MaxFileSize=5248997
log4j.appender.FILE.MaxBackupIndex=10
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
When you do a spark submit add 2>&1 at the the end. This means combine stderr(2) and stdout(1) into the stdout stream.
To do it through log4j.properties file, try adding the below properties.
# Log everything INFO and above to stdout
log4j.rootLogger=INFO,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.Threshold=INFO
log4j.appender.console.Target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=[%d] %-5p %.26c [%X{testName}] [%X{akkaSource}] - %m%n
Having already a defined custom rsyslog configuration like:
:msg, regex, "myappname", /appl/logs/myappname.log
How can I prevent the logs being written both on /var/log/messages and /appl/logs/myappname.log?
Figured this out as I should just add:
:msg, regex, "myappname" ~
as the 2nd line
So I whipped up a docker-based fluentd TCP log collector.
Following the examples here, https://docs.fluentd.org/input/tcp , led to successfully sending a line from my host Win 10 WSL (Debian) by saying
echo "my_service: 08:03:10 INFO [my_py_file:343]: My valuable log info." | netcat 127.0.0.1 5170
This arrived in fluentd as a nice JSON, as hoped-for. But I want to do it from python 3.7! So:
import socket
def netcat(hn: str, p: int, content: bytes):
"""https://www.instructables.com/id/Netcat-in-Python/"""
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((hn, p))
sock.sendall(content)
sock.close()
msg_raw = "my_service: 08:03:10 INFO [my_py_file:343]: My valuable log info."
netcat('127.0.0.1', 5170, bytes(msg_raw, 'utf-8'))
WSL or no: This python script runs through, no exceptions. Also no reaction at all from fluentd, which I cannot explain. Could and would any of you?
In case it is of any consequence: Here is the relevant section from my fluentd.conf.
<source>
#type tcp
#label mainstream
#id pawc_tcp
tag paws.tcp
port 5170
bind 0.0.0.0
# https://docs.fluentd.org/parser/regexp
<parse>
#type regexp
expression /^(?<service_uuid>[a-zA-Z0-9_-]+): (?<logtime>[^\s]+) (?<loglvl>[^\s]+) \[(?<file>[^\]:]+):(?<line>\d+)\]: (?<msg>.*)$/
time_key logtime
time_format %H:%M:%S
types line:integer
</parse>
</source>
<label mainstream>
<match paws.tcp>
#type file
#id output_tcp
path /fluentd/log/tcp.*.log
symlink_path /fluentd/log/tcp.log
</match>
</label>
Try sending a \r\n or \0 at the end of your message. The message is being sent as bytes over the network so it's probably being stored in buffers and the code reading the buffer needs a way to know the message is over. The regex is also matching on line terminators so will be necessary I think there as well.
As Alex W states above, a \n is needed for the TCP line being accepted by the fluentd regex I use. I'd like to add a second answer to improve the python code of the original question.
There actually is a readily-implemented logging.handler.SocketHandler class! However, it pickles its outputs, looking at a python log server. Using fluentd this means one has to override the emit function to use it. After that all works fine.
import logging, logging.handlers
class SocketHandlerBytes(logging.handlers.SocketHandler):
def emit(self, record):
try:
msg = bytes(self.format(record) + "\n", 'utf-8')
self.send(msg)
except Exception:
self.handleError(record)
sh = SocketHandlerBytes(host, port)
sh.setFormatter(logger_format_appropriate_for_your_fluentd_tcp_regex)
logging.root.addHandler(sh)
I am using Nlog for logs and rotating file on daily basis. Log files are read and processed further by another system to move to Splunk.
Here I am looking to find out what steps nlog takes to rotate file because it is failing my integration with other systems. It works fine with other log framework like log4net/logback. If that's case, I may need to switch to other framework.
Ideally steps should be - Approach 1
1. Open file - abc.txt
2. add log line 1, log line 2.. log line n.
3. 12.00 AM - file rotation - Rename file abc.txt => 2020-05-13 abc.txt
4. create new file - abc.txt 5. add log line a1, log line a2.. log line an.
Another approach may be - Approach 2
1. open file - abc.txt
2. add log line 1, log line 2.. log line n.
3. 12.00 AM - file rotation - create new file 2020-05-13 abc.txt
4. move contents from abc.txt to new file 2020-05-13 abc.txt
5. Update file offset/pointer to move to new location i.e. start of the file (as contents are removed from this file)
6. add log line a1, log line a2.. log line an - at the begining location (offset/file pointer)
Can anybody please confirm which approach or sequence of steps are exactly performed by Nlog for file rotation.
[UPDATES]
Below is how my config looks like-
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<variable name="logDir" value="C:\logs">
<targets>
<target name="logFile" xsi:type="File" fileName="${logDir}\API_${machinename}.json"
archiveFileName="${logDir}\{#}_API_${machinename}.json" archiveNumbering="Date"
archiveDateFormat="yyyy-MM-dd" archiveEvery="Day">
<layout xsi:type="JsonLayout" includeAllProperties="true">
<attribute name="time" layout="${date:format=yyyy-MM-ddTHH\:mm\:ss.fffzzz}"/>
<attribute name="level" layout="${level:upperCase=true}"/>
<attribute name="message" layout="${message}"/>
<attribute name="exception" layout="${exception:format=#}"/>
</layout>
</target>
</targets>
<rules>
<logger name="*" minlevel="Debug" writeTo="logFile" />
</rules>
</nlog>
I'm trying to setup a fluentd service to collect logs and send them to elastic search.
All is good, except I cannot get a custom index name AND keep the timestamp similar to what logstash_format: true would have.
Here is my fluent.conf file:
<source>
#type forward
port 24224
bind 0.0.0.0
</source>
<match *.**>
#type copy
<store>
#type elasticsearch
hosts hostaddressandport
user theuser
password password
include_tag_key true
tag_key #log_name
index_name myindex-%Y.%m
<buffer>
flush_interval 3s
</buffer>
</store>
</match>
The index gets created in elastic literally and it shows myindex-%Y.%m I've tried myindex-${%Y.%m} and get the same behaviour.
If I use logstash_format: true instead, then I get an index like logstash-2019.07.09, but I don't want that.
This is where I'm getting my idea from https://docs.fluentd.org/output/elasticsearch but I don't see the expected behaviour.
I have found the following in the docs mentioned above:
<buffer tag, time>
timekey 1h # chunks per hours ("3600" also available)
</buffer>
But it's pretty vague and I don't understand what chunk_keys are.
You can use logstash_format and logstash_prefix to change the index prefix. This will not use the date format you require though.
logstash_format true
logstash_prefix myindex
please use this config you will get your custom date format for your index name
Config file
<source>
#type forward
port 24224
bind 0.0.0.0
</source>
<match *.**>
#type copy
<store>
#type elasticsearch
hosts hostaddressandport
user theuser
password password
include_tag_key true
tag_key #log_name
logstash_format true
logstash_dateformat %Y.%m
logstash_prefix index_name
<buffer>
flush_interval 3s
</buffer>
</store>
</match>
output
indexname-2021.08