Probably a n00b issue trying to get the json_lines codec to read data from a file.
Here's what my config file looks like
input {
file {
path => ['C:/dev/logstash-5.1.2/data/sample.log']
start_position => "beginning"
sincedb_path => 'C:/dev/logstash-5.1.2/data/.sincedb'
codec => "json_lines"
}
}
output {
file {
path => ['C:/dev/logstash-5.1.2/data/sample-output.log']
flush_interval => 0
}
}
Here's what my super simple input file looks like
{"id":1,"name":"A green door","price":12.50,"tags":["home","green"]}
{"id":2,"name":"A red door","price":12.50,"tags":["home","red"]}
When I switch the codec to plain the file gets read and output gets written as expected. But no matter what I do I'm unable to get the json_lines codec to read and write this data.
I am pretty new to logstash, so this might just be something simple that I'm just not able to wrap my head around. Any help would be most appreciated!
Cheers!
On the json_lines documentation it has this warning:
NOTE: Do not use this codec if your source input is line-oriented JSON, for example, redis or file inputs. Rather, use the json codec. More info: This codec is expecting to receive a stream (string) of newline terminated lines. The file input will produce a line string without a newline. Therefore this codec cannot work with line oriented inputs.
Use the json codec instead.
Related
The below is the code that I want to ask.
input {
file {
path => "directory/*.log"
start_position => "beginning"
codec => my_own_codec_plugin {
....
}
sincedb_path => "/dev/null"
}
}
I have some log files in same directory. I can reach out them with using * in path. I have created "my_own_codec_plugin" for file input plugin.
I want to pass the log filename to "my_own_codec_plugin".
I mean if path reaches the logfile1.log send the name to codec plugin, then it reaches logfile2.log send the filename to the codec plugin again.
How can i do this ? Thanks for answering
In your custom codec, you're receiving the event and the event should have a path field with the actual path of the file that you can use.
I'm using node module csv-parser for the streaming csv parsing. It's working fine when uploading a CSV (Comma separated value) but when we upload a CSV (Macintosh) file the problem occurs with line breaks. The CSV that's generated on Windows contains the line breaks like this \r\n but with CSV (MAC) it contains only \r as it's the Mac format. What configuration needs to be done to make it work for both file types?
Here's the code snippet where the streams hooking is done.
// Create a read stream for the passed file path and abort if the file is not found
let readStream: fs.ReadStream;
try {
readStream = fs.createReadStream(filePath);
} catch (error) {
console.log('Skipped order batch file processing. File not found.');
resolve();
return;
}
// Create the CSV transform
let csvStream: Transform;
if (file.mapping) {
csvStream = csv({ headers: false });
} else {
csvStream = csv();
}
readStream
.pipe(csvStream);
CSV-PARSER has an option of newline parameter it's default value is "\n" using "\r" it worked.
csvStream = csv({ headers: false, newline:"\r" });
How can I make the newline value to conditionally set for example if it's csv (Mac) it should "\r" for CSV (Windows) "\r\n" and for linux "\n"?
Note: I need to detect this on File Reading.
Your Help would be really appreciated!
Thanks!
I have the following problem, I need logstash to reparse already parsed files:
Scenario that doesn't work but should:
upload file to watched folder
logstash processes it, saves to elastic, removes it (file_completed_action => "log_and_delete"), great
I upload the same file again, same name, same content.
logstash doesnt do anything, I want it to process it again
Here is my file input config:
file {
mode => "read"
exclude => "*.tif"
path => ["/home/xmls/*.xml"]
file_completed_action => "log_and_delete"
file_completed_log_path => "/var/log/logstash/completed.log"
sincedb_path => "/dev/null"
start_position => "beginning"
codec => multiline {
pattern => ".*"
what => "previous"
max_lines => 100000
max_bytes => "200 MiB"
}
type => "my-custom-type-1"
}
sincedb_path is set to /dev/null, it should not remember processed files, also tried setting ignore_older to 0, didn't help.
Also tried messing with queue settings in logstash.yml, changed it to persistent, didn't work ...
I'm using logstash version 7.5, logstash-input-file (4.1.11), running in linux machine.
When I restart logstash, then the unprocessed files get processed and cleaned up.
I need it to work without restarting.
I'd like to write an array of objects to an ANSI (windows-1252) encoded CSV. I'm using the fast-csv and iconv-lite packages. Is there a way to do this without going through a buffer or intermediate streams? My code (which writes a ASCII CSV at present) is as follows:
csv
.writeToStream(
fs.createWriteStream(filename, {encoding: "ascii"}), objectArray, {headers: true})
.on("finish", function() {
{console.log("done!");
});
If you're using iconv-lite, you can use an encodeStream like so:
iconv.encodeStream("win1252").pipe(fs.createWriteStream(filename))
in place of the usual createWriteStream invocation.
Hithere,
My log files is stored in remote server where the directory is only accessible via browser.
Each day if there is a new log files uploaded in the server, it will be stored like this,
fttp://serverip.com/logs/2014/10/08/log.txt
ftttpp://serverip.com/logs/2014/10/08/log2.txt
fffttpp://serverip.com/logs/2014/10/08/log.xml
fffttttppp://serverip.com/logs/2014/10/08/log.xlx
the timestamp would be the time its uploaded to the server(i can use curl to see its timestamp)
input {
exec {codec => plain { }
command => "curl ftp://serverip.com/logs/2014/10/08/" #this list the dir
interval => 3000}
}
output {
stdout { codec => rubydebug }
#elasticsearch {embedded => true}
}
the problem is how can i combine/link the timestamp with the event file in the directories, because there is no timestamp in the log files.