I am trying to develop a new logstash plugin in my development Elasticsearch stack environment.
I started with generating the skeleton for my new plug-in using the logstash-plugin generate utility. Once I had this in place I then was able to build my plug-in using the gem build utility which ran without error and created a gem for me. I then ran the logstash-plugin install utility which successfully installed my plug-in.
The entry in my logstash.gem looks like this--
gem "logstash-output-s3-4.3.3", "0.1.0", :path => "./vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.3.3"
I then took the code from the actual logstash plug-in that I'm hoping to modify and copied it into my new plug-in directory and then created a new logstash.conf file to test with using my new plug-in and started the logstash service. It ran as expected moving data from the servers that were being watched in the appropriate s3 buckets.
logstash.conf output section--
output {
s3{
region => "us-east-1"
bucket => "xxxx"
prefix => "Dev/xxxx/%{+YYYY}/%{+MM}/%{+dd}/"
server_side_encryption => true
server_side_encryption_algorithm => "aws:kms"
ssekms_key_id => "xxxx"
validate_credentials_on_root_bucket => false
codec => "json_lines"
size_file => 1024000
time_file => 5
rotation_strategy => "size_and_time"
temporary_directory => "../../LogstashS3OutputData/temp/"
canned_acl => "private"
}
s3{
region => "us-east-1"
bucket => "xxxx"
prefix => "Dev/xxxx/%{+YYYY}/%{+MM}/%{+dd}/"
server_side_encryption => true
server_side_encryption_algorithm => "aws:kms"
ssekms_key_id => "xxxx"
validate_credentials_on_root_bucket => false
codec => "json_lines"
size_file => 1024000
time_file => 5
rotation_strategy => "size_and_time"
temporary_directory => "../../LogstashS3OutputData/temp/"
canned_acl => "private"
}
}
My next step was and this is where I am having issues was to try and create a new property for the plug-in called file_prefix--
s3{
region => "us-east-1"
bucket => "xxxx"
prefix => "Dev/xxxx/xxxx/%{+YYYY}/%{+MM}/%{+dd}/"
file_prefix => "test"
server_side_encryption => true
server_side_encryption_algorithm => "aws:kms"
ssekms_key_id => "arn:aws:kms:xxxx"
validate_credentials_on_root_bucket => false
codec => "json_lines"
size_file => 1024000
time_file => 5
rotation_strategy => "size_and_time"
temporary_directory => "../../LogstashS3OutputData/temp/"
canned_acl => "private"
}
I modified the following files in my solution
\vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\spec\outputs\s3_spec.rb
added "file_prefix" => file_prefix at line 15
\vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\spec\supports\helpers.rb
added let(:file_prefix) {} at line line 12
added "file_prefix" => file_prefix, at line 19
\vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\lib\logstash\outputs\s3.rb
added config :file_prefix, :validate => :string right after the statement config :prefix, :validate => :string, :default => '' at line 154
added #file_prefix = file_prefix at line 226
Now when I start the logstash service I get this error message
Unknown setting 'file_prefix' for s3
I've searched and I've tried everything I can think but I can't get past this hurdle.
TIA,
Bill Youngman
This can be considered closed as I am using a similar feature request in the s3 output plugin developer branch as a starting point for my project.
Thanks,
-Bill
Related
I have two cookbooks: elasticsearch and curator.
Elasticsearch cookbook installs and configure an elasticsearch. The following resource (from elasticsearch cookbook), has to be modified from curator cookbook:
elasticsearch_configure 'elasticsearch' do
configuration ({
'http.port' => port,
'cluster.name' => cluster_name,
'node.name' => node_name,
'bootstrap.memory_lock' => false,
'discovery.zen.minimum_master_nodes' => 1,
'xpack.monitoring.enabled' => true,
'xpack.graph.enabled' => false,
'xpack.watcher.enabled' => true
})
end
I need to modify it on curator cookbook and add a single line:
'path.repo' => (["/backups/s3_currently_dev", "/backups/s3_currently", "/backups/s3_daily", "/backups/s3_weekly", "/backups/s3_monthly"])
How I can do that?
I initially was going to point you to the chef-rewind gem, but that actually points to the edit_resource provider that is now built into Chef. A basic example of this:
# cookbook_a/recipes/default.rb
file 'example.txt' do
content 'this is the initial content'
end
.
# cookbook_b/recipes/default.rb
edit_resource! :file, 'example.txt' do
content 'modified content!'
end
If both of these are in the Chef run_list, the actual content within example.txt is that of the edited resource, modified content!.
Without fully testing your case, I'm assuming the provider can be utilized the same way, like so:
edit_resource! :elasticsearch_configure, 'elasticsearch' do
configuration ({
'http.port' => port,
'cluster.name' => cluster_name,
'node.name' => node_name,
'bootstrap.memory_lock' => false,
'discovery.zen.minimum_master_nodes' => 1,
'xpack.monitoring.enabled' => true,
'xpack.graph.enabled' => false,
'xpack.watcher.enabled' => true,
'path.repo' => ["/backups/s3_currently_dev", "/backups/s3_currently", "/backups/s3_daily", "/backups/s3_weekly", "/backups/s3_monthly"]
})
end
I am completely new to Puppet and this is my first time writing code in puppet. I want to get a tar.gz file and then untar it to create the folder.
Here is my code:
file{ "${::filename}.tar.gz":
ensure => 'file',
mode => '0644',
notify => Exec['untar-file'],
}
exec{ 'download-file' :
command => "wget URL_FOR_TAR_GZ",
cwd => "PATH_WHERE_TO_STORE",
user => "my_name",
group => "our company name",
}
exec { 'untar-file':
command => "/bin/tar -xzvf tar_file_name",
cwd => "file_path",
creates => "foldername_to_be_createdc",
user => "my_name",
group => "our company name",
require => Exec['download-file']
}
As soon as I run this I get an error:
wget returned 8 instead of one of [0]" and "/Exec[download-file]/returns: change from notrun to 0 failed"
Where am I going wrong?
Using logstash 5.0.0, Taking kafka source as the input -> taking the data and producing the output in Elasticsearch. (ElasticSearch version 5.0.0)
Logstash conf:
input{
kafka{
bootstrap_servers => "XXX.XXX.XX.XXX:9092","XXX.XXX.XX.XXX:9092","XXX.XXX.XX.XXX:9092"
topics => ["a-data","f-data","n-data"]
group_id => "sound"
auto_offset_reset => "earliest"
consumer_threads => 2
}
}
filter{
json{
source => "message"
}
}
output {
elasticsearch {
hosts => [ "XXX.XXX.XX.XXX:9200" ]
}
}
When I run the below configuration , i am getting this following error.
$ ./logstash -f sound.conf
Sending Logstash logs to /logstash-5.0.0/logs which is now configured vi a log4j2.properties.
[2017-01-17T10:53:29,273][ERROR][logstash.agent ] fetched an invalid c onfig {:config=>"input{\nkafka{\nbootstrap_servers => \"XX.XXX.XXX.XX:9092\",\"XXX.XXX.XX.XXX:9092\",\"XXX.XXX.XX.XXX:9092\"\ntopics => [\"a-data\",\"f-data\ ",\"n-data\"]\ngroup_id => \"sound\"\nauto_offset_reset => \"earliest\"\nc onsumer_threads => 2\n}\n}\nfilter{\njson{\nsource => \"message\"\n}\n}\noutput {\nelasticsearch {\nhosts => [ \"XX.XX.XXX.XX:9200\" ]\n}\n}\n\n", :reason=>"Ex pected one of #, {, } at line 3, column 40 (byte 54) after input{\nkafka{\nboots trap_servers => \"XX.XX.XXX.XX:9092\""}
Can anyone help me with this configuration.
Shouldn't your topic be topics which is an array, where you've inserted the values as a hash:
topics => ["a-data","f-data","n-data"] <-- try changing this line
I have LS_JAVA_OPTS = -DproxySet=true -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=8888
And yet, I see no traffic to my elasticsearch node from logstash in Fiddler.
I know my elasticsearch is up and running. When I curl it, Fiddler clearly shows the requests, so it is something about jruby that does not route requests through Fiddler.
I am not calling jruby directly. Rather I use the bin\logstash.bat script.
Appendix
My conf file:
input {
file {
path => 'c:/log/bje-Error.log'
sincedb_path => "NUL"
codec => plain {
charset => "ISO-8859-1"
}
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
start_position => beginning
ignore_older => 0
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{BASE10NUM:thread:int}] %{WORD:machine}:%{WORD:service} \[%{BASE10NUM:localId:int}?:%{UUID:logId}?:(?<jobKind>[^:]+)?:%{BASE10NUM:jobDefinitionId:int}? %{WORD:namespace}?:%{WORD:job}?:(?<customCtx>[^\]]*)\] %{LOGLEVEL:level} %{NOTSPACE:logger} - (?<text>(?m:.*))" }
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
document_type => 'logs_bje'
hosts => ["ncesearch01"]
}
}
Testing in powershell:
PS E:\logstash-2.3.2\bin> (ConvertFrom-Json((Invoke-WebRequest "http://ncesearch01:9200/logstash-*/_count").Content)).count
24666
PS E:\logstash-2.3.2\bin> .\logstash.bat -f C:\dayforce\DayforceDEV\elk\logstach.conf
LS_JAVA_OPTS was set to [-DproxySet=true -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=8888]. This will be appended to the JAVA_OPTS [ -XX:HeapDumpPath="$LS_HOME/heapdump.hprof"]
io/console not supported; tty will not be manipulated
Settings: Default pipeline workers: 12
Pipeline main started
{
"message" => "2016-05-02 16:00:05.7079 [111] CANWS212:MyBJE [2251:e2737eeb-40d6-4b0e-9608-75ee3de894d3:ScheduledInstance:16 DFUnitTest:BillingDataCollectionJob:] ERROR
SharpTop.Engine.BackgroundJobs.Billing.BillingDataCollectionJob - The client database version is not defined in DFDatabaseIdentification \r",
"#version" => "1",
"#timestamp" => "2016-05-03T03:40:50.531Z",
"path" => "c:/log/bje-Error.log",
"host" => "CANWS212",
"timestamp" => "2016-05-02 16:00:05.7079",
"thread" => 111,
"machine" => "CANWS212",
"service" => "MyBJE",
"localId" => 2251,
"logId" => "e2737eeb-40d6-4b0e-9608-75ee3de894d3",
"jobKind" => "ScheduledInstance",
"jobDefinitionId" => 16,
"namespace" => "DFUnitTest",
"job" => "BillingDataCollectionJob",
"level" => "ERROR",
"logger" => "SharpTop.Engine.BackgroundJobs.Billing.BillingDataCollectionJob",
"text" => "The client database version is not defined in DFDatabaseIdentification \r"
}
^CTerminate batch job (Y/N)? ←[33mSIGINT received. Shutting down the agent. {:level=>:warn}←[0m
stopping pipeline {:id=>"main"}
Pipeline main has been shutdown
The signal HUP is in use by the JVM and will not work correctly on this platform
^CPS E:\logstash-2.3.2\bin> (ConvertFrom-Json((Invoke-WebRequest "http://ncesearch01:9200/logstash-*/_count").Content)).count
24667
PS E:\logstash-2.3.2\bin>
As you can see, http://ncesearch01:9200/logstash-*/_count returns incremented count, hence running logstash did send a request to the elasticsearch. However, it bypassed Fiddler, despite the LS_JAVA_OPTS.
I find some possible reasons for this condition，although I did not try.May this answer should be called "discussion"，I`m sorry.
1.You may need a linux OS instead of windows，for the reason，
I am not sure this question has been deal in the latest logstash version
you may be interested in this,Make JAVA_OPTS and LS_JAVA_OPTS work consistently on Windows
2.As we see，the most possible is that
logstash ES_output plugin use the http way to send message
after logstash-2.0,you may use the old version?
moreInfo about ES_output_plugin,logstash-output-plugin-elasticsearch
If anyone has any ideas,your share will be expected~
A small puppet question
I am creating a composer project like so.
composer::project { 'project-test':
ensure => 'latest', #or installed?
target => '/home/test/www',
dev => false,
require => Package ['php', 'apache']
}
And then from an exec I want to require it as a resource. How can I?
Example of exec:
exec { 'generate-tests' :
command => 'php tests.php',
path => '/usr/bin/',
cwd => "/home/test/www/bin",
logoutput => 'true',
#require => composer::project['project-test']
}
Since what you're requiring is a resource, it should be capitalized as follows:
require => Composer::Project['project-test']