I am using this config file to create index/import data from couchdb.
input {
couchdb_changes {
db => "roles"
host => "localhost"
port => 5984
}
}
output {
elasticsearch {
document_id => "%{[#metadata][_id]}"
document_type => "%{[#metadata][type]}"
host => "localhost"
index => "roles_index"
protocol => "http"
host => localhost
port => 9200
}
}
I was able to run logstash with this config file and import data
once. I closed command prompt to shutdown logstash and reran cmd prompt
and stash with the config file again. but now I cannot see any index
created. Is there anything that I might be doing wrong here. I am using
ctl+c to kill logstash in cmd prompt. Will appreciate any help.
Thanks.
in case someone comes here looking for the answer of same thing...I set sequence_path => "my_couchdb_seq" in couchdb_changes { } section of my config file and it worked. Each time i want to run logstash to create index, value in this file should be replaced with 0. Got to link: https://discuss.elastic.co/t/index-not-creating-from-couchdb-with-logstash/27848/9 for details
Related
I have the following problem, I need logstash to reparse already parsed files:
Scenario that doesn't work but should:
upload file to watched folder
logstash processes it, saves to elastic, removes it (file_completed_action => "log_and_delete"), great
I upload the same file again, same name, same content.
logstash doesnt do anything, I want it to process it again
Here is my file input config:
file {
mode => "read"
exclude => "*.tif"
path => ["/home/xmls/*.xml"]
file_completed_action => "log_and_delete"
file_completed_log_path => "/var/log/logstash/completed.log"
sincedb_path => "/dev/null"
start_position => "beginning"
codec => multiline {
pattern => ".*"
what => "previous"
max_lines => 100000
max_bytes => "200 MiB"
}
type => "my-custom-type-1"
}
sincedb_path is set to /dev/null, it should not remember processed files, also tried setting ignore_older to 0, didn't help.
Also tried messing with queue settings in logstash.yml, changed it to persistent, didn't work ...
I'm using logstash version 7.5, logstash-input-file (4.1.11), running in linux machine.
When I restart logstash, then the unprocessed files get processed and cleaned up.
I need it to work without restarting.
I am having issues with puppet modules, and this modules should replace /etc/ssh/sshd_config file based on Redhat version. So the issue is, after applying the code, puppet deleted the file, instead of replacing it.
someone please suggest any wrong with my code.
here is the my puppet manifest file;
class os_vul_ssh {
case $::operatingsystemmajrelease {
'6':{$sshconfigfile = 'sshd_config.rhel6'}
'7':{$sshconfigfile = 'sshd_config.rhel7'}
}
package { "openssh-server":
ensure => installed,
}
service { 'sshd':
ensure => "running",
enable => true,
require => Package["openssh-server"],
}
file { "/etc/ssh/sshd_config":
owner => root,
group => root,
mode => '0644',
source => "puppet:///modules/os_vul/${::sshconfigfile}",
require => Package["openssh-server"],
notify => Service["sshd"],
}
}
file { "/etc/ssh/sshd_config":
ensure => file, <----- this is missing
owner => root,
group => root,
mode => '0644',
source => "puppet:///modules/os_vul/${::sshconfigfile}",
require => Package["openssh-server"],
notify => Service["sshd"],
}
Might be more going on here, but this is the first issue that jumps out at me.
By the way, you can cleanup your code with this:
file { "/etc/ssh/sshd_config":
ensure => file,
owner => root,
group => root,
mode => '0644',
source => "puppet:///modules/os_vul/sshd_config.rhel${::operatingsystemmajrelease}",
require => Package["openssh-server"],
notify => Service["sshd"],
}
and if you are using Facter 3 then consider changing your fact to:
$facts['operatingsystemmajrelease']
and note that your sshconfigfile is a local variable and should be included in your file resource as a local variable $sshconfigfile and not global $::sshconfigfile.
Hithere,
My log files is stored in remote server where the directory is only accessible via browser.
Each day if there is a new log files uploaded in the server, it will be stored like this,
fttp://serverip.com/logs/2014/10/08/log.txt
ftttpp://serverip.com/logs/2014/10/08/log2.txt
fffttpp://serverip.com/logs/2014/10/08/log.xml
fffttttppp://serverip.com/logs/2014/10/08/log.xlx
the timestamp would be the time its uploaded to the server(i can use curl to see its timestamp)
input {
exec {codec => plain { }
command => "curl ftp://serverip.com/logs/2014/10/08/" #this list the dir
interval => 3000}
}
output {
stdout { codec => rubydebug }
#elasticsearch {embedded => true}
}
the problem is how can i combine/link the timestamp with the event file in the directories, because there is no timestamp in the log files.
I'm migrating our old process of doing our linux configurations to be managed by puppet but I'm having issues trying to figure out how to do this. We add some custom scripts to the init.d folder on our systems to manage some processes, and then these need this command executed on them to launch on startup:
update-rc.d $file defaults
So what I'm doing with puppet is that I have all these scripts residing a directory, and I copy them over to init.d. I then want to call 'exec' on each of these files with the former command and use the file name as an argument. This is what I have so far:
#copy init files
file { '/etc/init.d/':
ensure => 'directory',
recurse => 'remote',
source => ["puppet:///files/init_files/"],
mode => 755,
notify => Exec[echo],
}
exec { "echo":
command => "update-rc.d $file defaults",
cwd => "/tmp", #directory to execute from
path => "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:",
refreshonly => true
}
This will copy all the files and it calls exec when something is added/updated, but what I can't figure out is how to pass the name of the file as an argument into the exec command. It seems I'm really close but I just can't find anything to help with what I need to do. Is this the right way to try and achieve this?
Thanks.
Your probably not going to accomplish that if your using ensure => 'directory'. You will want to declare the file resource for each individual init script. And the exec isn't the way to go to enable a service. Use the service resource.
file { '/etc/init.d':
ensure => 'directory',
mode => '0755'
}
file { '/etc/init.d/init_script_1':
ensure => 'present',
owner => 'root',
group => 'root',
mode => '0755',
notify => Service['init_script_1']
}
file { '/etc/init.d/init_script_2':
ensure => 'present',
owner => 'root',
group => 'root',
mode => '0755',
notify => Service['init_script_2']
}
service { 'init_script_1':
ensure => running,
enable => true
}
service { 'init_script_2':
ensure => running,
enable => true
}
Hope this helps.
I try to write a puppet configuration in order to install lamp env.
But i have an issue with notify option.
I have an apache conf:
class apache inherits apache::params {
package { 'apache':
name => "${apache::params::package}",
ensure => present
}
service { 'apache':
ensure => running,
name => $apache::params::service,
enable => true,
subscribe => Package['apache'],
}
}
and and php module conf:
define php::module(
$notify = $php::params::notify,
$package_prefix = $php::params::module_package_prefix
) {
package { "php-module-${name}":
ensure => present,
name => "${package_prefix}${name}",
notify => Service['apache'],
require => [Class['apache'], Package['php', 'php-dev']]
}
}
but when I launch puppet I have this error:
Error: Parameter notify failed on Php::Module[mcrypt]: No title provided and "apache" is not a valid resource reference
I don't understand why it said that apache service is not a valid resources ?
I think there might be 2 issues here:
1) Puppet doesn't like this line in php::module:
$notify = $php::params::notify,
Can you try to remove that or check what is in $php::params::notify? (I don't see you using it)
2) Did you have something like
include apache
in your site.pp? The class still needs to be declared before you can reference the contained resources.