How to get a pretty print for my input file - python-3.x

I have a file that contains data as shown in the Input file below. My program reads the config file, and writes section of this file as individual files. I read this file using python3 and pyyaml module. I get all the data I want, but when the data is written out to the output file, instead of the pretty-printed output, there are all these extra characters HOw can I get it pretty printed?
My ugly output:
"input {\n beats {\n port => 5044\n host => \"0.0.0.0\"\n tags => [\"output_beats\"\
]\n add_field => {\n \"[es][port]\" => 5044\n \"[es][type]\" => \"\
beats\" \n \"[es][subtype]\" => \"%{[#metadata][beat]}\"\n \"[#metadata][queue_prefix]\"\
\ => \"%{[#metadata][beat]}\"\n }\n }\n}\n"
Input file:
---
# Source: logstash/templates/configmap-receiver.yaml
apiVersion: v1
... removed for clarity
data:
100_beats_receiver_input_5044.conf : |
input {
beats {
port => 5044
host => "0.0.0.0"
tags => ["output_beats"]
add_field => {
"[es][port]" => 5044
"[es][type]" => "beats"
"[es][subtype]" => "%{[#metadata][beat]}"
"[#metadata][queue_prefix]" => "%{[#metadata][beat]}"
}
}
}
My code is pretty simple:
def read_and_process_yaml_file(filePath, outputDir):
"""Read file, return parsed python structure"""
print("About to read " + filePath)
with open(filePath,'r') as input_file:
yamlDocs = load_yaml(input_file)
for doc in yamlDocs:
print(yaml.dump(doc))
if (doc is not None) \
and (doc["kind"] is not None) and (doc["kind"].lower() == 'configmap') \
and ("test" not in doc["metadata"]["name"]):
print("doc.name=" + doc["metadata"]["name"])
for name, data in doc["data"].items():
print("name=" + name)
basename = name.split(".",1)
filePath = outputDir + "/" + basename[0] + ".yaml"
print(name + "|-")
#print(yaml.dump(data))
write_yaml_file(yaml.dump(data), filePath)

data is the string containing the desired output. If you do yaml.dump(data), you encode that string as a YAML scalar. Just do
write_yaml_file(data, filePath)

Related

Logstash multiline plugin not working for empty lines

I have these logs:
2019-04-01 12:45:33.207 ERROR [validator,,,] 1 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_VALIDATOR/e5d3dc665009:validator:8789 - was unable to send heartbeat!
com.netflix.discovery.shared.transport.TransportException: Retry limit reached; giving up on completing the request
at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:138) ~[eureka-client-1.4.12.jar!/:1.4.12]
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89) ~[eureka-client-1.4.12.jar!/:1.4.12]
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92) ~[eureka-client-1.4.12.jar!/:1.4.12]
at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77) ~[eureka-client-1.4.12.jar!/:1.4.12]
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89) ~[eureka-client-1.4.12.jar!/:1.4.12]
...
I want to combine all these lines to the same line, so I used this input in logstash:
input {
tcp {
port => 5002
codec => json
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => previous
}
type => "logspout-logs-tcp"
}
}
But it is not working, I don't know if it's beacuase of the empty line on the second line, if so, how can I resolve this problem? I am using logstash version 5.6.14.
Please check the below code,
input {
tcp {
port => 5002
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => "previous"
}
type => "logspout-logs-tcp"
}
}

How to clean up files created by Exec resources?

I am trying to write a puppet class that will create a cirros image with OpenStacks Glance.
I have this puppet class. It downloads the image file and converts it to raw.
It then creates the glance image using the raw image format file.
I also want to remove the downloaded image file and the raw image file from
local disk.
Here is the manifest I tried:
class create_glance_cirros_image (
$cirrosver = '0.3.5',
$cirros_download_url = 'http://download.cirros-cloud.net',
$curl = '/usr/bin/curl',
$download_dir = '/root',
$qemu_img = '/usr/bin/qemu-img',
$qemu_img_args = 'convert -f qcow2 -O raw',
$image_name = 'cirros',
$is_public = 'no',
$container_format = 'bare',
$disk_format = 'raw',
$min_ram = '1024',
$min_disk = '1',
$properties = { 'img_key' => img_value },
$ensure = 'present',
) {
$cirros_image = "cirros-${cirrosver}-x86_64-disk.img"
$raw_cirros_image = "cirros-${cirrosver}-x86_64-disk.raw"
$image_url = "${cirros_download_url}/${cirrosver}/${cirros_image}"
$target_file = "${download_dir}/${cirros_image}"
$raw_target_file = "${download_dir}/${raw_cirros_image}"
$curl_args = "--output ${target_file}"
$download_command = "${curl} ${curl_args} ${image_url}"
$convert_command = "${qemu_img} ${qemu_img_args} ${target_file} ${raw_target_file}"
exec { $download_command:
creates => $target_file,
refreshonly => true,
}
exec { $convert_command:
creates => $raw_target_file,
refreshonly => true,
require => Exec[$download_command],
}
glance_image { $image_name:
ensure => $ensure,
name => $image_name,
is_public => $is_public,
container_format => $container_format,
disk_format => $disk_format,
source => $raw_target_file,
min_ram => $min_ram,
min_disk => $min_disk,
properties => $properties,
require => Exec[$convert_command],
}
file { $target_file:
ensure => 'absent',
}
file { $raw_target_file:
ensure => 'absent',
}
}
When I run it I get this error:
Error: Execution of '/usr/bin/openstack image create --format shell cirros --private --container-format=bare --disk-format=raw --min-disk=1 --min-ram=1024 --property img_key=img_value --file=/root/cirros-0.3.5-x86_64-disk.raw' returned 1: [Errno 2] No such file or directory: '/root/cirros-0.3.5-x86_64-disk.raw'
Error: /Stage[main]/Create_glance_cirros_image/Glance_image[cirros]/ensure: change from absent to present failed: Execution of '/usr/bin/openstack image create --format shell cirros --private --container-format=bare --disk-format=raw --min-disk=1 --min-ram=1024 --property img_key=img_value --file=/root/cirros-0.3.5-x86_64-disk.raw' returned 1: [Errno 2] No such file or directory: '/root/cirros-0.3.5-x86_64-disk.raw'
Why didn't the require cause the exec's to execute?
Update: Based on Matt's suggestions I modified my manifest to look like this:
exec { $download_command:
creates => $target_file,
unless => "/usr/bin/openstack image list --format=value | cut -d' ' -f2 | grep \"^${image_name}$\"",
notify => Exec[$convert_command],
}
exec { $convert_command:
creates => $raw_target_file,
refreshonly => true,
}
glance_image { $image_name:
ensure => present,
name => $image_name,
is_public => $is_public,
container_format => $container_format,
disk_format => $disk_format,
source => $raw_target_file,
min_ram => $min_ram,
min_disk => $min_disk,
properties => $properties,
}
exec { "/bin/rm -f ${target_file}":
subscribe => Exec[$convert_command],
refreshonly => true,
}
file { $raw_target_file:
ensure => 'absent',
require => Glance_image[$image_name],
}
Setting your exec resources to refreshonly means that they require a refresh signal to trigger and be applied. This can be done with a subscribe or a notify. Since your second exec depends upon the first, you can do this as:
exec { $download_command:
creates => $target_file,
refreshonly => true,
notify => Exec[$convert_command],
}
or:
exec { $convert_command:
creates => $raw_target_file,
refreshonly => true,
subscribe => Exec[$download_command],
}
The first one is trickier since it does not establish a relationship with anything. If you want the file download to be idempotent, I would recommend using a file resource instead.
file { $target_file:
source => $image_url,
}
This would cause both of your resources to be idempotent and apply when only when you want them to, thus achieving your goal.
You would need to modify your image file removal to be an exec though. Something like this would work:
exec { "/bin/rm -f ${target_file}":
subscribe => Exec[$convert_command]
refreshonly => true,
}
Your raw image file removal also needs to be applied after its creation and usage:
file { $raw_target_file:
ensure => 'absent',
require => Glance_image[$image_name],
}

Logstash - Data from Kafka to ES

Using logstash 5.0.0, Taking kafka source as the input -> taking the data and producing the output in Elasticsearch. (ElasticSearch version 5.0.0)
Logstash conf:
input{
kafka{
bootstrap_servers => "XXX.XXX.XX.XXX:9092","XXX.XXX.XX.XXX:9092","XXX.XXX.XX.XXX:9092"
topics => ["a-data","f-data","n-data"]
group_id => "sound"
auto_offset_reset => "earliest"
consumer_threads => 2
}
}
filter{
json{
source => "message"
}
}
output {
elasticsearch {
hosts => [ "XXX.XXX.XX.XXX:9200" ]
}
}
When I run the below configuration , i am getting this following error.
$ ./logstash -f sound.conf
Sending Logstash logs to /logstash-5.0.0/logs which is now configured vi a log4j2.properties.
[2017-01-17T10:53:29,273][ERROR][logstash.agent ] fetched an invalid c onfig {:config=>"input{\nkafka{\nbootstrap_servers => \"XX.XXX.XXX.XX:9092\",\"XXX.XXX.XX.XXX:9092\",\"XXX.XXX.XX.XXX:9092\"\ntopics => [\"a-data\",\"f-data\ ",\"n-data\"]\ngroup_id => \"sound\"\nauto_offset_reset => \"earliest\"\nc onsumer_threads => 2\n}\n}\nfilter{\njson{\nsource => \"message\"\n}\n}\noutput {\nelasticsearch {\nhosts => [ \"XX.XX.XXX.XX:9200\" ]\n}\n}\n\n", :reason=>"Ex pected one of #, {, } at line 3, column 40 (byte 54) after input{\nkafka{\nboots trap_servers => \"XX.XX.XXX.XX:9092\""}
Can anyone help me with this configuration.
Shouldn't your topic be topics which is an array, where you've inserted the values as a hash:
topics => ["a-data","f-data","n-data"] <-- try changing this line

unable to read file using logstash (issue with input plugin)

I am new to logstash on ubuntu. I am using file plugin in the input where I have given the path for displaying the content of the file on the stdout.This is my configuration file
input{
file{
path =>"/home/om/Desktop/app/logstash/logstash-1.4.22/logs.txt"
start_position => "beginning"
}
}
output{
stdout{}
}
I am not getting any output on the console. I want to dump the data from the .txt file on the console.
Use this configuration:
input {
file {
path => [ "\\IpAddress\logs/filename.*.*_bak"" ]
start_position => "beginning"
}
}
output {
elasticsearch {
bind_host => "127.0.0.1"
port => "9200"
protocol => http
}
stdout { codec => rubydebug }
}
And also checking with below commands:
>logstash --configtest -f logstash.conf --> for configuration testing
>logstash --debug -f logstash.conf --> for debugging

puppet: Syntax error at 'target'; expected '}' - parsing error

Syntax error while parsing puppet resource.
class nagios::export {
##nagios_host { $::fqdn:
address => $::ipaddress,
use => "linux-server",
check_command => 'check-host-alive!3000.0,80%!5000.0,100%!10',
hostgroups => 'all-servers',
target => "/etc/nagios/resource.d/host_${::fqdn}.cfg"
}
##nagios_service { "check_ping_${hostname}":
check_command => "check-host-alive!100.0,20%!500.0,60%",
use => "generic-service",
host_name => "$fqdn",
notification_period => "24x7",
#target => "/etc/nagios/resource.d/service_${::fqdn}.cfg"
service_description => "${hostname}_check_ping"
target => "/etc/nagios/resource.d/service_${::fqdn}.cfg"
}
}
When I run puppet apply , following error is seen..
[root#ip-10-172-161-25 manifests]# puppet apply export.pp --noop
Could not parse for environment production: Syntax error at 'target'; expected '}' at /etc/puppet/modules/nagios/manifests/export.pp:28 on node ip-10-172-161-25.us-west-1.compute.internal
class nagios::export {
##nagios_host { $::fqdn:
address => $::ipaddress,
use => "linux-server",
check_command => 'check-host-alive!3000.0,80%!5000.0,100%!10',
hostgroups => 'all-servers',
target => "/etc/nagios/resource.d/host_${::fqdn}.cfg",
}
##nagios_service { "check_ping_${hostname}":
check_command => "check-host-alive!100.0,20%!500.0,60%",
use => "generic-service",
host_name => "$fqdn",
notification_period => "24x7",
#target => "/etc/nagios/resource.d/service_${::fqdn}.cfg",
service_description => "${hostname}_check_ping",
target => "/etc/nagios/resource.d/service_${::fqdn}.cfg",
}
}
Was just a few missing commas at the end.
I generally always finish with a comma on the last line. It's not needed, but catches gotcha's that happen when you add an extra line to the end.

Resources