how can I add rabbitmq queue to elk - linux
I want to display RabbitMQ Queues in ELK, for this I ran Kibana, ElasticSearch and LogStash, below are all the details.
The result of running Kiabana
samira#elk:/var/www/apps/kibana-8.4.2-linux-x86_64(1)/kibana-8.4.2/bin$ ./kibana
[2022-10-20T12:37:57.842+03:30][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2022-10-20T12:38:08.711+03:30][INFO ][http.server.Preboot] http server running at http://localhost:5601
[2022-10-20T12:38:08.740+03:30][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2022-10-20T12:38:08.771+03:30][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2022-10-20T12:38:08.935+03:30][INFO ][plugins-system.standard] Setting up [121] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,newsfeed,fieldFormats,expressions,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,lists,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,telemetry,licenseManagement,eventLog,actions,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,unifiedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,eventAnnotation,dataViewFieldEditor,triggersActionsUi,transform,stackAlerts,ruleRegistry,discover,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,aiops,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,osquery,maps,dataVisualizer,ml,cases,timelines,sessionView,kubernetesSecurity,securitySolution,visTypeGauge,sharedUX,observability,synthetics,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,dataViewManagement]
[2022-10-20T12:38:08.948+03:30][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 114adb80-0285-4b29-b403-64ea1e454f19
[2022-10-20T12:38:09.009+03:30][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2022-10-20T12:38:09.028+03:30][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2022-10-20T12:38:09.034+03:30][INFO ][plugins.encryptedSavedObjects] Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: B6ABZzCc0sMI2CQc1eJYyeLXhC0I61v8xdNjUusVvp0=
[2022-10-20T12:38:09.159+03:30][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2022-10-20T12:38:09.189+03:30][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2022-10-20T12:38:09.678+03:30][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2022-10-20T12:38:09.707+03:30][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:9200
[2022-10-20T12:38:10.162+03:30][INFO ][plugins.screenshotting.chromium] Browser executable: /var/www/apps/kibana-8.4.2-linux-x86_64(1)/kibana-8.4.2/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
[2022-10-20T12:39:34.886+03:30][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2022-10-20T12:39:34.887+03:30][INFO ][savedobjects-service] Starting saved objects migrations
[2022-10-20T12:39:34.935+03:30][INFO ][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 28ms.
[2022-10-20T12:39:34.937+03:30][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 27ms.
[2022-10-20T12:39:34.948+03:30][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 1 in 2 seconds.
[2022-10-20T12:39:34.949+03:30][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 11ms.
[2022-10-20T12:39:34.950+03:30][ERROR][savedobjects-service] [.kibana] Action failed with 'search_phase_execution_exception: '. Retrying attempt 1 in 2 seconds.
[2022-10-20T12:39:34.950+03:30][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 15ms.
[2022-10-20T12:39:36.961+03:30][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 2011ms.
The result of running elasticsearch
samira#elk:/var/www/apps/elasticsearch-8.4.2/bin$ ./elasticsearch
warning: ignoring JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64; using bundled JDK
[2022-10-20T12:39:26,417][INFO ][o.e.n.Node ] [elk.kifarunix-demo.com] version[8.4.2], pid[3594], build[tar/89f8c6d8429db93b816403ee75e5c270b43a940a/2022-09-14T16:26:04.382547801Z], OS[Linux/5.15.0-52-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.2.1/18.0.2.1+1-1]
[2022-10-20T12:39:26,422][INFO ][o.e.n.Node ] [elk.kifarunix-demo.com] JVM home [/var/www/apps/elasticsearch-8.4.2/jdk], using bundled JDK [true]
[2022-10-20T12:39:26,422][INFO ][o.e.n.Node ] [elk.kifarunix-demo.com] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11974649730192173872, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms2g, -Xmx2g, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=tar, --module-path=/var/www/apps/elasticsearch-8.4.2/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
[2022-10-20T12:39:27,685][INFO ][c.a.c.i.j.JacksonVersion ] [elk.kifarunix-demo.com] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
[2022-10-20T12:39:28,669][INFO ][o.e.p.PluginsService ] [elk.kifarunix-demo.com] loaded module [aggs-matrix-stats]
[2022-10-20T12:39:28,670][INFO ][o.e.p.PluginsService ] [elk.kifarunix-demo.com] loaded module [analysis-common]
[2022-10-20T12:39:28,670][INFO ][o.e.p.PluginsService ] [elk.kifarunix-demo.com] loaded module [constant-keyword]
[2022-10-20T12:39:28,670][INFO ][o.e.p.PluginsService ] [elk.kifarunix-demo.com] loaded module [data-streams]
The result of running logstash
$ bin/logstash -f sdamiii.conf
this is my sdamiii.conf
input {
rabbitmq {
host => "localhost"
port => 5672
heartbeat => 30
durable => true
queue => "system_logs"
user => "guest"
password => "guest"
vhost => "/"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
rabbitmq {
exchange => "system_logs"
host => "localhost"
exchange_type => "fanout"
key => "logstash"
persistent => false
}
}
This is kibana.yml
elasticsearch.username: "kibana_system"
elasticsearch.password: "pass"
xpack.encryptedSavedObjects.encryptionKey: 5b6d5d7b20e971b2e562cc7c8ca181ae
xpack.reporting.encryptionKey: ba46a278d1dcb511339ea01f2a9d2651
xpack.security.encryptionKey: 11f8ec40b5bf442404f5c5a53b38ad13
This is elasticsearch.yml
network.host: localhost
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: false
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["elk.kifarunix-demo.com"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: localhost
This is logstash.yml
# Settings file in YAML
#
# Settings can be specified either in hierarchical form, e.g.:
#
# pipeline:
# batch:
# size: 125
# delay: 5
#
# Or as flat keys:
#
# pipeline.batch.size: 125
# pipeline.batch.delay: 5
#
# ------------ Node identity ------------
#
# Use a descriptive name for the node:
#
# node.name: test
#
# If omitted the node name will default to the machine's host name
#
# ------------ Data path ------------------
#
# Which directory should be used by logstash and its plugins
# for any persistent needs. Defaults to LOGSTASH_HOME/data
#
# path.data:
#
# ------------ Pipeline Settings --------------
#
# The ID of the pipeline.
#
# pipeline.id: main
#
# Set the number of workers that will, in parallel, execute the filters+outputs
# stage of the pipeline.
#
# This defaults to the number of the host's CPU cores.
#
# pipeline.workers: 2
#
# How many events to retrieve from inputs before sending to filters+workers
#
# pipeline.batch.size: 125
#
# How long to wait in milliseconds while polling for the next event
# before dispatching an undersized batch to filters+outputs
#
# pipeline.batch.delay: 50
#
# Force Logstash to exit during shutdown even if there are still inflight
# events in memory. By default, logstash will refuse to quit until all
# received events have been pushed to the outputs.
#
# WARNING: Enabling this can lead to data loss during shutdown
#
# pipeline.unsafe_shutdown: false
#
# Set the pipeline event ordering. Options are "auto" (the default), "true" or "false".
# "auto" automatically enables ordering if the 'pipeline.workers' setting
# is also set to '1', and disables otherwise.
# "true" enforces ordering on the pipeline and prevent logstash from starting
# if there are multiple workers.
# "false" disables any extra processing necessary for preserving ordering.
#
# pipeline.ordered: auto
#
# Sets the pipeline's default value for `ecs_compatibility`, a setting that is
# available to plugins that implement an ECS Compatibility mode for use with
# the Elastic Common Schema.
# Possible values are:
# - disabled
# - v1
# - v8 (default)
# Pipelines defined before Logstash 8 operated without ECS in mind. To ensure a
# migrated pipeline continues to operate as it did before your upgrade, opt-OUT
# of ECS for the individual pipeline in its `pipelines.yml` definition. Setting
# it here will set the default for _all_ pipelines, including new ones.
#
# pipeline.ecs_compatibility: v8
#
# ------------ Pipeline Configuration Settings --------------
#
# Where to fetch the pipeline configuration for the main pipeline
#
# path.config:
#
# Pipeline configuration string for the main pipeline
#
# config.string:
#
# At startup, test if the configuration is valid and exit (dry run)
#
# config.test_and_exit: false
#
# Periodically check if the configuration has changed and reload the pipeline
# This can also be triggered manually through the SIGHUP signal
#
# config.reload.automatic: false
#
# How often to check if the pipeline configuration has changed (in seconds)
# Note that the unit value (s) is required. Values without a qualifier (e.g. 60)
# are treated as nanoseconds.
# Setting the interval this way is not recommended and might change in later versions.
#
# config.reload.interval: 3s
#
# Show fully compiled configuration as debug log message
# NOTE: --log.level must be 'debug'
#
# config.debug: false
#
# When enabled, process escaped characters such as \n and \" in strings in the
# pipeline configuration files.
#
# config.support_escapes: false
#
# ------------ API Settings -------------
# Define settings related to the HTTP API here.
#
# The HTTP API is enabled by default. It can be disabled, but features that rely
# on it will not work as intended.
#
# api.enabled: true
#
# By default, the HTTP API is not secured and is therefore bound to only the
# host's loopback interface, ensuring that it is not accessible to the rest of
# the network.
# When secured with SSL and Basic Auth, the API is bound to _all_ interfaces
# unless configured otherwise.
#
# api.http.host: 127.0.0.1
#
# The HTTP API web server will listen on an available port from the given range.
# Values can be specified as a single port (e.g., `9600`), or an inclusive range
# of ports (e.g., `9600-9700`).
#
# api.http.port: 9600-9700
#
# The HTTP API includes a customizable "environment" value in its response,
# which can be configured here.
#
# api.environment: "production"
#
# The HTTP API can be secured with SSL (TLS). To do so, you will need to provide
# the path to a password-protected keystore in p12 or jks format, along with credentials.
#
# api.ssl.enabled: false
# api.ssl.keystore.path: /path/to/keystore.jks
# api.ssl.keystore.password: "y0uRp4$$w0rD"
#
# The HTTP API can be configured to require authentication. Acceptable values are
# - `none`: no auth is required (default)
# - `basic`: clients must authenticate with HTTP Basic auth, as configured
# with `api.auth.basic.*` options below
# api.auth.type: none
#
# When configured with `api.auth.type` `basic`, you must provide the credentials
# that requests will be validated against. Usage of Environment or Keystore
# variable replacements is encouraged (such as the value `"${HTTP_PASS}"`, which
# resolves to the value stored in the keystore's `HTTP_PASS` variable if present
# or the same variable from the environment)
#
# api.auth.basic.username: "logstash-user"
# api.auth.basic.password: "s3cUreP4$$w0rD"
#
# When setting `api.auth.basic.password`, the password should meet
# the default password policy requirements.
# The default password policy requires non-empty minimum 8 char string that
# includes a digit, upper case letter and lower case letter.
# Policy mode sets Logstash to WARN or ERROR when HTTP authentication password doesn't
# meet the password policy requirements.
# The default is WARN. Setting to ERROR enforces stronger passwords (recommended).
#
# api.auth.basic.password_policy.mode: WARN
#
# ------------ Module Settings ---------------
# Define modules here. Modules definitions must be defined as an array.
# The simple way to see this is to prepend each `name` with a `-`, and keep
# all associated variables under the `name` they are associated with, and
# above the next, like this:
#
# modules:
# - name: MODULE_NAME
# var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE
# var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE
#
# Module variable names must be in the format of
#
# var.PLUGIN_TYPE.PLUGIN_NAME.KEY
#
# modules:
#
# ------------ Cloud Settings ---------------
# Define Elastic Cloud settings here.
# Format of cloud.id is a base64 value e.g. dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy
# and it may have an label prefix e.g. staging:dXMtZ...
# This will overwrite 'var.elasticsearch.hosts' and 'var.kibana.host'
# cloud.id: <identifier>
#
# Format of cloud.auth is: <user>:<pass>
# This is optional
# If supplied this will overwrite 'var.elasticsearch.username' and 'var.elasticsearch.password'
# If supplied this will overwrite 'var.kibana.username' and 'var.kibana.password'
# cloud.auth: elastic:<password>
#
# ------------ Queuing Settings --------------
#
# Internal queuing model, "memory" for legacy in-memory based queuing and
# "persisted" for disk-based acked queueing. Defaults is memory
#
# queue.type: memory
#
# If `queue.type: persisted`, the directory path where the pipeline data files will be stored.
# Each pipeline will group its PQ files in a subdirectory matching its `pipeline.id`.
# Default is path.data/queue.
#
# path.queue:
#
# If using queue.type: persisted, the page data files size. The queue data consists of
# append-only data files separated into pages. Default is 64mb
#
# queue.page_capacity: 64mb
#
# If using queue.type: persisted, the maximum number of unread events in the queue.
# Default is 0 (unlimited)
#
# queue.max_events: 0
#
# If using queue.type: persisted, the total capacity of the queue in number of bytes.
# If you would like more unacked events to be buffered in Logstash, you can increase the
# capacity using this setting. Please make sure your disk drive has capacity greater than
# the size specified here. If both max_bytes and max_events are specified, Logstash will pick
# whichever criteria is reached first
# Default is 1024mb or 1gb
#
# queue.max_bytes: 1024mb
#
# If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.acks: 1024
#
# If using queue.type: persisted, the maximum number of written events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.writes: 1024
#
# If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page
# Default is 1000, 0 for no periodic checkpoint.
#
# queue.checkpoint.interval: 1000
#
# ------------ Dead-Letter Queue Settings --------------
# Flag to turn on dead-letter queue.
#
# dead_letter_queue.enable: false
# If using dead_letter_queue.enable: true, the maximum size of each dead letter queue. Entries
# will be dropped if they would increase the size of the dead letter queue beyond this setting.
# Default is 1024mb
# dead_letter_queue.max_bytes: 1024mb
# If using dead_letter_queue.enable: true, the interval in milliseconds where if no further events eligible for the DLQ
# have been created, a dead letter queue file will be written. A low value here will mean that more, smaller, queue files
# may be written, while a larger value will introduce more latency between items being "written" to the dead letter queue, and
# being available to be read by the dead_letter_queue input when items are written infrequently.
# Default is 5000.
#
# dead_letter_queue.flush_interval: 5000
# If using dead_letter_queue.enable: true, controls which entries should be dropped to avoid exceeding the size limit.
# Set the value to `drop_newer` (default) to stop accepting new events that would push the DLQ size over the limit.
# Set the value to `drop_older` to remove queue pages containing the oldest events to make space for new ones.
#
# dead_letter_queue.storage_policy: drop_newer
# If using dead_letter_queue.enable: true, the interval that events have to be considered valid. After the interval has
# expired the events could be automatically deleted from the DLQ.
# The interval could be expressed in days, hours, minutes or seconds, using as postfix notation like 5d,
# to represent a five days interval.
# The available units are respectively d, h, m, s for day, hours, minutes and seconds.
# If not specified then the DLQ doesn't use any age policy for cleaning events.
#
# dead_letter_queue.retain.age: 1d
# If using dead_letter_queue.enable: true, defines the action to take when the dead_letter_queue.max_bytes is reached,
# could be "drop_newer" or "drop_older".
# With drop_newer, messages that were inserted most recently are dropped, logging an error line.
# With drop_older setting, the oldest messages are dropped as new ones are inserted.
# Default value is "drop_newer".
# dead_letter_queue.storage_policy: drop_newer
# If using dead_letter_queue.enable: true, the directory path where the data files will be stored.
# Default is path.data/dead_letter_queue
#
# path.dead_letter_queue:
#
# ------------ Debugging Settings --------------
#
# Options for log.level:
# * fatal
# * error
# * warn
# * info (default)
# * debug
# * trace
#
# log.level: info
# path.logs:
#
# ------------ Other Settings --------------
#
# Allow or block running Logstash as superuser (default: true)
# allow_superuser: false
#
# Where to find custom plugins
# path.plugins: []
#
# Flag to output log lines of each pipeline in its separate log file. Each log filename contains the pipeline.name
# Default is false
# pipeline.separate_logs: false
#
# ------------ X-Pack Settings (not applicable for OSS build)--------------
#
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
#xpack.monitoring.enabled: false
#xpack.monitoring.elasticsearch.username: logstash_system
#xpack.monitoring.elasticsearch.password: password
#xpack.monitoring.elasticsearch.proxy: ["http://proxy:port"]
#xpack.monitoring.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
# an alternative to hosts + username/password settings is to use cloud_id/cloud_auth
#xpack.monitoring.elasticsearch.cloud_id: monitoring_cluster_id:xxxxxxxxxx
#xpack.monitoring.elasticsearch.cloud_auth: logstash_system:password
# another authentication alternative is to use an Elasticsearch API key
#xpack.monitoring.elasticsearch.api_key: "id:api_key"
#xpack.monitoring.elasticsearch.ssl.certificate_authority: "/path/to/ca.crt"
#xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint: xxxxxxxxxx
#xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file
#xpack.monitoring.elasticsearch.ssl.truststore.password: password
#xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.monitoring.elasticsearch.ssl.keystore.password: password
#xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
#xpack.monitoring.elasticsearch.sniffing: false
#xpack.monitoring.collection.interval: 10s
#xpack.monitoring.collection.pipeline.details.enabled: true
#
# X-Pack Management
# https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html
#xpack.management.enabled: false
#xpack.management.pipeline.id: ["main", "apache_logs"]
#xpack.management.elasticsearch.username: logstash_admin_user
#xpack.management.elasticsearch.password: password
#xpack.management.elasticsearch.proxy: ["http://proxy:port"]
#xpack.management.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
# an alternative to hosts + username/password settings is to use cloud_id/cloud_auth
#xpack.management.elasticsearch.cloud_id: management_cluster_id:xxxxxxxxxx
#xpack.management.elasticsearch.cloud_auth: logstash_admin_user:password
# another authentication alternative is to use an Elasticsearch API key
#xpack.management.elasticsearch.api_key: "id:api_key"
#xpack.management.elasticsearch.ssl.ca_trusted_fingerprint: xxxxxxxxxx
#xpack.management.elasticsearch.ssl.certificate_authority: "/path/to/ca.crt"
#xpack.management.elasticsearch.ssl.truststore.path: /path/to/file
#xpack.management.elasticsearch.ssl.truststore.password: password
#xpack.management.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.management.elasticsearch.ssl.keystore.password: password
#xpack.management.elasticsearch.ssl.verification_mode: certificate
#xpack.management.elasticsearch.sniffing: false
#xpack.management.logstash.poll_interval: 5s
# X-Pack GeoIP plugin
# https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html#plugins-filters-geoip-manage_update
#xpack.geoip.download.endpoint: "https://geoip.elastic.co/v1/database"
Now I don't know how to save the queues that Lagstash reads from RabbitMQ in Elasticsearch and watch them as a chart in Kiabana??
Related
How to use two databases: Postgres and Snowflake with alembic and sqlachemy?
I want to use two databases: Postgres and Snowflake using alembic migration tool in a single fastapi app. Able to perform alembic migrations and alembic upgrade if using single database i.e. Postgres but on using multiple databases alembic is creating problems. Tried using snowflake database independently on a different app with only snowflake as a database. It is working perfectly fine with revisions and alembic upgrade but not working on a single app with two databases. Here is my directory structure as suggested in few articles: main-project--> postgres-migration--> versions/ env.py README script.py.mako snowflake-migration--> versions/ env.py README script.py.mako Here is my alembic.ini file generated for postgres, but I manipulated it to support snowflake: # A generic, single database configuration. [alembic] # path to migration scripts script_location = db-migration [A_SNOWFLAKE_SCHEMA] # path to env.py and migration scripts for schema1 script_location = snowflake-db-migration # template used to generate migration files # file_template = %%(rev)s_%%(slug)s # sys.path path, will be prepended to sys.path if present. # defaults to the current working directory. prepend_sys_path = . # timezone to use when rendering the date # within the migration file as well as the filename. # string value is passed to dateutil.tz.gettz() # leave blank for localtime # timezone = # max length of characters to apply to the # "slug" field # truncate_slug_length = 40 # set to 'true' to run the environment during # the 'revision' command, regardless of autogenerate # revision_environment = false # set to 'true' to allow .pyc and .pyo files without # a source .py file to be detected as revisions in the # versions/ directory # sourceless = false # version location specification; this defaults # to db-migration/versions. When using multiple version # directories, initial revisions must be specified with --version-path # version_locations = %(here)s/bar %(here)s/bat db-migration/versions # the output encoding used when revision files # are written from script.py.mako # output_encoding = utf-8 #sqlalchemy.url = driver://user:pass#localhost/dbname [post_write_hooks] # post_write_hooks defines scripts or Python functions that are run # on newly generated revision scripts. See the documentation for further # detail and examples # format using "black" - use the console_scripts runner, against the "black" entrypoint # hooks = black # black.type = console_scripts # black.entrypoint = black # black.options = -l 79 REVISION_SCRIPT_FILENAME # Logging configuration [loggers] keys = root,sqlalchemy,alembic [handlers] keys = console [formatters] keys = generic [logger_root] level = WARN handlers = console qualname = [logger_sqlalchemy] level = WARN handlers = qualname = sqlalchemy.engine [logger_alembic] level = INFO handlers = qualname = alembic [handler_console] class = StreamHandler args = (sys.stderr,) level = NOTSET formatter = generic [formatter_generic] format = %(levelname)-5.5s [%(name)s] %(message)s datefmt = %H:%M:%S Above file is able to create revision scripts inside versions/ folder of snowflake-db-migration. But not able to run the generated revision using(throwing errors): alembic upgrade head How to convert this .ini file to support multi-database .ini file? Tried using command: alembic init --template multidb snowflake-db-migration But not sure how to integrate changes here. Also don't want to hardcode sqlalchemy.url variable in .ini file as required to use snowflake variables from a .toml file which contains all environment variables. There are few answers but none of them are accepted and exactly similar to my use case.
Logstash parsing data from two different filebeat inputs
I have one machine on which I have set Elasticsearch and Logstash and shipping there logs via Filebeat from another machine. I'd like to add a new machine from which I could ship logs to Logstash, parse them and store in the same elasticsearch index. I tried to configurate filebeat on new machine with the same Logstash output but it seems logstash doesn't recieve data from multiple sources... The logstash config file: input { beats { port => 5044 } } filter { grok { match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \[%{NOTSPACE:referrer}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)"} } grok { match => { "referrer" => "%{WORD:protocol}://%{WORD:domain1}.%{WORD:domain2}.%{WORD:domain3}:%{INT:port}" } } geoip { source => "clientip" } } output { elasticsearch { hosts => ["localhost:9200"] index => "my_index" } } Filebeat config file # This file is an example configuration file highlighting only the most common # options. The filebeat.reference.yml file from the same directory contains all the # supported options with more comments. You can use it as a reference. # # You can find the full configuration reference here: # https://www.elastic.co/guide/en/beats/filebeat/index.html # For more available modules and options, please see the filebeat.reference.yml sample # configuration file. #=========================== Filebeat inputs ============================= filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: log # Change to true to enable this input configuration. #enabled: false enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: #- /var/log/*.log - /var/log/hostname/proxy1/app/nginx.log - /var/log/hostname/proxy2/app/nginx.log #- c:\programdata\elasticsearch\logs\* # Exclude lines. A list of regular expressions to match. It drops the lines that are # matching any regular expression from the list. #exclude_lines: ['^DBG'] # Include lines. A list of regular expressions to match. It exports the lines that are # matching any regular expression from the list. #include_lines: ['^ERR', '^WARN'] # Exclude files. A list of regular expressions to match. Filebeat drops the files that # are matching any regular expression from the list. By default, no files are dropped. #exclude_files: ['.gz$'] # Optional additional fields. These fields can be freely picked # to add additional information to the crawled log files for filtering #fields: # level: debug # review: 1 ### Multiline options # Multiline can be used for log messages spanning multiple lines. This is common # for Java Stack Traces or C-Line Continuation # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [ #multiline.pattern: ^\[ # Defines if the pattern set under pattern should be negated or not. Default is false. #multiline.negate: false # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern # that was (not) matched before or after or as long as a pattern is not matched based on negate. # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash #multiline.match: after #============================= Filebeat modules =============================== filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s #==================== Elasticsearch template setting ========================== setup.template.settings: index.number_of_shards: 1 #index.codec: best_compression #_source.enabled: false #================================ General ===================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. #name: # The tags of the shipper are included in their own field with each # transaction published. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add additional information to the # output. #fields: # env: staging #============================== Dashboards ===================================== # These settings control loading the sample dashboards to the Kibana index. Loading # the dashboards is disabled by default and can be enabled either by setting the # options here or by using the `setup` command. #setup.dashboards.enabled: false # The URL from where to download the dashboards archive. By default this URL # has a value which is computed based on the Beat name and version. For released # versions, this URL points to the dashboard archive on the artifacts.elastic.co # website. #setup.dashboards.url: #============================== Kibana ===================================== # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. # This requires a Kibana endpoint configuration. setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: http://localhost:5601/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 #host: "localhost:5601" # Kibana Space ID # ID of the Kibana Space into which the dashboards should be loaded. By default, # the Default Space will be used. #space.id: #============================= Elastic Cloud ================================== # These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/). # The cloud.id setting overwrites the `output.elasticsearch.hosts` and # `setup.kibana.host` options. # You can find the `cloud.id` in the Elastic Cloud web UI. #cloud.id: # The cloud.auth setting overwrites the `output.elasticsearch.username` and # `output.elasticsearch.password` settings. The format is `<user>:<pass>`. #cloud.auth: #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ #output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"] # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme" #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["logstash:5045"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" #================================ Processors ===================================== # Configure processors to enhance or manipulate events generated by the beat. processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: error, warning, info, debug #logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["*"]. Examples of other selectors are "beat", # "publish", "service". #logging.selectors: ["*"] Thanks for any suggestions!
You should edit output section in filebeat.yml like below: output.logstash: # The Logstash hosts hosts: ["Logstash_server_private_ip:5044"] Logstash expects data from port 5044, not from 5045.
Use logstash pipeline Note: If xpack basic security not enabled username and password not required of ES (remove those lines) in directory /etc/logstash/conf.d You can write multiple conf on different port gunicorn.log input { beats { port => "5044" } } output { # stdout { codec => rubydebug } elasticsearch { hosts => ["xx.xx.xx.xx:xxxx"] user => "" password => "*******" index => "gunicorn" } } access.log input { beats { port => "5047" } } output { # stdout { codec => rubydebug } elasticsearch { hosts => ["xxxxxxx:xxxx"] user => "*********" password => "*********" index => "access" } } in directory /etc/logstash -> pipelines.yml - pipeline.id: gunicorn path.config: "/etc/logstash/conf.d/gunicorn.conf" - pipeline.id: access path.config: "/etc/logstash/conf.d/access.conf" on machine - 1 in directory /etc/filebeat filebeat.yml filebeat.inputs: - type: log paths: - "/home/ubuntu/data/gunicorn.log" queue.mem: events: 8000 flush.min_events: 2000 flush.timeout: 10s output.logstash: hosts: ["logstash public IP:5044"] on machine - 2 in directory /etc/filebeat filebeat.yml filebeat.inputs: - type: log paths: - "/home/ubuntu/data/access.log" queue.mem: events: 8000 flush.min_events: 2000 flush.timeout: 10s output.logstash: hosts: ["logstash public IP:5047"]
Resend old logs from filebeat to logstash
Thanks in advance for your help. I would like to reload some logs to customize additional fields. I have noticed that registry file in filebeat configuration keeps track of the files already picked. However, if I remove the content in that file, I am not getting the old logs back. I have tried also to change the timestamp of the source in registry file with no sucsess. What changes are needed to sent old logs from filebeat to logstash? How can I get the logs back? Update: This is the last log in tomcat container: 2019-03-11 06:22:48 [Thread-4 ] DEBUG: ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB This is the log obtained by filebeat: 2019-03-14T16:18:50.377-0700 DEBUG [publish] pipeline/processor.go:308 Publish event: { "#timestamp": "2019-03-14T23:18:45.376Z", "#metadata": { "beat": "filebeat", "type": "doc", "version": "6.6.0" }, "host": { "name": "tomcat", "architecture": "x86_64", "os": { "codename": "Core", "platform": "centos", "version": "7 (Core)", "family": "redhat", "name": "CentOS Linux" }, "id": "6aaed308aa5a419f880c5e45eea65414", "containerized": true }, "source": "/app/logs/WEB/WEB-rest-api/WEB-rest-api.log", "log": { "file": { "path": "/app/logs/WEB/WEB-rest-api/WEB-rest-api.log" } }, "message": "2019-03-11 06:22:48 [Thread-4 ] DEBUG: ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB", "beat": { "name": "tomcat", "hostname": "tomcat", "version": "6.6.0" }, "offset": 6771071, "prospector": { "type": "log" }, "input": { "type": "log" }, "meta": { "cloud": { "instance_name": "tomcat", "machine_type": "Standard_D8s_v3", "region": "CanadaCentral", "provider": "az", "instance_id": "6452bcf4-7f5d-4fc3-9f8e-5ea57f00724b" } } } This is the log ingest by Logstash: [2019-03-15T10:32:25,982][DEBUG][logstash.outputs.gelf ] Sending GELF event {:event=>{"short_message"=>["2019-03-11 06:22:48 [Thread-4 ] DEBUG: ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB", " Connection cache monitor in thread: Thread-4 shutting down for pool: WEB"], "full_message"=>"2019-03-11 06:22:48 [Thread-4 ] DEBUG: ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB, Connection cache monitor in thread: Thread-4 shutting down for pool: WEB", "host"=>"{\"name\":\"tomcat\",\"os\":{\"name\":\"CentOS Linux\",\"version\":\"7 (Core)\",\"codename\":\"Core\"}}", "_source"=>"/app/logs/WEB/WEB-rest-api/WEB-rest-api.log", "_class"=>"ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor, %{JAVACLASS}", "_tags"=>"beats_input_codec_plain_applied", "_beat_hostname"=>"tomcat", "_beat_name"=>"tomcat", "_meta_cloud"=>{}, "_log_file"=>{"path"=>"/app/logs/WEB/WEB-rest-api/WEB-rest-api.log"}, "level"=>6}} Filebeat.yml: ###################### Filebeat Configuration Example ######################### # This file is an example configuration file highlighting only the most common # options. The filebeat.reference.yml file from the same directory contains all the # supported options with more comments. You can use it as a reference. # # You can find the full configuration reference here: # https://www.elastic.co/guide/en/beats/filebeat/index.html # For more available modules and options, please see the filebeat.reference.yml sample # configuration file. #=========================== Filebeat inputs ============================= filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /apps/logs/WEB/web-api/web-api.log - /apps/logs/WEB/web-api/web-rest-api.log # Exclude lines. A list of regular expressions to match. It drops the lines that are # matching any regular expression from the list. #exclude_lines: ['^DBG'] # Include lines. A list of regular expressions to match. It exports the lines that are # matching any regular expression from the list. #include_lines: ['^ERR', '^WARN'] # Exclude files. A list of regular expressions to match. Filebeat drops the files that # are matching any regular expression from the list. By default, no files are dropped. #exclude_files: ['.gz$'] # Optional additional fields. These fields can be freely picked # to add additional information to the crawled log files for filtering #fields: # level: debug # review: 1 # Ignore files which were modified more then the defined timespan in the past # Time strings like 2h (2 hours), 5m (5 minutes) can be used. ignore_older: 0 ### Multiline options # Multiline can be used for log messages spanning multiple lines. This is common # for Java Stack Traces or C-Line Continuation # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [ multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' # Defines if the pattern set under pattern should be negated or not. Default is false. multiline.negate: true # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern # that was (not) matched before or after or as long as a pattern is not matched based on negate. # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash multiline.match: after #============================= Filebeat modules =============================== filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s #==================== Elasticsearch template setting ========================== setup.template.settings: index.number_of_shards: 3 #index.codec: best_compression #_source.enabled: false #================================ General ===================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. #name: # The tags of the shipper are included in their own field with each # transaction published. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add additional information to the # output. #fields: # env: staging #============================== Dashboards ===================================== # These settings control loading the sample dashboards to the Kibana index. Loading # the dashboards is disabled by default and can be enabled either by setting the # options here, or by using the `-setup` CLI flag or the `setup` command. #setup.dashboards.enabled: false # The URL from where to download the dashboards archive. By default this URL # has a value which is computed based on the Beat name and version. For released # versions, this URL points to the dashboard archive on the artifacts.elastic.co # website. #setup.dashboards.url: #============================== Kibana ===================================== # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. # This requires a Kibana endpoint configuration. #setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: http://localhost:5601/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 #host: "localhost:5601" # Kibana Space ID # ID of the Kibana Space into which the dashboards should be loaded. By default, # the Default Space will be used. #space.id: #============================= Elastic Cloud ================================== # These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/). # The cloud.id setting overwrites the `output.elasticsearch.hosts` and # `setup.kibana.host` options. # You can find the `cloud.id` in the Elastic Cloud web UI. #cloud.id: # The cloud.auth setting overwrites the `output.elasticsearch.username` and # `output.elasticsearch.password` settings. The format is `<user>:<pass>`. #cloud.auth: #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ #output.elasticsearch: # Array of hosts to connect to. #hosts: ["log1.cgi-dev.ca:9200"] # Enabled ilm (beta) to use index lifecycle management instead daily indices. #ilm.enabled: false # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme" #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["log1.cgi-dev.ca:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash.crt"] # Certificate for SSL client authentication ##ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key ##ssl.key: "/etc/pki/client/cert.key" #================================ Processors ===================================== # Configure processors to enhance or manipulate events generated by the beat. processors: - add_host_metadata: ~ - add_cloud_metadata: ~ #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: error, warning, info, debug logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["*"]. Examples of other selectors are "beat", # "publish", "service". #logging.selectors: ["*"] #============================== Xpack Monitoring =============================== # filebeat can export internal metrics to a central Elasticsearch monitoring # cluster. This requires xpack monitoring to be enabled in Elasticsearch. The # reporting is disabled by default. # Set to true to enable the monitoring reporter. #xpack.monitoring.enabled: false # Uncomment to send the metrics to Elasticsearch. Most settings from the # Elasticsearch output are accepted here as well. Any setting that is not set is # automatically inherited from the Elasticsearch output configuration, so if you # have the Elasticsearch output configured, you can simply uncomment the # following line. #xpack.monitoring.elasticsearch: However, I do not get the log neither Kibana nor Graylog. It is worth noting that the same kind of logs for INFO level related to the same class are visible in Kibana and Graylog, but not the ones with DEBUG level. Do you know what would be wrong? Thanks a lot
Stop filebeat and logstash. Clear old data from Elasticsearch if there. Delete registry files registry and registry.old. Run logstash. Run filebeat using command filebeat -e -once.
The registry keeps the inode and byte offset. Removing the content doesn't change the inode. Try shutting down filebeat and removing/resetting the byte offset in the registry.
How to setup syslog in yocto?
I like to configure syslog. It seems that are more than one way to set up syslog. I am asking for the common way/steps to do that. I have several use cases. To simplify I like to ask how to configure syslog to write an infinity long log file in /var/log/. Following steps: 1.) configure what messages 1.1) create own "syslog.conf" (define /var/log/myLog) 1.2) append it to "recipes-core/busybox" 2.) configure how to log ?? I found two possible places to do that: #meta-poky -> "meta-poky/recipes-core/busybox/busybox/poky-tiny/defconfig" # # System Logging Utilities # CONFIG_SYSLOGD=y CONFIG_FEATURE_ROTATE_LOGFILE=y CONFIG_FEATURE_REMOTE_LOG=y CONFIG_FEATURE_SYSLOGD_DUP=y CONFIG_FEATURE_SYSLOGD_CFG=y CONFIG_FEATURE_SYSLOGD_READ_BUFFER_SIZE=256 CONFIG_FEATURE_IPC_SYSLOG=y CONFIG_FEATURE_IPC_SYSLOG_BUFFER_SIZE=16 CONFIG_LOGREAD=y CONFIG_FEATURE_LOGREAD_REDUCED_LOCKING=y CONFIG_KLOGD=y CONFIG_FEATURE_KLOGD_KLOGCTL=y CONFIG_LOGGER=y add/change: "CONFIG_FEATURE_ROTATE_LOGFILE=n" by adding that line to meta-mylayer/conf/layer.conf" etc. ??? # "/etc/syslog-startup.conf" # This configuration file is used by the busybox syslog init script, # /etc/init.d/syslog[.busybox] to set syslog configuration at start time. DESTINATION=file # log destinations (buffer file remote) LOGFILE=/var/log/messages # where to log (file) REMOTE=loghost:514 # where to log (syslog remote) REDUCE=no # reduce-size logging DROPDUPLICATES=no # whether to drop duplicate log entries #ROTATESIZE=0 # rotate log if grown beyond X [kByte] #ROTATEGENS=3 # keep X generations of rotated logs BUFFERSIZE=64 # size of circular buffer [kByte] FOREGROUND=no # run in foreground (don't use!) #LOGLEVEL=5 # local log level (between 1 and 8) In the systemV init script "/etc/init.d/syslog.bussybox" the file "/etc/syslog-startup.con" is read and used for configuration. System behaviour: When running my system, the log wraps when the logfile reaches 200kBytes. One logfile + one log-rotate file is generated. Any ideas how to archive that syslog writes an infinite long log-file? I am working on the Yocto krogoth branch + meta-atmel / meta_openembedded (# krogoth too).
By checking the sources of syslog and busybox I found a possible solution. This solution shows how to configure syslog to log in two logs with max 10MByte: 1.) get valid syslog build config 1.1) download busybox -> git/busybox 1.2) build busybox via bitbake -> bitbake busybox 1.3) copy defconfig file to downloaded busybox -> cp /defconfig git/busybox/ 1.4) make menueconfig 1.5) goto "System Logging Utilities" 1.6) deselect klogd because it can colide with printk 1.7) save to "defconfig" # # System Logging Utilities # # CONFIG_KLOGD is not set # CONFIG_FEATURE_KLOGD_KLOGCTL is not set CONFIG_LOGGER=y CONFIG_LOGREAD=y CONFIG_FEATURE_LOGREAD_REDUCED_LOCKING=y CONFIG_SYSLOGD=y CONFIG_FEATURE_ROTATE_LOGFILE=y CONFIG_FEATURE_REMOTE_LOG=y CONFIG_FEATURE_SYSLOGD_DUP=y CONFIG_FEATURE_SYSLOGD_CFG=y CONFIG_FEATURE_SYSLOGD_READ_BUFFER_SIZE=256 CONFIG_FEATURE_IPC_SYSLOG=y CONFIG_FEATURE_IPC_SYSLOG_BUFFER_SIZE=64 CONFIG_FEATURE_KMSG_SYSLOG=y 2.) setup your log config Create "syslog.conf" and enter the rules: This is an example: # # /etc/syslog.conf Configuration file for busybox's syslogd utility # kern.notice /var/log/messages # # my software messages # user.err /var/log/mySWError user.* /var/log/mySWFull local0.* /var/log/mySWFull local0.err /var/log/mySWError # #this prevents from logging to default log file (-O FILE or /var/log/messages) # *.* /dev/null 3.) modify configuration for the busybox syslog deamon This example logs to files which are limited to 10 MBytes. If "ROTATESIZE" is not set syslog set the log filesize automatic to 200 kBytes. The content of "syslog-startup.conf" looks like: # This configuration file is used by the busybox syslog init script, # /etc/init.d/syslog[.busybox] to set syslog configuration at start time. DESTINATION=file # log destinations (buffer file remote) #LOGFILE=/var/log/messages # where to log (file) REMOTE=loghost:514 # where to log (syslog remote) REDUCE=no # reduce-size logging DROPDUPLICATES=no # whether to drop duplicate log entries ROTATESIZE=10000 # rotate log if grown beyond X [kByte] #ROTATEGENS=3 # keep X generations of rotated logs BUFFERSIZE=64 # size of circular buffer [kByte] FOREGROUND=no # run in foreground (don't use!) #LOGLEVEL=5 # local log level (between 1 and 8) 4.) get the configuration into yocto build 4.1) create following directory structure in your own layer(meta-custom): meta-custom/recipes-core/ meta-custom/recipes-core/busybox/ meta-custom/recipes-core/busybox/busybox 4.2) copy into "meta-custom/recipes-core/busybox/busybox": defconfig syslog.conf syslog-startup.conf 4.3) create in "meta-custom/recipes-core/busybox/" "busybox_1.24.1.bbappend". If you using an older/newer version of busybox you need to change the "1.24.1" number to yours. You can find your version in "/poky/meta/recipes-core/busybox/" Add this two lines to this file: FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:" FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}/poky-tiny:" 5.) build your custom defined syslog bitbake busybox and transfer it into the rootfs of the image bitbake core-image-minimal Now it should work!
Adding to the Stefan Jaritz comment, you can skip the download step by just issuing bitbake busybox -c devshell Then running make menuconfig and getting the new config file from it. Note that this will use Yocto's defconfig by default, so you don't need to do worry about "what's this default config".
Filebeat on Windows 2012 R2
I am using Filebeat > Logstash > Elasticsearch > Kibana to parse and analyse logs basically Java Stack Trace and other Logs. Here is YML for Filebeat filebeat: prospectors: - paths: - C:\logs\OCR\example.log input_type: log #document_type: UAT_EXAMPLE exclude_lines: [".+DEBUG"] multiline: pattern: ".+(ERROR|INFO)" negate: true match: after fields: app_name: EXAMPLE_APP environment: UAT fields_under_root: true #force_close_files: true spool_size: 2048 #publish_async: true #scan_frequency: 10s #close_older: 2h output: logstash: host: "10.0.64.14" port: 5044 index: filebeat timeout: 5 reconnect_interval: 3 bulk_max_size: 2048 shipper: tags: ["ABC_Engine", "UAT_EXAMPLE"] queue_size: 1000 ### Enable logging of the filebeat logging: level: warning to_files: true files: path: c:\logs\ name: mybeat.log rotateeverybytes: 20485760 # = 20MB keepfiles: 7 Enable logging of the filebeat is also not working on windows. Let me know if I am missing anything here. logging: level: warning to_files: true files: path: c:\logs\ name: mybeat.log rotateeverybytes: 20485760 # = 20MB keepfiles: 7 Problem - the Filebeat is not able to send logs to logstash at times, some times it start running shipping but sometimes it doesn't. Although If I use "test.log" as a prospector and save logs locally on disk via below config it works well. Writing Files to local File to Check the output. I have tried "file" output and "logstash" output one by one. output: file: path: c:\logs\ filename: filebeat rotate_every_kb: 100000 number_of_files: 7 Also, The things mostly run when I am using command Line: .\filebeat.exe -c filebeat.yml -e -v Kindly assist with the correct config for Windows. The log file "example.log" is getting rotated on every 30 MB of size. I am not very sure to use the below attributes and how they will function with Filebeat on Windows. "close_older" "ignore_older" "Logging"
output to logstash : comment elasticsearch line then logstash: # The Logstash hosts hosts: ["localhost:5044"] keep [] and config for log in debug mode for example logging: # Send all logging output to syslog. On Windows default is false, otherwise # default is true. #to_syslog: true # Write all logging output to files. Beats automatically rotate files if rotateeverybytes # limit is reached. #to_files: false # To enable logging to files, to_files option has to be set to true files: # The directory where the log files will written to. #path: /var/log/mybeat path: c:\PROGRA~1/filebeat # The name of the files where the logs are written to. name: filebeat.log # Configure log file size limit. If limit is reached, log file will be # automatically rotated rotateeverybytes: 10485760 # = 10MB # Number of rotated log files to keep. Oldest files will be deleted first. #keepfiles: 7 # Enable debug output for selected components. To enable all selectors use ["*"] # Other available selectors are beat, publish, service # Multiple selectors can be chained. #selectors: [ ] # Sets log level. The default log level is error. # Available log levels are: critical, error, warning, info, debug level: debug LOGGING is in LOGGING part, output is logstash or elastic search, if you want know you can install as service go to the elastic.co website : https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html