Clearing logstash memory - logstash

Question

Please explain me how logstash works with memory and events. I understand that when an event occurs, it is written to elasticsearch (in my case) and after that it should be cleaned from memory by the garbage collector.
But in debug mode, I see in the logs all the entries that went to elasticsearch and I don’t see them being cleaned out.
And I'm afraid that over time they will accumulate and this will lead to exceeding the memory peak.
On my volume of transmitted data, I still do not see a strong change in memory consumption, but I want to understand how to do it right.
Accordingly, the question is whether it is necessary to forcefully clean up the events so that they do not clog the memory?
If so, how to do it?
logstash.conf
input {
jdbc {
jdbc_connection_string => "jdbc:postgresql://<server>:5432/<DB>"
jdbc_user => "user"
jdbc_password => "pass"
jdbc_driver_class => "org.postgresql.Driver"
tracking_column => "unix_ts_in_secs"
schedule => "*/3 * * * * *"
statement => "SELECT uuid, header, location, price, parse_date AS unix_ts_in_secs FROM pdf_store WHERE (parse_date > :sql_last_value AND parse_date < NOW()) ORDER BY parse_date;"
}
}
output {
stdout { codec => json_lines }
elasticsearch {
hosts => ["http://localhost:9200"]
document_id => "%{uuid}"
index => "test_index7"
}
}
pipelines.yml
- pipeline.id: pipeline_test
pipeline.workers: 1
pipeline.batch.size: 1
pipeline.batch.delay: 1
path.config: "/config/logstash.conf"

Answer 1

According to Elastic recommandation you have to check the JVM heap:
Be aware of the fact that Logstash runs on the Java VM. This means that Logstash will always use the maximum amount of memory you allocate to it.
Look for other applications that use large amounts of memory and may be causing Logstash to swap to disk. This can happen if the total memory used by applications exceeds physical memory.
The recommended heap size for typical ingestion scenarios should be no less than 4GB and no more than 8GB.
CPU utilization can increase unnecessarily if the heap size is too low, resulting in the JVM constantly garbage collecting. You can check for this issue by doubling the heap size to see if performance improves.
Do not increase the heap size past the amount of physical memory. Some memory must be left to run the OS and other processes. As a general guideline for most installations, don’t exceed 50-75% of physical memory. The more memory you have, the higher percentage you can use.
Set the minimum (Xms) and maximum (Xmx) heap allocation size to the same value to prevent the heap from resizing at runtime, which is a very costly process.
Link can help you : https://www.elastic.co/guide/en/logstash/master/performance-troubleshooting.html