I have a test app with heroku with the logentries add on.
I was wondering if it was possible to filter out some types of logs from the live tail, in my case all logs with the terms "heroku router"
Simply negate your search terms. For the case you've specified, the following works:
-"heroku router"
Related
this is my first ever post on stackoverflow
Im sending json logs from filebeat to logstash to opensearch to grafana
and everything is working perfectly (if it comes to integer data)
i can even see that opensearch receives my string fields and boolean fields and even reads them.
but when i want to make a dashboard to visualize some strings and booleans, it only finds my integer fields
Can someone help me visualize Strings on grafana and not only numbers.
this is an image of what i can see when i try to select data, i only see the number field names
thanks andrew, now i see this, but i want to only see 1 field
and not all of them
logs added to grafana
You can try using the Logs panel
And an example of how I use - the request is something like this:
{namespace=~"$namespace", stream=~"$stream", container =~"$container"} |= "$query"
But I'm using fluent-bit + loki
I'm missing some index patterns in Kibana and I've been trying to figure out why this is the case. I have installed logstash, elasticsearch and kibana and started the services. How do I get logstash, apache-access etc to show in this section? Only filebeat shows.
I've used the CURL command for the localhost and port to see the indices and only kibana and filebeat are shown there are and apache-access and logstash are no where to be seen.
Can anyone guide me in the right direction to resolving this and being able to see 'logstash' and 'apache-access' under the patterns section.
Data is being saved inside indices in Elasticsearch cluster, in Kibana you can define index-patterns to show multiple indices at the same time.
When you look in the left menu of your screenshot you'll find a menu item called "Index Management", all indices will be shown there, here you'll find the name of the indices that exist in your Elasticsearch cluster.
An index pattern in Kibana is just a (wildcarded) pattern to allow you to see the data.
On the top right of your screenshot you see the button "+ Create Index Pattern", by clicking there you can define a new pattern which will live next to the existing one (filebeat-*).
Once you defined a second one, you'll be able to define which one is the default one chosen when you open Kibana and a dropdown will be available on your discover page in Kibana with the active index-pattern for your discovery at that time.
tash
So in short, press the "create index pattern" button twice entering once logstash* as the pattern and once apache-access* as pattern.
I was searching some query on Shodan and was getting lots of Honeypots and i wanted to filter out all the result which are honeypot, i tried adding -honeypot but still it showed results with honeypot, what am i doing wrong? Help will be appreciated. Thanks
To remove honeypots from the search results you can add -tag:honeypot to the search query. Note that this requires a Corporate subscription.
If you just add -honeypot then you are excluding results where the banner's data property doesn't include the word honeypot. For more information about the search query syntax please visit:
https://help.shodan.io/the-basics/search-query-fundamentals
You need to use search filters to narrow results based on a specific property. A list of available filters is available at:
https://beta.shodan.io/search/filters
You do not need pay for an Enterprise subscription. Make this filter in shodan web:
Add the filter after your pattern search -"792/71644"
In the API terminal type this:
shodan search <your pattern search> | egrep -v "792/71644"
obs: this number is in all honeypot expressions and it is removed with this filter.
I hope aid by by *:
We have an ELK setup and the Logstash is receiving all the logs from the Filebeat installed on the server. So when I open Kibana and it asks for an index I put just a * for the index value and go to the Discover tab to check the logs and it shows each line of the log in a separate expandable section.
I want to be able to group the logs based on the timestamp first and then on a common ID that is generated in our logs per request to identify it from the rest. An example of the logs we get :
DEBUG [2018-11-23 11:28:22,847][298b364850d8] Some information
INFO [2018-11-23 11:27:33,152][298b364850d8] Some information
INFO [2018-11-24 11:31:20,407][b66a88287eeb] Some information
DEBUG [2018-11-23 11:31:20,407][b66a88287eeb] Some information
I would like to see all logs for request ID : 298b364850d8 in the same drop down given they are continuous logs. Then it can break into the second dropdown again grouped by the request ID : b66a88287eeb in the order of timestamp.
Is this even possible or am I expecting too much from the tool?
OR if there is a better strategy to grouping of logs I'm more than happy to listen to suggestions.
I have been told by a friend that I could configure this in logstash to group logs based on some regex n stuff but I just don't know where and how to configure it to fo the grouping.
I am completely new to the whole ELK stack to bear with my questions which might be quite elementary in nature.
Your question is truly a little vague and broad as you say. However, I will try to help :)
Check the index that you define in the logstash output. This is the index that need to be defined Kibana - not *.
Create an Index Pattern to Connect to Elasticsearch. This will parse the fields of the logs and will allow you to filter as you want.
It recommend using a GUI tool (like Cerebro) to better understand what is going on in you ES. It would also help you to get better clue of the indices you have there.
Good Luck
You can use #timeStamp filter and search query as below sample image to filter what you want.
Purpose: Track privileged user activity in Windows logs.
Logic:
If logon event contains token_elev %1937 or %1938 save the logon_id (hex value) to a dynamic priv_logons list.
For subsequent events, if the logon_id in the event matches one of the entries in priv_logons list, add a 'privileged' tag to the event.
When receiving a logoff event with one of the logon_ids saved in priv_logons list, remove it from the list.
Is this doable in LogStash? If yes, how?
Not with Logstash alone.
Logstash does not maintain internal states or data objects in between events, it is simply a parsing engine.
Logstash may help you create this kind of solution by doing the parsing work and then passing clean and sensible data to a program which performs the logic you are looking for.