Is there a way to do a search in graylog2 and only return distinct results for a given field? What I'm trying to accomplish is a stream or dashboard widget that will give me the number of IIS errors from distinct ip addresses.
One way is to
extract the ip address as a field (ip_address),
search for IIS errors
In the search results message fields, on the right, use ip_address "quick values" to get the count of errors for a specific ip address
the approach to see the distribution of count-of-errors per ip via "quick values" is good. For a hint how to go about uniqueness/distinct ip addresses or any value, "card()" for cardinality can be used in graylog. But aggregation can be done in other ways.
To get the number of errors-per-ip in a Data Table, just add the rows and do a count() over them and restrict the search query to http_response:>=500
If it's a field, then "IpAddress:111.111.111.1" would search for all logs with 111.111.111.1 as the ip address.
Related
I was told that in order for my DNS query to work, I have to convert the domain name.
i.e. rit.edu -> rit3edu ; www.google.com to 3www6google3com
I cannot find where in the RFC https://www.ietf.org/rfc/rfc1035.txt that this is discussed, as I even tried performing a find. In my packets that I send to the DNS server. I keep the '.' domain name, and I am getting 'unknown extended label'. I cannot find any helpful threads either. I found a function on GitHub, but there is no discussion on the format. Does anyone have experience with this?
Thanks in advance
You may refer to this:
http://www.keyboardbanger.com/dns-message-format-name-compression/#Data_label
And in RFC1035 you attached:
4.1.2. Question section format
QNAME
a domain name represented as a sequence of labels, where
each label consists of a length octet followed by that
number of octets. The domain name terminates with the
zero length octet for the null label of the root....
In a DNS query, each label of the domain name should be preceded by a number that indicates length of the label. So if facebook.com is the question, the actual qname in the DNS query should be 8facebook3com.
I would like to list all/any DNS records including the DANE TLSA.
With
dig mailbox.org ANY
I get all records including DNSSEC etc. but nothing about DANE. Why?
With
dig _443._tcp.mailbox.org. ANY
I get the DANE TLSA records.
I've read the question where someone wants to query all subdomains
How can I list ALL DNS records?
and am aware that this is only possible with a zone transfer.
But '_443._tcp.' isn't a real subdomain, is it? I thought it is just an SRV record. So how can I query ANYthing including DANE TLSA?
The command dig mailbox.org ANY asks for all records for the name mailbox.org..
The command dig _443._tcp.mailbox.org. ANY asks for all records for the name _443._tcp.mailbox.org..
mailbox.org. is not the same name as _443._tcp.mailbox.org..
Asking for all the records for one of them will not show any records for the other one. If it helps, you can think of (fully qualified) names in DNS as primary keys in a database (because that is in practice exactly what they are). If you ask the database for data for the key FOO it will not give you any data for the key FOOBAR (unless it is very badly broken). Exactly the same thing is happening here. You ask for one thing, and you do not get answers for another, different, thing.
You'd find the answer in Section 3 of RFC 6698:
TLSA resource records are stored at a prefixed DNS domain name. The prefix is prepared in the following manner:
The decimal representation of the port number on which a TLS-based service is assumed to exist is prepended with an underscore character ("_") to become the left-most label in the prepared domain name. This number has no leading zeros.
The protocol name of the transport on which a TLS-based service is assumed to exist is prepended with an underscore character ("_") to become the second left-most label in the prepared domain name. The transport names defined for this protocol are "tcp", "udp", and "sctp".
The base domain name is appended to the result of step 2 to complete the prepared domain name. The base domain name is the fully qualified DNS domain name [RFC1035] of the TLS server, with the additional restriction that every label MUST meet the rules of [RFC0952]. The latter restriction means that, if the query is for an internationalized domain name, it MUST use the A-label form as defined in [RFC5890].
Basically since you can have different "TLS-Based service" (e.g., DTLS) on different ports and this data is not included in the TLSA record set, the naming convention is there to find the correct information for the desired protocol/port combination.
I'm using OpenSearchServer to provide search functionality on a web site. I want to crawl all pages on the site for links to follow but I want to exclude some pages from the index. I can't work out how to do this.
Specifically the website includes a shop that has its own product search and I am keeping this search for products and categories. The product pages have URLs like http://www.thesite/p/123 so I don't want to include any page like this in the search results. However some product pages reference background info pages and I want these to be included in the search index.
The problem I have is that the filter has no effect on the results - it doesn't filter out the /p/ and /c/ results. If I change the filter by unticking the negative box I get no results so it seems to be either the contents of the field or the filter criteria that is causing the problem.
I've tried adding a negative filter to the default query called search in the Query > Filter tab on the index with url:"http://www.thesite/p/*"
but it seems that wildcards are not supported for query filters although they are supported for Crawler > Exclusion list filters.
I've tried adding a new field called urlField in Schema > Fields and populating it using an analyzer configured using the Whitespace Tokenizer and a regular expression (http://www.thesite/(c|p)/). When I use the Test button it seems to generate two tokens for my test URL http://www.thesite/p/123:
http://www.thesite/p/
p
I'd hoped to be able to use the first one in a Query > Filter to exclude all the shop results and optionally be able to use the p (for product) or c (for category) if I need to search the product pages sometime in the future.
The urlShop field in the schema is set up as follows:
Indexed: yes
Stored: no (because I don't need the field back, just want to be able to filter on it)
TermVector: No
Analyzer: urlShop
Copy of: url
I've added urlFilter:"http://www.thesite/p/" to Query > Filters with the negative box ticked.
This seems to have no effect on the results when I use the default renderer.
To see whether it affects the returned results I unticked the negative box in the query filter I get no results in the default renderer. This leads me to believe that the urlShop field is not being populated but I'm not sure how to check this directly.
I would like to know whether there is an easier way to do this but if my approach makes sense in the context of OpenSearchServer please can you help me identify what's wrong?
The website is running under IIS and OpenSearchServer will be configured on the same server running in Tomcat.
Finally figured this out...
Go to query and hit edit for your configured query. Then go to the filters tab. Add a query filter like this:
urlExact:"http://myurltoexclude*"
Check the "negative" box. Click add.
Now make sure to click "save in the tiny little button on the right hand side. This is the part I missed. The URLS are still in the DB and crawl, but at least they aren't returned in results.
I have 2 fields that looks for username, the gets the email address and phone number from user's person document.
Creator_Email: #NameLookup([NoUpdate]; #UserName; "OfficePhoneNumber")
Creator_Ext: #NameLookup([NoUpdate]; #UserName; "OfficePhoneNumber")
The problem is there is one user reported that the extension did not pull out. It came out blank
I have checked on the person document and the phone number is there
the email address pull out correctly, but when I tested changing the first letter of first and last name to lower case (ie. Test.User#domain.com to test.user#domain.com), the field that pulls the email address still show up with Upper case.
I have tried to take the user.id and test it on a different PC and the problem persists.
Any idea why this happens? I know there are 2 address books set up in the company and that is not ideal but I have checked the 2 address books and make sure all the needed information are there.
The #NameLookup formula does a look up to a hidden view on the database, and sometimes you run into a situation where the index for that view isn't up-to-date. You can go to the database and press CTRL+SHIFT+F9 to rebuild all the views, or you can try to use the FORCEUPDATE flag in your formula.
I think you'll find that the #NameLookup call is finding people who have created entries for themselves in their personal address books. If those entries are incomplete, the lookups will fail to find the missing fields.
The #NameLookup will use the parameter...
Go to /File/Preferences/Location then is the current location find the Server Tab and check that you defined Domino Directory server (if empty user search on local)
Check also in the Mail tab, Recipient Name lookup that could "stop after first march" or exhaustively search ALL the names known on the client.
In #NameLookup you can also use [NoUpdate]:[Exhaustive]
be aware as mention before that view may be not up to date, that caching can occurs in the #NameLookup.
In place of this, I suggest to use:
#DbLookup("":"";YourServer:"names.nsf" ; "($VIMPeople)" ; #Name([ABBREVIATE] ;#UserName) ; "OfficePhoneNumber");
I'm storing information about local "events". They are described by 3 things - address, date, keywords(tags). I want to have only one search box for at least address and keywords. The date might go to a separate field. I'm assuming that most people will search for events that are taking place "today" so this filter won't get that much traffic.
I need those addresses to be correct (because I'm geocoding them afterwards) so I need to validate them before submitting the form and display a list of "did you mean" if a user made a typo there. I can't do life search here. I can do a live search on keywords. Keep in mind that a user can make a typo there too and I want to catch that.
Is there a clever way to design the input's parser in this case to guess which is supposed to be address and which keywords?
OR
Is there a way to actually parse it as user is entering his query? Maybe I should show autocomplete hints for keywords, after 3 first characters are entered, and if user denies to use them then to assume that it's a part of an address he's typing.
What do You think?
Take a look at Document Cloud's Visual search
http://documentcloud.github.com/visualsearch/#demo