I am developing a dashboard, which receives all Alertmanager readings and processes them. I looked for a unique field in the request payload to create a unique external alert id in my database. The request payload looks something like this:
{
"status": "firing",
"labels": {
"alertname": "",
"app": "",
"cluster": "",
"deployed_location": "",
"instance": "",
"job": "",
"kubernetes_namespace": "",
"kubernetes_pod_name": "",
"pod_template_hash": "",
"release": "",
"replica": "",
"severity": ""
},
"annotations": {
"description": "",
"summary": ""
},
"startsAt": "",
"endsAt": "",
"generatorURL": "",
"fingerprint": ""
}
I first used the generatorURL field, but then realized it many different alerts have the same value for generatorURL. I have been trying fingerprint, and the situation is much better. However, I am having instances where 2 to 15 alerts have the same fingerprint.
I am wondering:
Is there really no unique field in Alertmanager requests?
It is the nature of Alertmanager logic (or that of my alerts) that a number of alerts are created with the same fingerprint and I should just deal with it and handle it on my side, i.e. not create an incident in my DB if the given fingerprint is already used. I also worry that if I set unique=True on my alert model, some new alerts that have the same fingerprint will be missed...
If you jump to the alert.Fingerprint() definition, like this one, can find the impl of fingerprint
So, alert.Fingerprint() is just unique for labels
// labelSetToFingerprint works exactly as LabelsToSignature but takes a LabelSet as
// parameter (rather than a label map) and returns a Fingerprint.
func labelSetToFingerprint(ls LabelSet) Fingerprint {
if len(ls) == 0 {
return Fingerprint(emptyLabelSignature)
}
labelNames := make(LabelNames, 0, len(ls))
for labelName := range ls {
labelNames = append(labelNames, labelName)
}
sort.Sort(labelNames)
sum := hashNew()
for _, labelName := range labelNames {
sum = hashAdd(sum, string(labelName))
sum = hashAddByte(sum, SeparatorByte)
sum = hashAdd(sum, string(ls[labelName]))
sum = hashAddByte(sum, SeparatorByte)
}
return Fingerprint(sum)
}
Related
Below is the server response in a JSON format which I need to work on, In this JSON some of the objects I need to pass in the next request, which I am successfully able to do for the first occurrence, but the problem is coming on randomization in the below JSON
{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sword of Honour",
"price": 8.95
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "test_title",
"price": 12.99
},
{
"category": "fiction",
"author": "Herman Melville",
"title": "Sword of Honour",
"isbn": "0-553-21311-3",
"price": 8.99
},
{
"category": "fiction",
"author": "J. R. R. Tolkien",
"title": "India",
"isbn": "0-395-19395-8",
"price": 22.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
},
"expensive": 10
}
if I apply $..book[0][?(#.title == 'Sword of Honour')] condition I am seeing successful output
test_1={"category":"reference","title":"Sword of Honour","author":"Nigel Rees","price":8.95}
test_2={"category":"fiction","title":"Sword of Honour","author":"Herman Melville","price":8.99,"isbn":"0-553-21311-3"}
if I apply $..book[0][?(#.title == 'Sword of Honour')].author condition I am seeing successful output
test_1=Nigel Rees
test_2=Herman Melville
But I want to extract both author's name and price for the book that has the title == 'Sword of Honour'
Actual output = able to retrieve one value with the condition $..book[0][?(#.title == 'Sword of Honour')].author
Expected output = I want to receive two values from the same list randomly. and it should look like this
Iteration 1-
test_1={"author":"Nigel Rees","price":8.95}
Iteration 2-
test_2={"author":"Herman Melville","price":8.99,}
Iteration 3-
test_2={"author":"Herman Melville","price":8.99,}
I had tried with the V function as well but got only one value. (book{__V([?(#.title == 'Sword of Honour')]_${__Random(1,${title_matchNr},)},)})
Add JSR223 PostProcessor as a child of the request which returns the above JSON
Put the following code into "Script" area:
def swords = new groovy.json.JsonSlurper().parse(prev.getResponseData()).store.book.findAll { book -> book.title == 'Sword of Honour' }
swords.eachWithIndex { def sword, int index ->
def payload = [:]
payload.put('author', sword.author)
payload.put('price', sword.price)
vars.put('test_' + (index + 1), new groovy.json.JsonBuilder(payload).toPrettyString())
}
vars.put('test_matchNr', swords.size() as String)
Now you should be able to refer random author/price pair using __V() and __Random() functions combination like: ${__V(test_${__Random(1,${test_matchNr},)},)}
More information:
Apache Groovy - Parsing and producing JSON
Apache Groovy: What Is Groovy Used For?
If my JSON file is coming out like this, what needs to happen is go to the API link within the u_parent and populate the values from that API link with sysparm_display_value=true into the df. Possible? I need to do this because this API link is giving me the same name and parent and only the link in u_parent will give me the correct parent details.
{
"u_name": "******",
"u_parent": {
"display_value": "*****",
"link": "https://*****.******.com/api/now/table/u_region_hierarchies/ed7f652f1b29341051380e93cc4bcbd7"
},
"sys_id": "159967df1b75601070bfdb9cbc4bcb35",
"sys_updated_by": "mlarcheveque",
"sys_created_on": "01/24/2021 17:31:26",
"sys_mod_count": "1",
"u_active": "true",
"u_region_id": "**********",
"sys_updated_on": "07/30/2021 14:13:33",
"sys_tags": "",
"sys_created_by": "admin"
},
The API link from that u_parent displays the following values and i want the display value from u_parent
{
"result": {
"u_name": "*****",
"u_parent": {
"display_value": "*****",
"link": "https://*****.*****.com/api/now/table/u_region_hierarchies/6d7f252f1b29341051380e93cc4bcbd7"
},
"sys_id": "217f652f1b29341051380e93cc4bcbd4",
"sys_updated_by": "mlarcheveque",
"u_id": "*****",
"sys_created_on": "07/30/2021 14:11:49",
"sys_mod_count": "0",
"sys_updated_on": "07/30/2021 14:11:49",
"sys_tags": "",
"sys_created_by": "mlarcheveque"
}
}
So i am thinking this would involve a do while loop that goes through each row and gets the value from the nested API link
I do have simple JSON data with an array, when I query array contains or join, the Request Unit is too high.
select value count(0)
From c
where ARRAY_CONTAINS(c.signerDetails, {'displayStatus':0}, true)
and c.orgId = "e27dd002-bad3-4444-aa2b-855e5d4e79c8"
For the above ARRAY_CONTAINS query, we get 1882 RU for 40518 Count
select count(0)
From c WHERE c.orgId = "e27dd002-bad3-4444-aa2b-855e5d4e79c8"
and EXISTS(SELECT VALUE t FROM t IN c.signerDetails WHERE t.displayStatus = 0)
For the above exists query, we do get 1960 Ru for 40518 count
{
"id": "1b675bb2-f783-48a0-93bb-967a990b5204f901795b-d18d-4c2d-a68c-61adf7e8e8da",
"orgId": "e27dd002-bad3-4444-aa2b-855e5d4e79c8", //Partition key
"userDetails": [
{
"userName": "",
"userEmail": "",
"displayStatus": 4,
},
{
"userName": "",
"userEmail": "",
"displayStatus": 1,
}
],
"status": "Completed",
"_ts": 1619535494
}
Please share your thoughts on reducing the RU, or do I need to JSON data format or do I need to consider any other database for this kind of operation.
PS: I do have complex array querying, on the above sample I have removed the same for the code brevity.
Let's say we're handling the advertising of various job openings across several channels (newspapers, job boards, etc.). For each channel, we can buy a "publication period" which will mean the channel will advertise our job openings during that period. How can we find the jobs for a given channel that have a publication period valid for today (i.e. starting on or before today, and ending on or after today)? The intent is to be able to generate a feed of "active" job openings that (e.g.) a job board can consume periodically to determine which jobs should be displayed to its users.
Another wrinkle is that each job opening is associated with a given tenant id: the feeds will have to be generated scoped to tenant and channel.
Let's say we have the following simplified documents (if you think the data should be modeled differently, please let me know also):
{
"_id": "A",
"tenant_id": "foo",
"name": "Job A",
"publication_periods": [
{
"channel": "linkedin",
"start": "2021-03-10T00:00:0.0Z",
"end": "2021-03-17T00:00:0.0Z"
},
{
"channel": "linkedin",
"start": "2021-04-10T00:00:0.0Z",
"end": "2021-04-17T00:00:0.0Z"
},
{
"channel": "monster.com",
"start": "2021-03-10T00:00:0.0Z",
"end": "2021-03-17T00:00:0.0Z"
}
]
}
{
"_id": "B",
"tenant_id": "foo",
"name": "Job B",
"publication_periods": [
{
"channel": "linkedin",
"start": "2021-04-10T00:00:0.0Z",
"end": "2021-04-17T00:00:0.0Z"
},
{
"channel": "monster.com",
"start": "2021-03-15T00:00:0.0Z",
"end": "2021-03-20T00:00:0.0Z"
}
]
}
{
"_id": "C",
"tenant_id": "foo",
"name": "Job C",
"publication_periods": [
{
"channel": "monster.com",
"start": "2021-05-15T00:00:0.0Z",
"end": "2021-05-20T00:00:0.0Z"
}
]
}
{
"_id": "D",
"tenant_id": "bar",
"name": "Job D",
"publication_periods": [
...
]
}
How can I query the jobs linked to tenant "foo" that have an active publication period for "monster.com" on for the date of 17.03.2021? (I.e. this query should return both jobs A and B.)
Note that the DB will contain documents of other (irrelevant) types.
Since I essentially need to "find all job openings containing an object in the publication_periods array having: CHAN as the channel value, "start" <= DATE, "end" >= DATE" it appears I'd require a Mango query to achieve this, as standard view queries don't provide comparison operators (if this is mistaken, please correct me).
Naturally, I want the Mango query to be executed only on relevant data (i.e. exclude documents that aren't job openings), but I can find references on how to do this (whether in the docs or elsewhere): all resources I found simply seem to define the Mango index on the entire set of documents, relying on the fact that documents where the indexed field is absent won't be indexed.
How can I achieve what I'm after?
Initially, I was thinking of creating a view that would emit the publication period information along with a {'_id': id} object in order to "JOIN" the job opening document to the matching periods at query time (per Best way to do one-to-many "JOIN" in CouchDB). However, I realized that I wouldn't be able to query this view as needed (i.e. "start" value before today, "end" value after today) since I wouldn't have a definite start/end key to use... And I have no idea how to properly leverage a Mango index/query for this. Presumably I'd have to create a partial index based on document type and the presence of publication periods, but how can I even index the multiple publication periods that can be located within a single document? Can a Mango index be defined against a specific view as opposed to all documents in the DB?
I stumbled upon this answer Mango search in Arrays indicating that I should be able to index the data with
{
"index": {
"fields": [
"tenant_id",
"publication_periods.[].channel",
"publication_periods.[].start",
"publication_periods.[].end"
]
},
"ddoc": "job-openings-periods-index",
"type": "json"
}
And then query them with
{
"selector": {
"tenant_id": "foo",
"publication_periods": {
"$elemMatch": {
"$and": [
{
"channel": "monster.com"
},
{
"start": {
"$lte": "2021-03-17T00:00:0.0Z"
}
},
{
"end": {
"$gte": "2021-03-17T00:00:0.0Z"
}
}
]
}
}
},
"use_index": "job-openings-periods-index"
"execution_stats": true
}
Sadly, I'm informed that the index "was not used because it does not contain a valid index for this query" and terrible performance, which I will leave for another question.
Does the gmail API allow for a way to search for a string and return all the mail that has this string but limit the list to 1 email per email address.
For example, I want to find all mail that contain a certain username but the list only contains one email per address.
There is no support for these types of complex queries with the Gmail API, sadly.
Something along these lines will get it done though:
List messages with your search string as the q-parameter until there is no nextPageToken in the response (then you will have gotten every result of the query).
Get all matching messages, only asking for the From-header, and find all unique senders yourself (batch requests would work great for this use case).
Example
List message ids
q = feel the bern
fields = messages(id),nextPageToken
GET https://www.googleapis.com/gmail/v1/users/me/messages?q=feel+the+bern&fields=messages(id)%2CnextPageToken&access_token={YOUR_API_KEY}
Response
{
"messages": [
{
"id": "151c1de994ad5e43"
},
{
"id": "1514f614e6c07c3f"
}, ...
]
}
Get every message (in batch)
format = metadata
metadataHeaders = from
fields = id,payload/headers
GET https://www.googleapis.com/gmail/v1/users/me/messages/151c1de994ad5e43?format=metadata&metadataHeaders=from&fields=id%2Cpayload%2Fheaders&access_token={YOUR_API_KEY}
Response
{
"id": "151c1de994ad5e43",
"payload": {
"headers": [
{
"name": "From",
"value": "Official Bernie Store <info#berniesanders.com>"
}
]
}
},
{
"id": "151c1de994ad3e22",
"payload": {
"headers": [
{
"name": "From",
"value": "Emil Tholin <emtholin#gmail.com>"
}
]
}
}, ...
Then you can just filter messages out by email address, and then get the entire mail of the ones that are left.