Description for a stopped containers - prometheus-alertmanager

Question

I added a rule into the rules.yml in order to get an alert whenever a container stops.
In order to get an alert for each stopped container with the suffix "dev-23", I used this rule:
- alert: ContainerKilled
expr: absent(container_start_time_seconds{name=~".*dev-23"})
for: 0m
labels:
severity: 'critical'
annotations:
summary: 'Container killed (instance {{ $labels.instance }})'
description: 'A container has disappeared\n VALUE = {{ $value }}\n LABELS: {{ $labels }}'
This indeed works, and I get an alert whenever a container that ends with "dev-23" stops. However, the summary and description of the receieved alert do not tell me what is the name of the stopped container.
In the alert I get this description:
description = A container has disappeared\n VALUE = 1\n LABELS: map[]
summary = Container killed (instance )
What should I use in order to get the exact name of the stopped container?

Answer 1

The issue is, that the metrics created with the absent() function does not have any labels and its value is always 1 (if the queried metric is not present)
You can leave it like that, without any label, if you just want to check if there is NO container running at all, without any details about its previous state and that is enough information to detect the issue.
Or you can use the UP metric, that has the labels, but potentially disappers, e.g. when you have a dynamic service discovery.
In general it a good practice to have an alert on UP{...}==0 and on absent({...})