I am using alertmanager configured to read from a timescale db shared with other Prometheus/alertmanager systems.
I would like to set/check alerts only for services including a specific tag, therefore wondering how could I configure prometheus to apply only for specific tags?
This is what currently I am using:
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']
remote_write:
- url: https://promscale.host:9201/write
remote_read:
- url: https://promscale.host:9201/read
read_recent: true
...
I found there is an option alert_relabel_configs but is unclear for me the usage of it.
Any ideas?
FYI, alert_relabel_configs are used for alert relabeling to alerts before they are sent to the Alertmanager
To use alert_relabel_configs below is the example to add a new tag on matching the relabel config set:
alert_relabel_configs:
- source_labels: [ log_level ]
regex: warn
target_label: severity
replacement: warn
Note: The alerts are only changed when sent to alertmanager. They are
not changed in the Prometheus UI.
To test the relabel config online you can use https://relabeler.promlabs.com/
If you are using Prometheus Operator configuring alert relabeling rules should be done in additionalAlertRelabelConfigs of PrometheusSpec, more details: https://github.com/prometheus-operator/prometheus-operator/issues/1805
Related
i'm building a grafana dashboard where i can monitor multiple cluster in one place. The problem is that when i try to filter metrics based on the cluster name but it doesn't exist in the metrics like kube_node_info and kube_pod_info. So far i created two labels but they are not injected in the mentioned metrics just in some other metrics that i do not find useful, many thanks.
Here's how i created the label:
- job_name: 'cluster-1'
static_configs:
- targets: ['localhost:9100']
labels:
cluster_name: 'cluster-1'
I have several python programs are running as POD in Kubernetes Cluster on AWS EKS. I want Filebeat to automatically pick-up these logging events/messages from standard output to send it to ELK.
The current Filebeat configuration does not seem to work
data:
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
How I can configure Filebeat or Logstash to collect standard out from several python program(s) and automatically ship it to ELK machine?
The above configuration already generates combined logs from all the programs running on a container.
So, this is a correct configuration and one doesn't need to do anything else. It's just that I was not familiar with Kibana hence I could not find it earlier.
data:
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
For Kibana, one needs to apply following wildcard filters to get the respective logs.
*<program_name>*
One can apply additional filter on top of that with kubernetes.namespace:<namespace>
I'm trying to setup Alert Manager in a simple setup where it would send one Slack notification for each notification in receives.
I've hoped to disable grouping by removing the group_by configuration.
The problem is, that when I send 2 alert one after the other, even though the Alert Manager shows the 2 alerts as 'Not Grouped' when I get Slack notifications, I get one message for the first alert, and then a second message, where the 2 alerts are grouped.
Here is the config.yml
route:
receiver: default-receiver
group_wait: 1s #30s
group_interval: 1s #5m
# repeat_interval: 10m
# group_by: [cluster, alertname]
receivers:
- name: default-receiver
slack_configs:
- channel: "#alerts-test"
Any ideas?
From the Prometheus documentation for configuration
You can use group_by: ['...'] in your Alert Manager as a solution.
However, this was introduced in v0.16. For More info, see this GitHub issue.
I've installed Prometheus operator 0.34 (which works as expected) on cluster A (main prom)
Now I want to use the federation option,I mean collect metrics from other Prometheus which is located on other K8S cluster B
Secnario:
have in cluster A MAIN prometheus operator v0.34 config
I've in cluster B SLAVE prometheus 2.13.1 config
Both installed successfully via helm, I can access to localhost via port-forwarding and see the scraping results on each cluster.
I did the following steps
Use on the operator (main cluster A) additionalScrapeconfig
I've added the following to the values.yaml file and update it via helm.
additionalScrapeConfigs:
- job_name: 'federate'
honor_labels: true
metrics_path: /federate
params:
match[]:
- '{job="prometheus"}'
- '{__name__=~"job:.*"}'
static_configs:
- targets:
- 101.62.201.122:9090 # The External-IP and port from the target prometheus on Cluster B
I took the target like following:
on prometheus inside cluster B (from which I want to collect the data) I use:
kubectl get svc -n monitoring
And get the following entries:
Took the EXTERNAL-IP and put it inside the additionalScrapeConfigs config entry.
Now I switch to cluster A and run kubectl port-forward svc/mon-prometheus-operator-prometheus 9090:9090 -n monitoring
Open the browser with localhost:9090 see the graph's and click on Status and there Click on Targets
And see the new target with job federate
Now my main question/gaps. (security & verification)
To be able to see that target state on green (see the pic) I configure the prometheus server in cluster B instead of using type:NodePort to use type:LoadBalacer which expose the metrics outside, this can be good for testing but I need to secure it, how it can be done ?
How to make the e2e works in secure way...
tls
https://prometheus.io/docs/prometheus/1.8/configuration/configuration/#tls_config
Inside cluster A (main cluster) we use certificate for out services with istio like following which works
tls:
mode: SIMPLE
privateKey: /etc/istio/oss-tls/tls.key
serverCertificate: /etc/istio/oss-tls/tls.crt
I see that inside the doc there is an option to config
additionalScrapeConfigs:
- job_name: 'federate'
honor_labels: true
metrics_path: /federate
params:
match[]:
- '{job="prometheus"}'
- '{__name__=~"job:.*"}'
static_configs:
- targets:
- 101.62.201.122:9090 # The External-IP and port from the target
# tls_config:
# ca_file: /opt/certificate-authority-data.pem
# cert_file: /opt/client-certificate-data.pem
# key_file: /sfp4/client-key-data.pem
# insecure_skip_verify: true
But not sure which certificate I need to use inside the prometheus operator config , the certificate of the main prometheus A or the slave B?
You should consider using Additional Scrape Configuration
AdditionalScrapeConfigs allows specifying a key of a Secret
containing additional Prometheus scrape configurations. Scrape
configurations specified are appended to the configurations generated
by the Prometheus Operator.
I am affraid this is not officially supported. However, you can update your prometheus.yml section within the Helm chart. If you want to learn more about it, check out this blog
I see two options here:
Connections to Prometheus and its exporters are not encrypted and
authenticated by default. This is one way of fixing that with TLS
certificates and
stunnel.
Or specify Secrets which you can add to your scrape configuration.
Please let me know if that helped.
A couple of options spring to mind:
Put the two clusters in the same network space and put a firewall in-front of them
VPN tunnel between the clusters.
Use istio multicluster routing (but this could get complicated): https://istio.io/docs/setup/install/multicluster
I'm trying to configure filebeat for IIS logs for multiple IIS application.
IIS logs are stored in separate folders for each app.
My main goal to achieve, is to have separate set of tags fields for each application logs
So I added two prospectors configuration like that:
filebeat.prospectors:
- input_type: log
paths:
- c:\inetpub\logs\LogFiles\W3SVC2\*.log
document_type: iis_log
tags: ["firstAPP", "serverName"]
fields:
env: production
- input_type: log
paths:
- c:\inetpub\logs\LogFiles\W3SVC3\*.log
document_type: iis_log
tags: ["secondAPP", "serverName", "API"]
fields:
env: production
Unfortunately something is wrong with this configuration, because filebeat won't start.
When I use configuration for only one log path, everything works fine.
According to this example: https://www.elastic.co/guide/en/beats/filebeat/current/multiple-prospectors.html, configuration of multiple filebeat.prospectors is possible. But I'm not sure if I can add multiple tags.
If that's a perfect copy of your config, the tags line for your second prospector just needs to be indented.