Prometheus is empty (no targets) - azure

Question

Describe the bug
After rolling out i try to access prometheus unfortunately no targets are displayed.
Version of Helm and Kubernetes:
Helm Version:
$ helm version
version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}
Kubernetes Version:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform
:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"9a45ba1752db920873e084791faff8d470278b09", GitTreeState:"clean", BuildDate:"2021-05-19T22:28:02Z", GoVersion:"go1.15.8", Compiler:"gc", Platform
:"linux/amd64"}
Which chart:
prometheus-kube-stack
Which version of the chart:
16.6.0
What you expected to happen:
Targets and service discovery should contain information
Changed values of values.yaml (only put values which differ from the defaults):
values.yaml
alertmanagerSpec:
nodeSelector:
beta.kubernetes.io/os: linux
------------------
grafana:
enabled: true
namespaceOverride: ""
## ForceDeployDatasources Create datasource configmap even if grafana deployment has been disabled
##
forceDeployDatasources: false
nodeSelector:
beta.kubernetes.io/os: linux
## ForceDeployDashboard Create dashboard configmap even if grafana deployment has been disabled
##
forceDeployDashboards: false
## Deploy default dashboards.
##
storageSpec:
volumeClaimTemplate:
metadata:
name: grafana-pvc
spec:
storageClassName: default
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 32Gi
selector:
defaultDashboardsEnabled: true
adminPassword: admin
------------------
kubeStateMetrics:
nodeSelector:
beta.kubernetes.io/os: linux
enabled: true
serviceMonitor:
-------------------
kube-state-metrics:
namespaceOverride: ""
rbac:
create: true
podSecurityPolicy:
enabled: true
nodeSelector:
beta.kubernetes.io/os: linux
--------------------
nodeExporter:
enabled: true
nodeSelector:
beta.kubernetes.io/os: linux
---------------------
prometheus-node-exporter:
namespaceOverride: ""
podLabels:
## Add the 'node-exporter' label to be used by serviceMonitor to match standard common usage in rules and grafana dashboards
##
jobLabel: node-exporter
extraArgs:
- --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
- --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
nodeSelector:
beta.kubernetes.io/os: linux
----------------
prometheusOperator:
enabled: true
patch:
enabled: true
image:
repository: jettech/kube-webhook-certgen
tag: v1.5.2
sha: ""
pullPolicy: IfNotPresent
resources: {}
## Provide a priority class name to the webhook patching job
##
priorityClassName: ""
podAnnotations: {}
nodeSelector:
beta.kubernetes.io/os: linux
affinity: {}
tolerations: []
-------------------------
prometheusSpec:
nodeSelector:
beta.kubernetes.io/os: linux
storageSpec:
volumeClaimTemplate:
metadata:
name: prometheus-pvc
spec:
storageClassName: default
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 32Gi
selector:
Anything else we need to know:
EDIT
NAME READY STATUS RESTARTS AGE
pod/alertmanager-kube-monitoring-kube-prome-alertmanager-0 2/2 Running 0 9h
pod/kube-monitoring-grafana-6896d856d9-krtgd 2/2 Running 0 9h
pod/kube-monitoring-kube-prome-operator-74b76d89f7-td5t9 1/1 Running 0 9h
pod/kube-monitoring-kube-state-metrics-7db74b856-p2qmx 1/1 Running 0 9h
pod/kube-monitoring-prometheus-node-exporter-96mdj 1/1 Running 0 9h
pod/kube-monitoring-prometheus-node-exporter-lmstk 1/1 Running 0 9h
pod/prometheus-kube-monitoring-kube-prome-prometheus-0 2/2 Running 1 9h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 9h
service/kube-monitoring-grafana ClusterIP 10.0.122.52 <none> 80/TCP 9h
service/kube-monitoring-kube-prome-alertmanager ClusterIP 10.0.115.147 <none> 9093/TCP 9h
service/kube-monitoring-kube-prome-operator ClusterIP 10.0.127.119 <none> 443/TCP 9h
service/kube-monitoring-kube-prome-prometheus ClusterIP 10.0.229.127 <none> 9090/TCP 9h
service/kube-monitoring-kube-state-metrics ClusterIP 10.0.106.71 <none> 8080/TCP 9h
service/kube-monitoring-prometheus-node-exporter ClusterIP 10.0.32.130 <none> 9100/TCP 9h
service/prometheus-operated ClusterIP None <none> 9090/TCP 9h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kube-monitoring-prometheus-node-exporter 2 2 2 2 2 beta.kubernetes.io/os=linux 9h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kube-monitoring-grafana 1/1 1 1 9h
deployment.apps/kube-monitoring-kube-prome-operator 1/1 1 1 9h
deployment.apps/kube-monitoring-kube-state-metrics 1/1 1 1 9h
NAME DESIRED CURRENT READY AGE
replicaset.apps/kube-monitoring-grafana-6896d856d9 1 1 1 9h
replicaset.apps/kube-monitoring-kube-prome-operator-74b76d89f7 1 1 1 9h
replicaset.apps/kube-monitoring-kube-state-metrics-7db74b856 1 1 1 9h
If more information is needed please let me know. Thanks in advance!