I would like to install the alertmanager datasource (https://grafana.com/grafana/plugins/camptocamp-prometheus-alertmanager-datasource/) to my kube-prometheus-stack installation which is being built using terraform and the helm provider. I cannot work out how to get the plugin files to the node running grafana though.
Using a modified values.yaml and feeding to helm with -f values.yaml (please ignore values):
additionalDataSources:
- name: Alertmanager
editable: false
type: camptocamp-prometheus-alertmanager-datasource
url: http://localhost:9093
version: 1
access: default
# optionally
basicAuth: false
basicAuthUser:
basicAuthPassword:
I can see the datasource in grafana but the plugin files do not exist.
Alertmanager visible in list of datasources
However, clicking on the datasource I see
Plugin not found, no installed plugin with that ID
Please note that the grafana pod seems to require a restart to pick up datasource changes as well which I would consider needs fixing at a higher level.
It's actually quite simple to get the files there and I cannot believe I overlooked the simplistic solution. Posting this here in the hope others find it useful.
In the kube-prometheus-stack, values.yaml file, just override the grafana section as follows:
grafana:
.
.
.
plugins:
- camptocamp-prometheus-alertmanager-datasource
- grafana-googlesheets-datasource
- doitintl-bigquery-datasource
- redis-datasource
- xginn8-pagerduty-datasource
- marcusolsson-json-datasource
- grafana-kubernetes-app
- yesoreyeram-boomtable-panel
- savantly-heatmap-panel
- bessler-pictureit-panel
- grafana-polystat-panel
- dalvany-image-panel
- michaeldmoore-multistat-panel
additionalDataSources:
- name: Alertmanager
editable: false
type: camptocamp-prometheus-alertmanager-datasource
url: http://prometheus-kube-prometheus-alertmanager.monitoring:9093
version: 1
access: default
# optionally
basicAuth: false
basicAuthUser:
basicAuthPassword:
where the name / type of the plugin can be found on the installation instructions on the Grafana Plugins page
I made some progress by discovering I could get onto the pod running grafana using:
kubectl exec -it --container grafana prometheus-grafana-5d844b67c6-5p46b -- /bin/sh
The one listed in kubectl get pods was the sidecar.
Then I could run:
kubectl exec -it --container grafana prometheus-grafana-5d844b67c6-5p46b -- grafana-cli plugins install camptocamp-prometheus-alertmanager-datasource
which did the required file installation. After deleting and recreating the pod, there is progress
Keen to hear any comments on the approach or better ideas!
Related
We have a small collection of Kubernetes pods which run react/next.js UIs in a node 16 alpine container (node:16.18.1-alpine3.15 to be precise). All of this runs in AWS EKS 1.23. We make use of annotations on these pods in order to inject secrets from Hashicorp Vault at start up. The annotations pull the desired secrets from Vault and write these to a file on the pod. Example of said annotations below :
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-init-first: "true"
vault.hashicorp.com/agent-pre-populate-only: "true"
vault.hashicorp.com/role: "onejourney-ui"
vault.hashicorp.com/agent-inject-secret-config: "secret/data/onejourney-ui"
vault.hashicorp.com/agent-inject-template-config: |
{{- with secret "secret/data/onejourney-ui" -}}
export AUTH0_CLIENT_ID="{{ .Data.data.auth0_client_id }}"
export SENTRY_DSN="{{ .Data.data.sentry_admin_dsn }}"
{{- end }}
When the pod starts up, we source this file (which is created by default at /vault/secrets/config) to set environment variables and then delete the file. We do that with the following pod arguments in our helm chart :
node:
args:
- /bin/sh
- -c
- source /vault/secrets/config; rm -rf /vault/secrets/config; yarn start-admin;
We recently upgraded some of AWS EKS clusters from 1.23 to 1.24. After doing so, we noted that our node applications were failing to start and entering a crash loop. Looking in the logs of these containers, the problem seemed to be that the pod was unable to locate the secrets file anymore.
Interestingly, the Vault init container completed successfully and shows that the file was successfully created...
Out of curiosity, I removed the node args to source the file which allowed the container to start successfully, but I found when execing into the pod, the file WAS infact present and had the content I was expecting. The file also had the correct owner and permissions as we see in a good working instance in EKS 1.23.
We have other containers (php-fpm) which consume secrets in the same manner however these were not affected on 1.24, only node containers were affected. There were no namespace, pod or deployment annotations I saw added which would have been a possible cause. After rolling the cluster back down to EKS 1.23, the deployment worked as expected.
I'm left scratching my head as to why the pod is unable to source that file on 1.24. Any suggestions on what to check or a possible cause would be greatly appreciated.
I am new to Kustomize and am getting the following error:
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "Kustomization" in version "kustomize.config.k8s.io/v1beta1"
but I am using the boilerplate kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
Question: What does the group name (kustomize.config.k8s.io) mean and why does Kustomize not recognize the kind?
So this api version is correct, although I am still not certain why. In order to get past this error message, I needed to run:
kubectl apply -k dir/.
I hope this helps someone in the future!
If you used apply -f you would see this error. Using -k would definitely work.
You are using kustomize tool (Kustomize is a standalone tool to customize the creation of Kubernetes objects through a file called kustomization.yaml). For applying customization you have to use:
kubectl apply -k foldername(where you store the deploy,service yaml file)
I've deployed Spark Operator to GKE using the Helm Chart to a custom namespace:
helm install --name sparkoperator incubator/sparkoperator --namespace custom-ns --set sparkJobNamespace=custom-ns
and confirmed the operator running in the cluster with helm status sparkoperator.
However when I'm trying to run the Spark Pi example kubectl apply -f examples/spark-pi.yaml I'm getting the following error:
the path "examples/spark-pi.yaml" does not exist
There are few things that I probably still don't get:
Where is actually examples/spark-pi.yaml located after deploying the operator?
What else should I check and what other steps should I take to make the example work?
Please find the spark-pi.yaml file here.
You should copy it to your filesystem, customize it if needed, and provide a valid path to it with kubectl apply -f path/to/spark-pi.yaml.
kubectl apply needs a yaml file either locally on the system where you are running kubectl command or it can be a http/https endpoint hosting the file.
I am running a kubernetes cluster with NFS NAStorage, and when I mount volumes they get a .snapshot directory created at the mountpoint. This causes problems for example when using Helm Charts, as these don't expect an unknown Read Only directory in certain paths (e.g. chown ... <dir> can fail, crashing the container).
When installing the Graylog Helm Chart, I noticed the initContainer for the graylog pod crashing due to chown: ... Read-only file system after running the following chown line:
chown -R 1100:1100 /usr/share/graylog/data/
where the following volume is mounted:
...
volumeMounts:
- mountPath: /usr/share/graylog/data/journal
name: journal
...
I tried working around this by modifying the command to fail "quietly" by making it run : upon failure:
chown -fR 1100:1100 /usr/share/graylog/data/ || :
This made the initContainer succeed, but now the main container crashes instead, this time due to the mere presence of the .snapshot dir.
...
kafka.common.KafkaException: Found directory /usr/share/graylog/data/journal/.snapshot, '.snapshot' is not in the form of topic-partition
If a directory does not contain Kafka topic data it should not exist in Kafka's log directory
...
I have tried modifying the mount point of the volume, too, moving it up one level, but this causes new issues:
...
volumeMounts:
- mountPath: /usr/share/graylog/data
name: data-journal
...
com.github.joschi.jadconfig.ValidationException: Parent path /usr/share/graylog/data/journal for Node ID file at /usr/share/graylog/data/journal/node-id is not a directory
I would have expected there to be some way of disabling the creation of the .snapshot directory, ideally a way to unmount/never mount it in the first place. That, or any way to have the container properly ignore the directory entirely, to make it not interfere with the processes in the container, since it seems the very presence of it can seriously disrupt. However, I have yet to find anything of the sort, and I can't seem to find anyone having had a similar issue (the introduction of Volume Snapshots in kubernetes has not made the searching easier, to say the least).
Edit 1
I tried (semi successfully, I get the Parent path ... is not a directoryerror above) to implement subPath: journal, thus circumventing the .snapshot directory (or so I believe), but this still means potentially editing every Chart that is used in my cluster. Hopefully an alternative on a higher level can be found.
volumeMounts:
- mountPath: /usr/share/graylog/data/journal
name: data-journal
subPath: journal
Edit 2
I am running a bare-metal cluster, with MetalLB and Nginx as loadbalancer+ingress controller.
The storage solution is provided by a third party provider, and it is from their backup solution that the .snapshot directory is made.
My imagined workaround
Since this will mainly be a problem when using Helm Charts or other deployments where volume mounts will be more or less out of our control, I will look into applying a "kustomization" that adds a single line to each volumeMount, adding
...
subPath: mount
or something like that. By doing that, I should be separating the actual mount point in the volume and the directory that actually gets mounted in the container by one level, keepin the .snapshot directory hidden in the abstract volume object. I will post my findings and potential kustomization that may come of it, if anyone else runs into a similar problem.
If someone thinks of a more streamlined solution, it is still very welcome - I'm sure it is possible to improve upon this one.
We finally got this fixed by the storage service provider, after them figuring out which configuration needed to be applied. If anyone has run into the same problem and needs to know which configuration, please reach out and I will ask our service provider.
The workaround that worked before we got the configuration fixed was as follows:
(Including --namespace is optional)
Install mongodb-replicaset and elasticsearch (v 6.8.1)
$ helm install --name mongodb-replicaset --namespace graylog stable/mongodb-replicaset
# We add the elastic repo since the 'stable' repo will be deprecated further on
$ helm repo add elastic https://helm.elastic.co
# We run elasticsearch version 6.8.1 since Graylog v3 currently is incompatible with later versions.
$ helm install elastic/elasticsearch --name elasticsearch --namespace graylog --set imageTag=6.8.1
# Wait for deployments to complete, then you can test to see all went well
$ helm test mongodb-replicaset
$ helm test elasticsearch
Extraxt Graylog deployment template
$ helm fetch --untar stable/graylog
$ helm dependency update graylog
$ helm template graylog -n graylog -f graylog-values.yaml > graylog-template.yaml
#graylog-values.yaml
tags:
install-mongodb: false
install-elasticsearch: false
graylog:
mongodb:
uri: "mongodb://mongodb-replicaset-0.mongodb-replicaset.graylog.svc.cluster.local:27017/graylog?replicaSet=rs0"
elasticsearch:
hosts: "http://elasticsearch-client.graylog.svc.cluster.local:9200"
# + any further values
Add namespace: graylog to all objects in graylog-template.yaml
Add subPath: mount to all volumeMounts where a persistent volume is used (in this case name: journal) in graylog-template.yaml
...
volumeMounts:
- mountPath: /usr/share/graylog/data/journal
name: journal
+ subPath: mount
...
volumeMounts:
- mountPath: /usr/share/graylog/data/journal
name: journal
+ subPath: mount
...
volumeClaimTemplates:
- metadata:
creationTimestamp: null
name: journal
This can be done quickly in vim by typing :g/name: <volume-name>/norm osubPath: mount. Please note the lack of a space between "o" and "subPath", and note that this will add the line to the volumeClaimTemplate as well, which needs to be removed. "mount" can also be called something else.
Deploy
$ kubectl apply -f graylog-template.yaml
I'm trying to install Istio with automatic sidecar injection into Kubernetes. My environment consists of three masters and two nodes and was built on Azure using the Azure Container Service marketplace product.
Following the documentation located here, I have so far enabled RBAC and DynamicAdmissionControl. I have accomplished this by modifying /etc/kubernetes/istio-inializer.yaml on the Kubernetes Master by adding the following content outlined in red and then restarting the Kubernetes Master using the Unix command, reboot.
The next step in the documentation is to apply the yaml using kubectl. I assume that the documentation intends for the user to clone the Istio repository and cd into it before this step but that is unmentioned.
git clone https://github.com/istio/istio.git
cd istio
kubectl apply -f install/kubernetes/istio-initializer.yaml
After which the following error occurs:
user#hostname:~/istio$ kubectl apply -f install/kubernetes/istio-initializer.yaml
configmap "istio-inject" configured
serviceaccount "istio-initializer-service-account" configured
error: error validating "install/kubernetes/istio-initializer.yaml": error validating data: found invalid field initializers for v1.ObjectMeta; if you choose to ignore these errors, turn validation off with --validate=false
If I attempt to execute kubectl apply with the mentioned flag, validate=false, then this error is generated instead:
user#hostname:~/istio$ kubectl apply -f install/kubernetes/istio-initializer.yaml --validate=false
configmap "istio-inject" configured
serviceaccount "istio-initializer-service-account" configured
deployment "istio-initializer" configured
error: unable to recognize "install/kubernetes/istio-initializer.yaml": no matches for admissionregistration.k8s.io/, Kind=InitializerConfiguration
I'm not sure where to go from here. The problem appears to be related to the admissionregistration.k8s.io/v1alpha1 block in the yaml but I'm unsure what specifically is incorrect in this block.
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: InitializerConfiguration
metadata:
name: istio-sidecar
initializers:
- name: sidecar.initializer.istio.io
rules:
- apiGroups:
- "*"
apiVersions:
- "*"
resources:
- deployments
- statefulsets
- jobs
- daemonsets
Installed version of Kubernetes:
user#hostname:~/istio$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:21:54Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:21:54Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
I suspect this is a versioning mismatch. As a follow up question, is it possible to deploy a version of kubernetes >= 1.7.4 to Azure using ACS?
I'm fairly new to working with Kubernetes so if anyone could help I would greatly appreciate it. Thank you for your time.
Seems to be a versioning problem as the alpha feature is supported for k8s version> 1.7 as mentioned here (https://kubernetes.io/docs/admin/extensible-admission-controllers/#what-are-initializers).
1.7 introduces two alpha features, Initializers and External Admission
Webhooks, that address these limitations. These features allow admission
controllers to be developed out-of-tree and configured at runtime.
And it is possible to deploy a version of kubernetes >= 1.7.4 to Azure. Note sure about the deployed version using the portal. But if you use acs-egnine to generate the ARM template, it is possible to deploy a cluster with version 1.7.5.
You can refer here for the procedures https://github.com/Azure/acs-engine. Basically it involves three steps. First, you should create the json file by referring to the clusterDefinition section. To use version 1.7.5, you should specify the attribute "orchestratorRelaease" to be "1.7" and also enable the RBAC by specifying the attribute "enableRbac" to be true. Second, use the acs engine (version >= 0.6.0) to parse the json file to ARM template (azuredeploy.json & azuredeploy.parameters.json should be created). Lastly, use the command "New-AzureRmResourceGroupDeployment" in powershell to deploy the cluster to Azure.
Hope this helps :)