I have two Grafana Loki installations. Done with helm from official repository.
Both are exact the same configured (expecting DNS)
The only difference is, one is on Azure and one is on own Esxi.
The problem I have is the log file parsing. The installation on Azure seems to parse the log files always with - cri {} settings and not with - docker {}
A quick search inside the promtail pods show me inside the promtail.yaml the - docker {} setting. But I always get the output:
2023-01-16 10:39:15
2023-01-16T09:39:15.604384089Z stdout F {"level":50,"time":1673861955603,"service
On our own Esxi I have the correct:
2023-01-13 16:58:18
{"level":50,"time":1673625498068,"service"
From what I read the stdout F is -cri {} parsing, default by promtail.
Any idea why this happen? My installation yaml is:
#helm upgrade --install loki --namespace=monitoring grafana/loki-stack -f value_mus.yaml
grafana:
enabled: true
admin:
existingSecret: grafana-admin-credentials
sidecar:
datasources:
enabled: true
maxLines: 1000
image:
tag: latest
persistence:
enabled: true
size: 10Gi
storageClassName: managed-premium
accessModes:
- ReadWriteOnce
grafana.ini:
users:
default_theme: light
server:
domain: xxx
smtp:
enabled: true
from_address: xxx
from_name: Grafana Notification
host: xxx
user: xxx
password: xxx
skip_verify: false
startTLS_policy:
promtail:
enabled: true
config:
snippets:
pipelineStages:
- docker: {}
Any help will be welcome.
Related
Skaffold setValues is getting missing helm values.
Instead of setValues, when I save the relevant values in the values.yml file and use valuesFiles, there is no problem and the rendering is successful.
I guess setValues doesn't recognize nested arrays. Please review the example below.
Why the ingress.tls[0].hosts doesn't exist?
skaffold.yaml
apiVersion: skaffold/v2beta29
kind: Config
build:
local:
push: false
tagPolicy:
sha256: {}
artifacts:
- image: example
jib: {}
sync:
auto: false
deploy:
helm:
releases:
- name: example
chartPath: backend-app
artifactOverrides:
image: example
imageStrategy:
helm: {}
setValues:
ingress:
enabled: true
className: nginx
hosts:
- host: example.minikube
paths:
- path: /
pathType: ImplementationSpecific
tls:
- secretName: example-tls
hosts:
- example.minikube
skaffold run
skaffold run -v TRACE
# Output
[...]
[...]
[...]
DEBU[0106] Running command: [
helm --kube-context minikube install example backend-app --set-string image.repository=example,image.tag=6ad72230060e482fef963b295c0422e8d2f967183aeaca0229838daa7a1308c3 --set ingress.className=nginx --set --set ingress.enabled=true --set ingress.hosts[0].host=example.minikube --set ingress.hosts[0].paths[0].path=/ --set ingress.hosts[0].paths[0].pathType=ImplementationSpecific --set ingress.tls[0].secretName=example-tls] subtask=0 task=Deploy
[...]
[...]
[...]
Ingress Manifest
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
spec:
ingressClassName: nginx
tls:
- hosts:
secretName: example-tls
rules:
- host: "example.minikube"
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: example
port:
number: 80
This was fixed recently via the PR here:
https://github.com/GoogleContainerTools/skaffold/pull/8152
This is currently in skaffold in main and will be available in the v2.1.0 Skaffold release (to be released 12/7/2022) and onward
EDIT: v2.1.0 release is delayed w/ some of the maintainers on holiday. Currently planned to be available late Dec or early Jan
EDIT #2: v2.1.0 is out now (1/23/2023)
I need to pull the image from public docker repository i.e hello-world:latest and run that image on kubernetes. I created cluster using Kind . I ran that image using the below command
kubectl run test-pod --image=hello-world
Then I did
kubectl describe pods
to get the status of the pods. It threw me ImagePullBackOff error . Please find the snapshot below. It seems there is some network issue when pulling the image using kind cluster. Although I am able to pull image from docker easily .
Have searched the whole internet regarding this issue but nothing worked out. Following is my pod specification :
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2022-05-16T15:01:17Z"
labels:
run: test-pod
name: test-pod
namespace: default
resourceVersion: "4370"
uid: 6ef121e2-805b-4022-9a13-c17c031aea88
spec:
containers:
- image: hello-world
imagePullPolicy: Always
name: test-pod
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-jjsmp
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: kind-control-plane
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: kube-api-access-jjsmp
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-05-16T15:01:17Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-05-16T15:01:17Z"
message: 'containers with unready status: [test-pod]'
reason: ContainersNotReady
status: "False"
type: Ready
containerStatuses:
- image: hello-world
imageID: ""
lastState: {}
name: test-pod
ready: false
restartCount: 0
started: false
state:
waiting:
message: Back-off pulling image "hello-world"
reason: ImagePullBackOff
hostIP: 172.18.0.2
phase: Pending
podIP: 10.244.0.5
podIPs:
- ip: 10.244.0.5
qosClass: BestEffort
startTime: "2022-05-16T15:01:17Z"
The ImagePullBackOff error means that Kubernetes couldn't pull the image from the registry and will keep trying to pull the image until it reaches a compiled-in limit of 300 seconds (5 minutes). This issue could happen because Kubernetes is facing one of the following conditions:
You have exceeded the rate or download limit on the registry.
The image registry requires authentication.
There is a typo in the image name or tag.
The image or tag does not exist.
You can start reviewing if you can pull the image locally or try to ssh jumping on the node and run docker pull and get the image directly.
If you still can't pull the image, another option is to add 8.8.8.8 to /etc/resolv.conf.
Update:
To avoid exposing your kind cluster to the internet try to pull the image locally at your PC by manually specifying a new path from a different registry.
Sample:
docker pull myregistry.local:5000/testing/test-image
From Azure we try to create container using the Azure Container Instances with prepared YAML. From the machine where we execute az container create command we can login successfully to our private registry (e.g fa-docker-snapshot-local.docker.comp.dev on JFrog Artifactory ) after entering password and we can docker pull it as well
docker login fa-docker-snapshot-local.docker.comp.dev -u svc-faselect
Login succeeded
So we can pull it successfully and the image path is the same like when doing manually docker pull:
image: fa-docker-snapshot-local.docker.comp.dev/fa/ads:test1
We have YAML file for deploy, and trying to create container using the az command from the SAME server. In the YAML file we have set up the same registry information: server, username and password and the same image
az container create --resource-group FRONT-SELECT-NA2 --file ads-azure.yaml
When we try to execute this command, it takes for 30 minutes and after that message is displayed: "Deployment failed. Operation failed with status 200: Resource State Failed"
Full Yaml:
apiVersion: '2019-12-01'
location: eastus2
name: ads-test-group
properties:
containers:
- name: front-arena-ads-test
properties:
image: fa-docker-snapshot-local.docker.comp.dev/fa/ads:test1
environmentVariables:
- name: 'DBTYPE'
value: 'odbc'
command:
- /opt/front/arena/sbin/ads_start
- ads_start
- '-unicode'
- '-db_server test01'
- '-db_name HEDGE2_ADM_Test1'
- '-db_user sqldbadmin'
- '-db_password pass'
- '-db_client_user HEDGE2_ADM_Test1'
- '-db_client_password Password55'
ports:
- port: 9000
protocol: TCP
resources:
requests:
cpu: 1.0
memoryInGB: 4
volumeMounts:
- mountPath: /opt/front/arena/host
name: ads-filesharevolume
imageRegistryCredentials: # Credentials to pull a private image
- server: fa-docker-snapshot-local.docker.comp.dev
username: svcacct-faselect
password: test
ipAddress:
type: Private
ports:
- protocol: tcp
port: '9000'
volumes:
- name: ads-filesharevolume
azureFile:
sharename: azurecontainershare
storageAccountName: frontarenastorage
storageAccountKey: kdUDK97MEB308N=
networkProfile:
id: /subscriptions/746feu-1537-1007-b705-0f895fc0f7ea/resourceGroups/SELECT-NA2/providers/Microsoft.Network/networkProfiles/fa-aci-test-networkProfile
osType: Linux
restartPolicy: Always
tags: null
type: Microsoft.ContainerInstance/containerGroups
Can you please help us why this error occurs?
Thank you
According to my knowledge, there is nothing wrong with your YAML file, I only can give you some possible reasons.
Make sure the configurations are all right, the server URL, username, and password, also include the image name and tag;
Change the port from '9000' into 9000``, I mean remove the double quotes;
Take a look at the Note, maybe the mount volume makes a crash to the container. Then you need to mount the file share to a new folder, I mean the new folder that does not exist before.
I created an AKS cluster with http enabled.Also I have my project with dev spaces enabled to use the cluster.While runing azds up the app is creating all necessary deployment files (helm.yaml,charts.yaml,values.yaml).However I want to access my app using a public endpoint with dev space url but when I do azds list-uris it is only giving localhost url and not the url with dev space enabled.
Can anyone please help?
My azds.yaml looks like below
kind: helm-release
apiVersion: 1.1
build:
context: .
dockerfile: Dockerfile
install:
chart: charts/webfrontend
values:
- values.dev.yaml?
- secrets.dev.yaml?
set:
# Optionally, specify an array of imagePullSecrets. These secrets must be manually created in the namespace.
# This will override the imagePullSecrets array in values.yaml file.
# If the dockerfile specifies any private registry, the imagePullSecret for that registry must be added here.
# ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
#
# For example, the following uses credentials from secret "myRegistryKeySecretName".
#
# imagePullSecrets:
# - name: myRegistryKeySecretName
replicaCount: 1
image:
repository: webfrontend
tag: $(tag)
pullPolicy: Never
ingress:
annotations:
kubernetes.io/ingress.class: traefik-azds
hosts:
# This expands to form the service's public URL: [space.s.][rootSpace.]webfrontend.<random suffix>.<region>.azds.io
# Customize the public URL by changing the 'webfrontend' text between the $(rootSpacePrefix) and $(hostSuffix) tokens
# For more information see https://aka.ms/devspaces/routing
- $(spacePrefix)$(rootSpacePrefix)webfrontend$(hostSuffix)
configurations:
develop:
build:
dockerfile: Dockerfile.develop
useGitIgnore: true
args:
BUILD_CONFIGURATION: ${BUILD_CONFIGURATION:-Debug}
container:
sync:
- "**/Pages/**"
- "**/Views/**"
- "**/wwwroot/**"
- "!**/*.{sln,csproj}"
command: [dotnet, run, --no-restore, --no-build, --no-launch-profile, -c, "${BUILD_CONFIGURATION:-Debug}"]
iterate:
processesToKill: [dotnet, vsdbg, webfrontend]
buildCommands:
- [dotnet, build, --no-restore, -c, "${BUILD_CONFIGURATION:-Debug}"]
I followed below guide
https://microsoft.github.io/AzureTipsAndTricks/blog/tip228.html
AZDS up is giving end point to my localhost
Service 'webfrontend' port 80 (http) is available via port forwarding at http://localhost:50597
Has your azds.yaml file ingress definition to the public 'webfrontend' domain?
Here is an example azds.yaml file created using .NET Core sample application:
kind: helm-release
apiVersion: 1.1
build:
context: .
dockerfile: Dockerfile
install:
chart: charts/webfrontend
values:
- values.dev.yaml?
- secrets.dev.yaml?
set:
replicaCount: 1
image:
repository: webfrontend
tag: $(tag)
pullPolicy: Never
ingress:
annotations:
kubernetes.io/ingress.class: traefik-azds
hosts:
# This expands to [space.s.][rootSpace.]webfrontend.<random suffix>.<region>.azds.io
# Customize the public URL by changing the 'webfrontend' text between the $(rootSpacePrefix) and $(hostSuffix) tokens
# For more information see https://aka.ms/devspaces/routing
- $(spacePrefix)$(rootSpacePrefix)webfrontend$(hostSuffix)
configurations:
develop:
build:
dockerfile: Dockerfile.develop
useGitIgnore: true
args:
BUILD_CONFIGURATION: ${BUILD_CONFIGURATION:-Debug}
container:
sync:
- "**/Pages/**"
- "**/Views/**"
- "**/wwwroot/**"
- "!**/*.{sln,csproj}"
command: [dotnet, run, --no-restore, --no-build, --no-launch-profile, -c, "${BUILD_CONFIGURATION:-Debug}"]
iterate:
processesToKill: [dotnet, vsdbg]
buildCommands:
- [dotnet, build, --no-restore, -c, "${BUILD_CONFIGURATION:-Debug}"]
More about it: https://learn.microsoft.com/pl-pl/azure/dev-spaces/how-dev-spaces-works-prep
How many service logs do you see in 'azds up' log, are you watching something similar to:
Service 'webfrontend' port 'http' is available at `http://webfrontend.XXX
Did you follow this guide?
https://learn.microsoft.com/pl-pl/azure/dev-spaces/troubleshooting#dns-name-resolution-fails-for-a-public-url-associated-with-a-dev-spaces-service
Do you have the latest version of the azds?
I have been using helm chart to install elasticserahc and kibana into kubernetes,
using the defualt configuration everything went ok but I want to enable the security on both elasticsearch and kibana
I didi what's recommanded in the documentation , the security was enabled for elasticsearch but I have probleme upgrading kibana with security configuratuion it gives me this error :
Error: release helm-kibana-security failed: timed out waiting for the condition
once I run make ( from /kibana/examples/security )
I even tried to install is directly without using the Makefile :
helm install --wait --timeout=600 --values ./security.yml --name helm-kibana-security ../../
but having the same issue , can any one help me please
"failed: timed out waiting for the condition"
This message occurs when you install a release with the --wait flag, however, the pods are unable to start for some reason.
The problem is most likely in "./security.yml"
try running the below commands to debug the issue:
kubectl describe pod kibana-pod-name
kubectl logs kibana-pod-name
this is the security.yml file
---
elasticsearchHosts: "https://security-master:9200"
extraEnvs:
- name: 'ELASTICSEARCH_USERNAME'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
- name: 'ELASTICSEARCH_PASSWORD'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
kibanaConfig:
kibana.yml: |
server.ssl:
enabled: true
key: /usr/share/kibana/config/certs/kibana/kibana.key
certificate: /usr/share/kibana/config/certs/kibana/kibana.crt
xpack.security.encryptionKey: something_at_least_32_characters
elasticsearch.ssl:
certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
verificationMode: certificate
protocol: https
secretMounts:
- name: elastic-certificate-pem
secretName: elastic-certificate-pem
path: /usr/share/kibana/config/certs
- name: kibana-certificates
secretName: kibana-certificates
path: /usr/share/kibana/config/certs/kibana