How to connect Vault with Consul agent on Kubernetes via Helm chart (Consul server is on Azure managed app) - azure

Question

Requesting for some help on : How to connect Vault with Consul agent on Kubernetes via Helm chart (Consul server is on Azure managed app)
I'm trying to build a POC, for Vault and consul and got some questions.
Deployed Azure managed app using - https://learn.hashicorp.com/tutorials/consul/hashicorp-consul-service-deploy
Installed consul agent on AKS with the steps in the https://learn.hashicorp.com/tutorials/consul/hashicorp-consul-service-aks?in=consul/hcs-azure
Consul helm chart : https://github.com/hashicorp/consul-helm
Installed vault via helm chart: https://github.com/hashicorp/vault-helm
Kubernetes services and pods for consul.
~$kubectl get svc -n consul
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul-connect-injector-svc ClusterIP 10.0.252.97 <none> 443/TCP 2d13h
consul-controller-webhook ClusterIP 10.0.169.80 <none> 443/TCP 2d13h
~$kubectl get pods -n consul
NAME READY STATUS RESTARTS AGE
consul-27j4j 1/1 Running 0 2d13h
consul-connect-injector-webhook-deployment-9454b8d68-778rd 1/1 Running 0 2d13h
consul-controller-7857456f99-mhzpw 1/1 Running 1 2d13h
consul-lkhpl 1/1 Running 0 2d13h
consul-webhook-cert-manager-cfbb689f7-fgtlw 1/1 Running 0 2d13h
consul-zf989 1/1 Running 0 2d13h
vault config as below:
ui:
enabled: true
serviceType: LoadBalancer
server:
ingress:
enabled: true
extraPaths:
- path: /
backend:
serviceName: vault-ui
servicePort: 8200
hosts:
- host: vault.something_masked.com
ha:
enabled: true
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "consul" {
path = "vault/"
scheme = "https"
address = "HOST_IP:8500"
}
Vault pods
kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 0/1 Running 0 7m14s
vault-1 0/1 Running 0 7m11s
vault-2 0/1 Running 0 7m11s
vault-agent-injector-cbbb6f4df-rmbd7 1/1 Running 0 7m22s
ERROR : Vault is unable to make communication with consul agent.
Logs for vault-0 pod
kubectl logs vault-0 -n vault
WARNING! Unable to read storage migration status.
2021-06-27T08:37:17.801Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2021-06-27T08:37:18.824Z [WARN] storage migration check error: error="Get "https://10.54.0.206:8500/v1/kv/vault/core/migration": dial tcp 10.54.0.206:8500: connect: connection refused"
Logs for vault-agent-injector pod
kubectl logs vault-agent-injector-cbbb6f4df-rmbd7 -n vault
2021-06-27T08:37:09.189Z [INFO] handler: Starting handler..
Listening on ":8080"...
2021-06-27T08:37:09.218Z [INFO] handler.auto-tls: Generated CA
2021-06-27T08:37:09.219Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-27T08:37:18.252Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=10s
2021-06-27T08:37:18.452Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=10s
Any suggestions or advice in the vault configuration if I have missed something ??
Thank you in advance.
Regards
Pooja

Answer 1

There are many points to debug the issue and the problem can be anywhere
the major issue I am seeing right now is your vault pods are running but not in a Ready state.
You have to unseal the vault from seal status.
Read more about the Unseal : https://learn.hashicorp.com/tutorials/vault/ha-with-consul#step-5-start-vault-and-verify-its-state
also how your consul is iteracting with vault ? Using LB, Ingress, service name ?

Answer 2

I am using consul azure managed app server and i have installed consul agent on aks.
kubectl get svc -n consul
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul-connect-injector-svc ClusterIP 10.0.252.97 <none> 443/TCP 3d13h
consul-controller-webhook ClusterIP 10.0.169.80 <none> 443/TCP 3d13h
as i am using consul agent i do not see consul-server running.
Helm chart config for consul
global:
enabled: false
name: consul
datacenter: dc1
acls:
manageSystemACLs: true
bootstrapToken:
secretName: XXX-sandbox-managed-app-bootstrap-token
secretKey: token
gossipEncryption:
secretName: XXX-sandbox-managed-app-hcs
secretKey: gossipEncryptionKey
tls:
enabled: true
enableAutoEncrypt: true
caCert:
secretName: XXX-sandbox-managed-app-hcs
secretKey: caCert
externalServers:
enabled: true
hosts:
['XXX.az.hashicorp.cloud']
httpsPort: 443
useSystemRoots: true
k8sAuthMethodHost: https://XXX.uksouth.azmk8s.io:443
client:
enabled: true
# If you are using Kubenet in your AKS cluster (the default network),
# uncomment the line below.
# exposeGossipPorts: true
join:
['XXX.az.hashicorp.cloud']
connectInject:
enabled: true
controller:
enabled: true
Helm chart config for vault
ui:
enabled: true
serviceType: LoadBalancer
server:
ingress:
enabled: true
extraPaths:
- path: /
backend:
serviceName: vault-ui
servicePort: 8200
hosts:
- host: something.com
ha:
enabled: true
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "consul" {
path = "vault/"
scheme = "https"
address = "HOST_IP:8500"
}
Error in vault pod which is unable to connect to consul agent.
kubectl logs vault-0 -n vault
WARNING! Unable to read storage migration status.
2021-06-28T08:13:13.041Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2021-06-28T08:13:13.042Z [WARN] storage migration check error: error="Get "https://127.0.0.1:8500/v1/kv/vault/core/migration": dial tcp 127.0.0.1:8500: connect: connection refused"
i am not sure if some configuration is missed in consul helm chart as i do not see any service running on port 8500 in consul namespace.
Any suggestion would be much appreciated.
Thanks,
pooja