I created a NATS Streaming Server on my Kubernetes cluster.
And "Kubectl get services" output like that:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
api-gateway-srv NodePort 10.106.100.181 <none> 8080:30440/TCP 16m
auth-mongo-srv ClusterIP 10.101.9.123 <none> 27017/TCP 16m
auth-srv ClusterIP 10.102.227.91 <none> 3000/TCP 16m
radio-srv ClusterIP 10.111.20.153 <none> 3003/TCP 16m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d13h
tv-srv ClusterIP 10.111.88.212 <none> 3001/TCP 16m
nats-srv ClusterIP 10.105.230.126 <none> 4222/TCP,8222/TCP 16m
On on my nats-publisher.js file like that:
const nats = require('node-nats-streaming');
const stan = nats.connect('natsserver', 'nats-cli1', {
url: 'nats://nats-srv:4222'
});
stan.on('connect', () => {
console.log('Links publisher connected to NATS')
}, (err, guid) => {
if(err) console.log(err)
else console.log(guid)
})
And I get :
NatsError: Could not connect to server: Error: connect ECONNREFUSED 10.105.230.126:4222
But on the another service I used same connection codes for nats connection. And this service can connect successfully nats server.
Why I getting this error?
Same code run as correctly on the another service but this code How can crash from this service?
When you use command kubectl port-forward [pod_name] 4222:4222
You can see next lines at the terminal:
Forwarding from 127.0.0.1:4222 -> 4222
Forwarding from [::1]:4222 -> 4222
Use 127.0.0.1:4222 in config
Related
Requesting for some help on : How to connect Vault with Consul agent on Kubernetes via Helm chart (Consul server is on Azure managed app)
I'm trying to build a POC, for Vault and consul and got some questions.
Deployed Azure managed app using - https://learn.hashicorp.com/tutorials/consul/hashicorp-consul-service-deploy
Installed consul agent on AKS with the steps in the https://learn.hashicorp.com/tutorials/consul/hashicorp-consul-service-aks?in=consul/hcs-azure
Consul helm chart : https://github.com/hashicorp/consul-helm
Installed vault via helm chart: https://github.com/hashicorp/vault-helm
Kubernetes services and pods for consul.
~$kubectl get svc -n consul
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul-connect-injector-svc ClusterIP 10.0.252.97 <none> 443/TCP 2d13h
consul-controller-webhook ClusterIP 10.0.169.80 <none> 443/TCP 2d13h
~$kubectl get pods -n consul
NAME READY STATUS RESTARTS AGE
consul-27j4j 1/1 Running 0 2d13h
consul-connect-injector-webhook-deployment-9454b8d68-778rd 1/1 Running 0 2d13h
consul-controller-7857456f99-mhzpw 1/1 Running 1 2d13h
consul-lkhpl 1/1 Running 0 2d13h
consul-webhook-cert-manager-cfbb689f7-fgtlw 1/1 Running 0 2d13h
consul-zf989 1/1 Running 0 2d13h
vault config as below:
ui:
enabled: true
serviceType: LoadBalancer
server:
ingress:
enabled: true
extraPaths:
- path: /
backend:
serviceName: vault-ui
servicePort: 8200
hosts:
- host: vault.something_masked.com
ha:
enabled: true
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "consul" {
path = "vault/"
scheme = "https"
address = "HOST_IP:8500"
}
Vault pods
kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 0/1 Running 0 7m14s
vault-1 0/1 Running 0 7m11s
vault-2 0/1 Running 0 7m11s
vault-agent-injector-cbbb6f4df-rmbd7 1/1 Running 0 7m22s
ERROR : Vault is unable to make communication with consul agent.
Logs for vault-0 pod
kubectl logs vault-0 -n vault
WARNING! Unable to read storage migration status.
2021-06-27T08:37:17.801Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2021-06-27T08:37:18.824Z [WARN] storage migration check error: error="Get "https://10.54.0.206:8500/v1/kv/vault/core/migration": dial tcp 10.54.0.206:8500: connect: connection refused"
Logs for vault-agent-injector pod
kubectl logs vault-agent-injector-cbbb6f4df-rmbd7 -n vault
2021-06-27T08:37:09.189Z [INFO] handler: Starting handler..
Listening on ":8080"...
2021-06-27T08:37:09.218Z [INFO] handler.auto-tls: Generated CA
2021-06-27T08:37:09.219Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-27T08:37:18.252Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=10s
2021-06-27T08:37:18.452Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=10s
Any suggestions or advice in the vault configuration if I have missed something ??
Thank you in advance.
Regards
Pooja
There are many points to debug the issue and the problem can be anywhere
the major issue I am seeing right now is your vault pods are running but not in a Ready state.
You have to unseal the vault from seal status.
Read more about the Unseal : https://learn.hashicorp.com/tutorials/vault/ha-with-consul#step-5-start-vault-and-verify-its-state
also how your consul is iteracting with vault ? Using LB, Ingress, service name ?
I am using consul azure managed app server and i have installed consul agent on aks.
kubectl get svc -n consul
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul-connect-injector-svc ClusterIP 10.0.252.97 <none> 443/TCP 3d13h
consul-controller-webhook ClusterIP 10.0.169.80 <none> 443/TCP 3d13h
as i am using consul agent i do not see consul-server running.
Helm chart config for consul
global:
enabled: false
name: consul
datacenter: dc1
acls:
manageSystemACLs: true
bootstrapToken:
secretName: XXX-sandbox-managed-app-bootstrap-token
secretKey: token
gossipEncryption:
secretName: XXX-sandbox-managed-app-hcs
secretKey: gossipEncryptionKey
tls:
enabled: true
enableAutoEncrypt: true
caCert:
secretName: XXX-sandbox-managed-app-hcs
secretKey: caCert
externalServers:
enabled: true
hosts:
['XXX.az.hashicorp.cloud']
httpsPort: 443
useSystemRoots: true
k8sAuthMethodHost: https://XXX.uksouth.azmk8s.io:443
client:
enabled: true
# If you are using Kubenet in your AKS cluster (the default network),
# uncomment the line below.
# exposeGossipPorts: true
join:
['XXX.az.hashicorp.cloud']
connectInject:
enabled: true
controller:
enabled: true
Helm chart config for vault
ui:
enabled: true
serviceType: LoadBalancer
server:
ingress:
enabled: true
extraPaths:
- path: /
backend:
serviceName: vault-ui
servicePort: 8200
hosts:
- host: something.com
ha:
enabled: true
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "consul" {
path = "vault/"
scheme = "https"
address = "HOST_IP:8500"
}
Error in vault pod which is unable to connect to consul agent.
kubectl logs vault-0 -n vault
WARNING! Unable to read storage migration status.
2021-06-28T08:13:13.041Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2021-06-28T08:13:13.042Z [WARN] storage migration check error: error="Get "https://127.0.0.1:8500/v1/kv/vault/core/migration": dial tcp 127.0.0.1:8500: connect: connection refused"
i am not sure if some configuration is missed in consul helm chart as i do not see any service running on port 8500 in consul namespace.
Any suggestion would be much appreciated.
Thanks,
pooja
I've sucessfully installed MetalLB on my Bare Metal Kubernetes cluster, but only pods assigned to the master Node seems to work.
MLB is configured on layer2, in the range of 192.168.0.100-192.168.0.200, and pods do get an IP when assigned to worker nodes, but those ips do not respond to any request.
If the assigned ip is curled inside the node, it works, yet if its curled from another node or machine, it doesnt respond.
Example:
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx2-658ffbbcb6-w5w28 1/1 Running 0 4m51s 10.244.1.2 worker2.homelab.com <none> <none>
nginx21-65b87bcbcb-fv856 1/1 Running 0 4h32m 10.244.0.10 master1.homelab.com <none> <none>
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h49m
nginx2 LoadBalancer 10.111.192.206 192.168.0.111 80:32404/TCP 5h21m
nginx21 LoadBalancer 10.108.222.125 192.168.0.113 80:31387/TCP 4h43m
# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master1.homelab.com Ready control-plane,master 5h50m v1.20.2 192.168.0.20 <none> CentOS Linux 7 (Core) 3.10.0-1160.15.2.el7.x86_64 docker://20.10.3
worker2.homelab.com Ready <none> 10m v1.20.2 192.168.0.22 <none> CentOS Linux 7 (Core) 3.10.0-1160.15.2.el7.x86_64 docker://20.10.3
Deployment nginx2 (Worker2, the one that doest work)
kubectl describe svc nginx2
Name: nginx2
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=nginx2
Type: LoadBalancer
IP: 10.111.192.206
LoadBalancer Ingress: 192.168.0.111
Port: http 80/TCP
TargetPort: 80/TCP
NodePort: http 32404/TCP
Endpoints: 10.244.1.2:80
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal nodeAssigned 10m (x6 over 5h23m) metallb-speaker announcing from node "master1.homelab.com"
Normal nodeAssigned 5m18s metallb-speaker announcing from node "worker2.homelab.com"
[root#worker2 ~]# curl 192.168.0.111
<!DOCTYPE html> ..... (Works)
[root#master1 ~]# curl 192.168.0.111
curl: (7) Failed connect to 192.168.0.111:80; No route to host
Deployment nginx21 (Master1, the one that works)
kubectl describe svc nginx21
Name: nginx21
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=nginx21
Type: LoadBalancer
IP: 10.108.222.125
LoadBalancer Ingress: 192.168.0.113
Port: http 80/TCP
TargetPort: 80/TCP
NodePort: http 31387/TCP
Endpoints: 10.244.0.10:80
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal nodeAssigned 12m (x3 over 4h35m) metallb-speaker announcing from node "master1.homelab.com"
[root#worker2 ~]# curl 192.168.0.113
<!DOCTYPE html> ..... (Works)
[root#master1 ~]# curl 192.168.0.113
<!DOCTYPE html> ..... (Works)
--------- PING WORKS FROM OTHER MACHINES ----------
I've just found out this, so it might be a problem with iptables? i dont really know how it works on MetalLB, i can ping the ip (192.168.0.111) from other machines and it responds
i figured out, after Matt response, it was the firewall that was blocking the access, so i just simply added the whole network to the port 80 and it worked.
[root#worker2 ~]# firewall-cmd --new-zone=kubernetes --permanent
success
[root#worker2 ~]# firewall-cmd --zone=kubernetes --add-source=192.168.0.1/16 --permanent
success
[root#worker2 ~]# firewall-cmd --zone=kubernetes --add-port=80/tcp --permanent
success
[root#worker2 ~]# firewall-cmd --reload
While trying to setup Cassandra database in a local Kubernetes cluster on a Mac OS (via Minikube), I am getting connection issues.
It seems like Node.js is not able to resolve DNS settings correctly, but resolving via command line DOES work.
The setup is as following (simplified):
Cassandra Service
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
type: NodePort
ports:
- port: 9042
targetPort: 9042
protocol: TCP
name: http
selector:
app: cassandra
In addition, there's a PersistentVolume and a StatefulSet.
The application itself is very basic
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: app1
labels:
app: app1
spec:
replicas: 1
selector:
matchLabels:
app: app1
template:
metadata:
labels:
app: app1
spec:
containers:
- name: app1
image: xxxx.dkr.ecr.us-west-2.amazonaws.com/acme/app1
imagePullPolicy: "Always"
ports:
- containerPort: 3003
And a service
apiVersion: v1
kind: Service
metadata:
name: app1
namespace: default
spec:
selector:
app: app1
type: NodePort
ports:
- port: 3003
targetPort: 3003
protocol: TCP
name: http
there also a simple ingress setup
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: dev.acme.com
http:
paths:
- path: /app1
backend:
serviceName: app1
servicePort: 3003
And added to /etc/hosts the minikube ip address
192.xxx.xx.xxx dev.acme.com
So far so good.
When trying to call dev.acme.com/app1 via Postman, the node.js app itself is being called correctly (can see in the logs), HOWEVER, the app can not connect to Cassandra and times out with the following error:
"All host(s) tried for query failed. First host tried,
92.242.140.2:9042: DriverError: Connection timeout. See innerErrors."
The IP 92.242.140.2 seems to be just a public IP that is related to my ISP, I believe since the app is not able to resolve the service name.
I created a simple node.js script to test dns:
var dns = require('dns')
dns.resolve6('cassandra', (err, res) => console.log('ERR:', err, 'RES:', res))
and the response is
ERR: { Error: queryAaaa ENOTFOUND cassandra
at QueryReqWrap.onresolve [as oncomplete] (dns.js:197:19) errno: 'ENOTFOUND', code: 'ENOTFOUND', syscall: 'queryAaaa', hostname:
'cassandra' } RES: undefined
However, and this is where it gets confusing - when I ssh into the pod (app1), I am able to connect to cassandra service using:
cqlsh cassandra 9042 --cqlversion=3.4.4
So it seems as the pod is "aware" of the service name, but node.js runtime is not.
Any idea what could cause the node.js to not being able to resolve the service name/dns settings?
UPDATE
After re-installing the whole cluster, including re-installing docker, kubectl and minikube I am getting the same issue.
While running ping cassandra from app1 container via ssh, I am getting the following
PING cassandra.default.svc.cluster.local (10.96.239.137) 56(84) bytes
of data. 64 bytes from cassandra.default.svc.cluster.local
(10.96.239.137): icmp_seq=1 ttl=61 time=27.0 ms
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
Which seems to be fine.
However, when running from Node.js runtime I am still getting the same error -
"All host(s) tried for query failed. First host tried,
92.242.140.2:9042: DriverError: Connection timeout. See innerErrors."
These are the services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
app1 ClusterIP None <none> 3003/TCP 11m
cassandra NodePort 10.96.239.137 <none> 9042:32564/TCP 38h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 38h
And these are the pods (all namespaces)
NAMESPACE NAME READY STATUS RESTARTS AGE
default app1-85d889db5-m977z 1/1 Running 0 2m1s
default cassandra-0 1/1 Running 0 38h
kube-system calico-etcd-ccvs8 1/1 Running 0 38h
kube-system calico-node-thzwx 2/2 Running 0 38h
kube-system calico-policy-controller-5bb4fc6cdc-cnhrt 1/1 Running 0 38h
kube-system coredns-86c58d9df4-z8pr4 1/1 Running 0 38h
kube-system coredns-86c58d9df4-zcn6p 1/1 Running 0 38h
kube-system default-http-backend-5ff9d456ff-84zb5 1/1 Running 0 38h
kube-system etcd-minikube 1/1 Running 0 38h
kube-system kube-addon-manager-minikube 1/1 Running 0 38h
kube-system kube-apiserver-minikube 1/1 Running 0 38h
kube-system kube-controller-manager-minikube 1/1 Running 0 38h
kube-system kube-proxy-jj7c4 1/1 Running 0 38h
kube-system kube-scheduler-minikube 1/1 Running 0 38h
kube-system kubernetes-dashboard-ccc79bfc9-6jtgq 1/1 Running 4 38h
kube-system nginx-ingress-controller-7c66d668b-rvxpc 1/1 Running 0 38h
kube-system registry-creds-x5bhl 1/1 Running 0 38h
kube-system storage-provisioner 1/1 Running 0 38h
UPDATE 2
The code to connect to Cassandra from Node.js:
const cassandra = require('cassandra-driver');
const client = new cassandra.Client({ contactPoints: ['cassandra:9042'], localDataCenter: 'datacenter1', keyspace: 'auth_server' });
const query = 'SELECT * FROM user';
client.execute(query, [])
.then(result => console.log('User with email %s', result.rows[0].email));
It DOES work when replacing cassandra:9042 with 10.96.239.137:9042 (10.69.239.137 is the ip address received from pinging cassandra via cli).
The Cassandra driver for Node.js uses resolve4/resolve6 to do its dns lookup, which bypasses your resolv.conf file. A program like ping uses resolv.conf to resolve 'cassandra' to 'cassandra.default.svc.cluster.local', the actual dns name assigned to your Cassandra service. For a more detailed explanation of name resolution in node.js see here.
The fix is simple, just pass in the full service name to your client:
const client = new cassandra.Client({ contactPoints: ['cassandra.default.svc.cluster.local:9042'], localDataCenter: 'datacenter1', keyspace: 'auth_server' });
Please help. I am trying to deploy a web API to Azure ACS Kubernetes cluster, it is a simple web API created in VSTS and the result should be like this: { "value1", "value2" }.
I plan to make the type as Cluster-IP but I want to test and access it first that is why this is LoadBalancer, the pods is running and no restart (I think it's good).
The guide I'm following is: Running Web API using Docker and Kubernetes
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 3d
sampleapi-service LoadBalancer 10.0.238.155 102.51.223.6 80:31676/TCP 1h
When I tried to browse the IP 102.51.223.6/api/values it says:
"This site can’t be reached"
service.yaml
kind: Service
apiVersion: v1
metadata:
name: sampleapi-service
labels:
name: sampleapi
app: sampleapi
spec:
selector:
name: sampleapi
ports:
- protocol: "TCP"
# Port accessible inside the cluster
port: 80
# Port to forwards inside the pod
targetPort: 80
# Port accessible oustide the cluster
#nodePort: 80
type: LoadBalancer
deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: sampleapi-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: sampleapi
spec:
containers:
- name: sampleapi
image: mycontainerregistry.azurecr.io/sampleapi:latest
ports:
- containerPort: 80
POD
Name: sampleapi-deployment-498305766-zzs2z
Namespace: default
Node: c103facs9001/10.240.0.4
Start Time: Fri, 27 Jul 2018 00:20:06 +0000
Labels: app=sampleapi
pod-template-hash=498305766
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"sampleapi-deployme
-498305766","uid":"d064a8e0-9132-11e8-b58d-0...
Status: Running
IP: 10.244.2.223
Controlled By: ReplicaSet/sampleapi-deployment-498305766
Containers:
sampleapi:
Container ID: docker://19d414c87ebafe1cc99d101ac60f1113533e44c24552c75af4ec197d3d3c9c53
Image: mycontainerregistry.azurecr.io/sampleapi:latest
Image ID: docker-pullable://mycontainerregistry.azurecr.io/sampleapi#sha256:9635a9df168ef76a6a27cd46cb15620d762657e9b57a5ac2514ba0b9a8f47a8d
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Fri, 27 Jul 2018 00:20:48 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mj5m1 (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
default-token-mj5m1:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mj5m1
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 50m default-scheduler Successfully assigned sampleapi-deployment-498305766-zzs2z to c103facs9001
Normal SuccessfulMountVolume 50m kubelet, c103facs9001 MountVolume.SetUp succeeded for volume "default-token-mj5m1"
Normal Pulling 49m kubelet, c103facs9001 pulling image "mycontainerregistry.azurecr.io/sampleapi:latest"
Normal Pulled 49m kubelet, c103facs9001 Successfully pulled image "mycontainerregistry.azurecr.io/sampleapi:latest"
Normal Created 49m kubelet, c103facs9001 Created container
Normal Started 49m kubelet, c103facs9001 Started container
It seems like to me that your service isn't set to a port on the container. You have your targetPort commented out. So the service is reachable on port 80 but the service doesn't know to target the pod on that port.
You will need to start the service which exposes the internal port to some external Ip:port that can be used in your browser to access the service. try this after deploying your deployment and service yml files:
kubectl service sampleapi-service
I have created a headless statefull service in kubernates. and cassandra db is running fine.
PS C:\> .\kubectl.exe get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cassandra None <none> 9042/TCP 50m
kubernetes 10.0.0.1 <none> 443/TCP 6d
PS C:\> .\kubectl.exe get pods
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 49m
cassandra-1 1/1 Running 0 48m
cassandra-2 1/1 Running 0 48m
I am running all this on minikube. From my laptop i am trying to connect to 192.168.99.100:9402 using a java program. But it is not able to connect.
Looks like your service not defined with NodePort. can you change service type to NodePort and test it.
when we define svc to NodePort we should get two port number for the service.