Why mounted hostPath doesn't work on kubernetes of GKE - linux

Question

I deployed these 2 kinds of services on GKE. Just want to confirm if the nginx data been mounted to the host.
Yaml
Nginx deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: beats
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
volumeMounts:
- name: nginx-data
mountPath: /var/log/nginx
volumes:
- name: nginx-data
hostPath:
path: /var/lib/nginx-data
type: DirectoryOrCreate
Filebeat
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: beats
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.modules:
- module: nginx
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: false
templates:
- condition.contains:
kubernetes.namespace: beats
config:
- module: nginx
access:
enabled: true
var.paths: ["/var/lib/nginx-data/access.log*"]
subPath: access.log
tags: ["access"]
error:
enabled: true
var.paths: ["/var/lib/nginx-data/error.log*"]
subPath: error.log
tags: ["error"]
output.logstash:
hosts: ["logstash.beats.svc.cluster.local:5044"]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: beats
labels:
k8s-app: filebeat
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.10.0
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
subPath: filebeat.yml
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
- name: nginx-data
mountPath: /var/lib/nginx-data
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
- name: nginx-data
hostPath:
path: /var/lib/nginx-data
type: DirectoryOrCreate
Check deploy
Nginx
kubectl describe po nginx-658f45f77-dpflp -n beats
...
Volumes:
nginx-data:
Type: HostPath (bare host directory volume)
Path: /var/lib/nginx-data
HostPathType: DirectoryOrCreate
Filebeat pod
kubectl describe po filebeat-42wh7 -n beats
....
Volumes:
....
nginx-data:
Type: HostPath (bare host directory volume)
Path: /var/lib/nginx-data
HostPathType: DirectoryOrCreate
Check on nginx pod
# mount | grep nginx
/dev/sda1 on /var/log/nginx type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30)
/dev/sda1 on /var/cache/nginx type ext4 (rw,nosuid,nodev,relatime,commit=30)
root#nginx-658f45f77-dpflp:/# ls /var/log/nginx/
access.log error.log
Check on filebeat pod
# mount | grep nginx
/dev/sda1 on /var/lib/nginx-data type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30)
# ls /var/lib/nginx-data
(NULL)
The hostPath - /var/lib/nginx-data doesn't work. If use minikube, it can work. I can use minikube ssh to check the path on the host.
But on GKE, how can I check the hostPath on remote machine?

Answer 1

On GKE (and other hosted Kubernetes offerings from public-cloud providers) you can't directly connect to the nodes. You'll have to confirm using debugging tools like kubectl exec that content is getting from one pod to the other; since you're running filebeat as a DaemonSet, you'll need to check the specific pod that's running on the same node as the nginx pod.
The standard Docker Hub nginx image is configured to send its logs to the container stdout/stderr (more specifically, absent a volume mount, /var/log/nginx/access.log is a symlink to /proc/self/stdout). In a Kubernetes environment, the base log collector setup you show will be able to collect its logs. I'd just delete the customizations you're asking about in this question – don't create a hostPath directory, don't mount anything over the container's /var/log/nginx, and don't have special-case log collection for this one pod.