Does windows minikube support a persistent volume with a hostpath? If so what is the syntax?
I tried:
apiVersion: v1
kind: PersistentVolume
metadata:
name: kbmongo002
labels:
type: local
spec:
storageClassName: mongostorageclass
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/temp/mongo"
persistentVolumeReclaimPolicy: Retain
---
This passed validation and created the PV and a PVC claimed it, but nothing was written to my expected location of C:\temp\mongo
I also tried:
hostPath:
path: "c:/temp/mongo"
persistentVolumeReclaimPolicy: Retain
---
That resulted in:
Error: Error response from daemon: Invalid bind mount spec
"c:/temp/mongo:/data/db": invalid mode: /data/db
Error syncing pod
If you use virtualbox in windows, only the c:/Users is mapped into vm as /c/Users which is kubernetes system can access. It is the feature in Virtualbox.
Minikube use VM to simulate the kubernetes VM.
Minikube provides mount feature as well, not so user-friendly for persitency.
You can try choose one of the solutions below
use folders under /c/Users for your yaml file
map extra folders into virtualbox VM like C:\Users
use minikube mount, see host folder mount
I have tried k8s hostpath on windows, it works well.
You should use drive letter in pod mount path, see example: https://github.com/andyzhangx/Demo/blob/master/windows/azuredisk/aspnet-pod-azuredisk.yaml#L14
As there is a docker mount path related bug on windows, you need to use drive letter as mount path in pod, see issue: https://github.com/moby/moby/issues/34729
Related
Cluster information:
Kubernetes version: 1.19.11
Cloud being used: Azure
Installation method: Manual creation in Azure online UI/Azure CLI
Host OS: Linux
CNI and version: Azure container networking interface, most recent
Hey everyone! I'm a relatively new user of Kubernetes, but I think I've got the basics down. I'm mainly trying to understand a more complex file share feature.
I’m essentially trying to use JupyterHub on Kubernetes for a shared development environment for a team of about a dozen users (we may expand this to larger/other teams later, but for now I want to get this working for just our team), and one feature that would be extremely helpful, and looks doable, is having a shared directory for notebooks, files, and data. I think I’m pretty close to getting this set-up, but I’m running into a mounting issue that I can’t quite resolve. I’ll quickly explain my setup first and then the issue. I’d really appreciate any help/comments/hints that anyone has!
Setup
Currently, all of this setup is on a Kubernetes cluster in Azure or other Azure-hosted services. We have a resource group with a kubernetes cluster, App Service Domain, DNS Zone, virtual network, container registry (for our custom docker images), and storage account. Everything works fine, except that in the storage account, I have an Azure NFS (and plain SMB if needed) file share that I’ve tried mounting via a PV and PVC to a JupyterHub server, but to no avail.
To create the PV, I set up an NFS file share in Azure and created the appropriate kubernetes secret as follows:
# Get storage account key
STORAGE_KEY=$(az storage account keys list --resource-group $resourceGroupName --account-name $storageAccountName --query "[0].value" -o tsv)
kubectl create secret generic azure-secret \
--from-literal=azurestorageaccountname=$storageAccountName \
--from-literal=azurestorageaccountkey=$STORAGE_KEY
I then tried to create the PV with this YAML file:
apiVersion: v1
kind: PersistentVolume
metadata:
name: shared-nfs-pv
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
azureFile:
secretName: azure-secret
shareName: aksshare
readOnly: false
nfs:
server: wintermutessd.file.core.windows.net:/wintermutessd/wintermutessdshare
path: /home/shared
readOnly: false
storageClassName: premium-nfs
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=1000
- gid=1000
- mfsymlinks
- nobrl
Issue
During the creation of the PV, I get the error Failed to create the persistentvolume 'shared-nfs-pv'. Error: Invalid (422) : PersistentVolume "shared-nfs-pv" is invalid: spec.azureFile: Forbidden: may not specify more than 1 volume type. Removing the azureFile options solves this error, but I feel like it would be necessary to specify the kubernetes secret that I created. If I do remove the azureFile options, it does successfully create and bind the PV. Then I created the corresponding PVC with
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shared-nfs-pvc
spec:
accessModes:
- ReadWriteMany
# Match name of PV
volumeName: shared-nfs-pv
storageClassName: premium-nfs
resources:
requests:
storage: 50Gi
which also successfully bound. However, when I add the configuration to my Helm config for JupyterHub with
singleuser:
storage:
extraVolumes:
- name: azure
persistentVolumeClaim:
claimName: azurefile
extraVolumeMounts:
- name: azure
mountPath: /home/shared
I get the following error when the jupyterhub server tries to spawn and mount the PVC:
Just in case this is relevant, the NFS azure file share is only accessible via a private endpoint, but this should be fine since my kubernetes cluster is running in the same virtual network. In fact, Azure tells me that I could just mount this NFS share on linux with
sudo apt-get -y update
sudo apt-get install nfs-common
sudo mkdir -p /mount/wintermutessd/wintermutessdshare
sudo mount -t nfs wintermutessd.file.core.windows.net:/wintermutessd/wintermutessdshare /mount/wintermutessd/wintermutessdshare -o vers=4,minorversion=1,sec=sys
But when I add this to my Dockerfile for the docker image that I'm using in my container, the build fails and tells me that systemctl isn't installed. Trying to add this through apt-get install systemd doesn't resolve the issue either.
From looking at other K8s discourse posts, I found this one ( File based data exchange between pods and daemon-set - General Discussions - Discuss Kubernetes) which looked helpful and has a useful link to deploying an NSF server, but I think the fact that my NFS server is an Azure file share makes this a slightly different scenario.
If anyone has any ideas or suggestions, I'd really appreciate it!
P.S. I had previously posted on the JupyterHub discourse here ( Mounting an SMB or NFT Azure File share onto JupyterHub on kubernetes for a shared directory - JupyterHub - Jupyter Community Forum), but it was suggested that my issue is more of a k8s issue rather than a JupyterHub one. I also looked at this other stackoverflow post, but, even though I am open to an SMB file share, it has to do more with VMs and not with PV/PVCs on kubernetes.
Thank you! :)
so I actually managed to figure this out using a dynamically allocated Azure file share. I'm writing an internal documentation for this, but I thought I'd post the relevant bit here. I hope this helps people!
Dynamically creating an Azure file share and storage account by defining a PVC and storage class
Here, we're mainly following the documentation for dynamically creating a PV with Azure Files in AKS. The general idea is to create a storage class that will define what kind of Azure file share we want to create (premium vs. standard and the different redundancy modes) and then create a PVC (persistent volume claim) that adheres to that storage class. Consequently, when JupyterHub tries to mount the PVC we created, it will automatically create a PV (persistent volume) for the PVC to bind to, which will then automatically create a storage account and file share for the PV to actually store filese in. This will all be done in the resource group that backs the one we're already using (these generally start with "MC_"). Here, we will be using the premium storage class with zone reduntant storage. First, create the storage class to be used (more info on the available tags here can be found in this repository) with the following YAML
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: shared-premium-azurefile
provisioner: kubernetes.io/azure-file
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=0
- gid=0
- mfsymlinks
- cache=strict
- actimeo=30
parameters:
skuName: Premium_ZRS
Name this file azure-file-sc.yaml and run
kubectl apply -f azure-file-sc.yaml
Next, we will create a PVC which will dynamically provision from our Azure file share (it automatically creates a PV for us). Create the file azure-file-pvc.yaml with the following code
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shared-premium-azurefile-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: shared-premium-azurefile
resources:
requests:
storage: 100Gi
and apply it with
kubectl apply -f azure-file-pvc.yaml
This will create the file share and the corresponding PV. We can check that our PVC and storage class were successfully created with
kubectl get storageclass
kubectl get pvc
It might take a couple of minutes for the PVC to bind.
On the Azure side, this is all that has to be done, and the dynamic allocation of the PV and file share are taken care of for us.
Mounting the PVC to JupyterHub in the home directory
JupyterHub, by default, creates a PVC of 10Gi for each new user, but we can also tell it to mount existing PVCs as external volumes (think of this as just plugging in your computer to a shared USB drive). To mount our previously created PVC in the home folder of all of our JupyterHub users, we simply add the following to our config.py Helm config:
singleuser:
storage:
extraVolumes:
- name: azure
persistentVolumeClaim:
claimName: shared-premium-azurefile-pvc
extraVolumeMounts:
- name: azure
mountPath: /home/jovyan/shared
Now, when JupyterHub starts up, all users should have a shared directory in their home folders with read and write permission.
I am trying to create a Kubernetes pod and mounting a volume from local hostpath. I am using Azure Kubernetes cluster. Following is my yaml for creating pod
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
volumeMounts:
- mountPath: /opt/myfolder
name: test-volume
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /Users/kkadam/minikube/myfolder
# this field is optional
I have few files under myfolder which I want to use inside container. Files are present in local volume but not inside container.
What could be an issue?
You can not add local path to container running on AKS. You have to add the file on specific node where POD is scheduled.
If both are on same node POD and files then you can mount the files as the volume to the container and use it.
However if your POD is schedule to another node then you will not be able to access the files inside the container.
If due to any reason your node restarted or deleted during auto-scaling you might lose the data.
Judging by what you said in your comment and your config, especially the path /Users/kkadam/minikube/myfolder which is typically a Mac OS path, it seems that you're trying to mount your local volume (probably your mac) in a pod deployed on AKS.
That's the problem.
In order to make it work, you need to put the files you're trying to mount on the node running your pod (which is in AKS).
OS: Windows 10
Kubernetes version: 1.14.8
Helm version: 3
Docker Desktop version: 2.1.0.5
Trying to deploy a Kubernetes cluster using a Helm-chart that contains a pod that connects to a statically provisioned Azure File Share.
Deploying to an Azure Kubernetes cluster works, but when we try to deploy the cluster locally on docker-desktop it gets the error message when trying to mount the share:
Unable to mount volumes for pod "": timeout expired waiting
for volumes to attach or mount for pod "". list of unmounted
volumes=[servicecatalog-persistent-storage]. list of unattached
volumes=[interactor-properties servicecatalog-persistent-storage
default-token-9fp7j]
Mounting arguments: -t cifs -o
username=,password=,file_mode=0777,dir_mode=0777,vers=3.0
//.file.core.windows.net/spps
/var/lib/kubelet/pods/44a70ebf-1b26-11ea-ab13-00155d0a4406/volumes/kubernetes.io~azure-file/servicecatalog-spp-pv
Output: mount error(11): Resource temporarily unavailable
Helm charts (removed redundant information):
Deployment:
apiVersion: apps/v1
kind: Deployment
spec:
spec:
containers:
- name: {{ .Release.Name }}-{{ .Chart.Name }}
volumeMounts:
- name: servicecatalog-persistent-storage
mountPath: /data/sppstore
volumes:
- name: servicecatalog-persistent-storage
persistentVolumeClaim:
claimName: servicecatalog-pv-claim
Persistent Storage / Claims:
apiVersion: v1
kind: PersistentVolume
metadata:
name: servicecatalog-spp-pv
labels:
usage: servicecatalog-spp-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
azureFile:
secretName: azurefile-secret
shareName: spps
readOnly: false
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: servicecatalog-pv-claim
annotations:
volume.beta.kubernetes.io/storage-class: ""
storageClass:
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
selector:
matchLabels:
usage: servicecatalog-spp-pv
Secret:
apiVersion: v1
kind: Secret
metadata:
name: azurefile-secret
type: Opaque
data:
azurestorageaccountname: <acc name>
azurestorageaccountkey:<acc key>
We have tried:
Using the Azure File Diagnostics to ensure ports are open and we are able to connect from our machine. link
Connecting using Azure Storage Explorer (works)
Microsoft says that connecting to an Azure File Share locally requires SMB 3.0 for security reasons which Windows 10 supports, but Kubernetes seems to use CIFS (which is a dialect of SMB?), but we cant figure out if its supported for access to Azure File Share. Any ideas?
The recommended way to mount an Azure file share on Linux is using SMB
3.0. By default, Azure Files requires encryption in transit, which is only supported by SMB 3.0. Azure Files also supports SMB 2.1, which
does not support encryption in transit, but you may not mount Azure
file shares with SMB 2.1 from another Azure region or on-premises for
security reasons.
https://learn.microsoft.com/en-us/azure/storage/files/storage-how-to-use-files-linux
so if you are using smb 2.1 you can only mount the file share from inside the same region. not from local workstation or from another azure region
since your cifs mount mentions vers=3.0 - I would assume this should work in your case. check storage account network access restrictions? or your network restrictions. say port 445, or other concerns mentioned in the linked article
I am moving our .net core(2.1) API from IIS to linux containers in kubernetes in GCP and am having some trouble with files being retrieved from the mounted fileserver when the filenames contain non ASCII characters. Similar to this GitHub issue, but these are standard UTF8 characters. åäö
var di = new DirectoryInfo(directory);
if (di.Exists)
{
var dFiles = di.GetFiles("*.pdf", SearchOption.TopDirectoryOnly);
Log.Information($"Files found in {di.Name} : {string.Join(",", dFiles.Select(f => f.Name))}");
//Log.Information($"Length of files found in {di.Name} : {string.Join(",", dFiles.Select(f => f.Length))}");
files.AddRange(dFiles);
}
bost�der.pdf is returned, but it should be bostäder.pdf.
In all other cases, characters are handled correctly, such as reading from the database. It is only when reading filenames that there is a problem. I get a FileNotFoundException when attempting to read the length of these files. LastWriteTime returns 1601-01-01.
The file is in GCP filestore and mounted to the kubernetes cluster using a PersistentVolume and PersistentVolumeClaim and then mounted to the containers.
I have tried to alter the currentthread culture when running this method, but with no luck. The same code works fine on the windows/IIS version using symlinks to point to the fileshare.
If I give the path of the file directly using new FileInfo rather than DirectoryInfo.GetFiles then the resulting FileInfo object returns the correctly encoded filename although the FileInfo.Length method still causes a FileNotFoundException.
Have used multiple Docker images
mcr.microsoft.com/dotnet/core/aspnet:2.1-bionic
mcr.microsoft.com/dotnet/core/aspnet:2.1-stretch-slim
Have also attempted to set the
ENV LC_ALL=sv_SE.UTF-8 \
LANG=sv_SE.UTF-8
in the Dockerfile with no luck. Any ideas what else I can try?
Edit:
I have now investigated further and the problem is not with .net, but with the mounted drive on the pods. Local files are shown correctly using ls, but the files on the mount are not. So it appears to be something to do with the way the drive is mounted.
I have a persistantvolume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: fileserver
spec:
capacity:
storage: 1T
accessModes:
- ReadWriteMany
nfs:
path: /mymount
server: 1.2.3.4
and a persistantvolumeclaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: fileserver-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: ""
resources:
requests:
storage: 1T
which is mounted to my pods using
volumeMounts:
- mountPath: /mnt/fileserver
name: pvc
and
volumes:
- name: pvc
persistentVolumeClaim:
claimName: fileserver-claim
readOnly: false
in my deployment.yaml
Edit 2:
Seems to be a problem with Cloud Filestore only supporting NFSv3 - is there any way that I could read these files using NFSv3
Lets say we have an application which accesses a file. This App is a jar which is packaged into an image and pushed to Registry for the Kubernetes to run it. But when we create the Pod, we need to configure a volume also in it. When we specify a volume we give a path, how do we place the file in that volume from lets say our virtual machine?
Please help me in understanding this with an explanation. Also should we create a storage so that its accessible from kubernetes cluster? please explain relevent topic as well to understand this.
Note: we are using azure cli
I think the best approach would be to create a ConfigMap with the data you want to use from your application. Then you just need to mount the ConfigMap as a volume in the Pod's (explained here) that need the data.
You can easily create a ConfigMap from a file like
kubectl create configmap your-configmap-name --from-file=/some/path/to/file
And then mount it in your Pod
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "ls /etc/config/" ]
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
# Provide the name of the ConfigMap containing the files you want
# to add to the container
name: special-config