Today quite different topic, using ZFS/NFS on FreeBSD as backend storage for containers/pods running on OpenShift platform. Serving storage for OpenShift is not as simple as OpenShift will not just consume NFS or iSCSI volumes “just like that” … OpenShift requires a special CSI driver in between to make that even work.My first association with CSI is of course CSI series like CSI: Miami … but the real name is Container Storage Interface of course :While the Democratic CSI driver is suited mostly for Linux based systems like TrueNAS SCALE for example – its zfs-generic-nfs driver also works with FreeBSD – and this is what we will use today. The FreeBSD setup is very simple – its just NFSv4 only server and a dedicated NFS share for OpenShift along with dedicated ZFS dataset. One can also use any regular user with added ZFS permissions done via zfs allow command or a regular user with sudo(8) to lift up the permissions.The FreeBSD/ZFS/NFS part was done my be – as I am not expert in the OpenShift domain – the OpenShift commands were done by luckyonesl and shared with his approval – thank You for help.FreeBSD Server ZFS/NFS SetupLatest FreeBSD 15.1-RELEASE was used for installation with Auto (ZFS) option. One can also use ready to download one of the FreeBSD VM-IMAGES that project also creates – in that case something ZFS based is needed. Just set the root password to something You will later use in the config. Using SSH keys instead of password is also possible. FreeBSD will use 10.0.0.9 IP address.freebsd # echo "something+VERY-insecure-123" | pw usermod -n root -h 0First create the ZFS dataset – as this is only for demonstration purposes I will just create zroot/openshift with /zroot/openshift as its mountpoint.freebsd # zfs create -o /zroot/openshift zroot/openshiftAs /etc/rc.conf is very basic – I will only focus on the NFS part here.nfsv4_server_enable=YESnfsv4_server_only=YESmountd_enable=YESnfs_server_enable=YESnfs_server_flags="-t -n 64"nfs_server_maxio=131072The /etc/exports file for the NFS share. The hosts 10.0.0.10-13 are the nodes of OpenShift cluster.V4: / -sec=sys/zroot/openshift -sec=sys -maproot=root 10.0.0.10 10.0.0.11 10.0.0.12 10.0.0.13Now start the NFS server.freebsd # service nfsd startOpenShift SetupNext are the needed OpenShift commands which were done by luckyonesl and shared with his approval as I am not that proficient in OpenShift.One can use privateKey: instead of password: for more security – this is only for demonstration purposes.Now the installation of Democratic CSI driver with helm(8) command.openshift # cat /root/democratic-csi-install.yamlimage: repository: ghcr.io/democratic-csi/democratic-csi tag: v1.9.3csiDriver: name: "org.democratic-csi.controller-zfs-generic"controller: hostNetwork: true dnsPolicy: ClusterFirstWithHostNetdriver: config: logLevel: debug driver: zfs-generic-nfs sshConnection: host: 10.0.0.9 port: 22 username: root password: something+VERY-insecure-123 zfs: cli: paths: zfs: /sbin/zfs zpool: /sbin/zpool sudo: /usr/local/bin/sudo datasetParentName: "zroot/openshift" detachedSnapshots: enabled: false datasetPermissionsMode: "0777" nfs: shareStrategy: "setDatasetProperties" shareStrategySetDatasetProperties: properties: sharenfs: "on" shareHost: 10.0.0.9storageClasses: - name: democratic-nfs defaultClass: false reclaimPolicy: Delete volumeBindingMode: Immediate allowVolumeExpansion: truevolumeSnapshotClasses: - name: democratic-nfs-snapshots parameters: detachedSnapshots: enabled: falseopenshift # helm repo add democratic-csi https://democratic-csi.github.io/charts/openshift # helm repo updateopenshift # helm upgrade \ --install democratic-csi democratic-csi/democratic-csi \ -n democratic-csi \ --create-namespace \ -f /root/democratic-csi-install.yamlNext check how the installation went.openshift # oc get deployment,ds -n democratic-csi -o yaml | grep -iE 'hostNetwork'openshift # oc get deployment,ds -n democratic-csi -o yaml | grep -iE 'hostNetwork|mountPropergation|privileged'Sometimes additional polices/privileges are needed – so here are the commands for them.openshift # oc adm policy add-scc-to-user privileged system:serviceaccount:democratic-csi:democratic-csi-controller-saopenshift # oc adm policy add-scc-to-user privileged system:serviceaccount:democratic-csi:democratic-csi-node-ssTests on OpenShiftNow the Democratic CSI driver seems to be installed – lets use it as storage for OpenShift containers and create some snapshot(s). First the YAML files that will be used.openshift # cat /root/csi-snapshot-test.yamlapiVersion: v1kind: Namespacemetadata: name: csi-test---apiVersion: v1kind: PersistentVolumeClaimmetadata: name: source-pvc namespace: csi-testspec: storageClassName: democratic-nfs accessModes: - ReadWriteMany resources: requests: storage: 1Gi---apiVersion: apps/v1kind: Deploymentmetadata: name: writer namespace: csi-testspec: replicas: 1 selector: matchLabels: app: writer template: metadata: labels: app: writer spec: containers: - name: writer image: busybox:1.36 command: - sh - -c - | mkdir -p /data echo "Hello from FreeBSD snapshot test." > /data/test.txt date >> /data/test.txt echo "Sleeping..." sleep 3600 volumeMounts: - name: storage mountPath: /data volumes: - name: storage persistentVolumeClaim: claimName: source-pvcopenshift # cat /root/restore-pvc.yamlapiVersion: v1kind: PersistentVolumeClaimmetadata: name: restored-pvc namespace: csi-testspec: accessModes: - ReadWriteMany storageClassName: democratic-nfs resources: requests: storage: 1Gi dataSource: name: source-snapshot kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.ioopenshift # cat /root/restore-reader.yamlapiVersion: apps/v1kind: Deploymentmetadata: name: restore-reader namespace: csi-testspec: replicas: 1 selector: matchLabels: app: restore-reader template: metadata: labels: app: restore-reader spec: containers: - name: shell image: alpine command: - /bin/sh - -c - sleep 3600 volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: restored-pvcopenshift # cat /root/snapshot.yamlapiVersion: snapshot.storage.k8s.io/v1kind: VolumeSnapshotmetadata: name: source-snapshot namespace: csi-testspec: volumeSnapshotClassName: democratic-nfs-snapshots source: persistentVolumeClaimName: source-pvc… and now to messing with them.openshift # oc apply -f /root/csi-snapshot-test.yamlopenshift # oc get pods -n csi-testopenshift # oc exec -n csi-test deploy/writer -- cat /data/test.txtopenshift # oc exec -n csi-test deploy/writer -- mount | grep /dataopenshift # oc apply -f /root/snapshot.yamlopenshift # oc exec -n csi-test deploy/writer -- sh -c 'echo after-snapshot >> /data/test.txt'openshift # oc get volumesnapshot -n csi-test -wopenshift # oc exec -n csi-test deploy/writer -- cat /data/test.txt openshift # oc scale deployment -n csi-test writer --replicas=0 openshift # oc apply -f /root/restore-pvc.yamlopenshift # oc apply -f /root/restore-reader.yamlopenshift # oc exec -ti deployment/restore-reader -n csi-test -- sh -c 'cat /data/test.txt'openshift # oc exec -ti deployment/writer -n csi-test -- sh -c 'cat /data/test.txt'Sorry that I do not have output of the commands but it was some fast try to find out if and how it will work on FreeBSD …Results on FreeBSDNow this is how the FreeBSD system changed within these OpenShift operations. Just after NFS server startup and before any of these OpenShift commands were executed the ZFS datasets and snapshots looked like that:freebsd # zfs list -t allNAME USED AVAIL REFER MOUNTPOINTzroot 20.3G 68.6G 96K /zrootzroot/ROOT 3.03G 68.6G 96K nonezroot/ROOT/default 3.03G 68.6G 3.03G /zroot/home 96K 68.6G 96K /homezroot/openshift 15.4G 68.6G 112K /zroot/openshiftzroot/snaps 96K 68.6G 96K /zroot/snapszroot/tmp 128K 68.6G 128K /tmpzroot/usr 1.88G 68.6G 96K /usrzroot/usr/ports 96K 68.6G 96K /usr/portszroot/usr/src 1.88G 68.6G 1.88G /usr/srczroot/var 1.41M 68.6G 96K /varzroot/var/audit 96K 68.6G 96K /var/auditzroot/var/crash 100K 68.6G 100K /var/crashzroot/var/log 880K 68.6G 880K /var/logzroot/var/mail 172K 68.6G 172K /var/mailzroot/var/tmp 96K 68.6G 96K /var/tmpNext – as OpenShift requested the storage new ZFS datasets have been created and also requested snapshots.freebsd # zfs list -t allNAME USED AVAIL REFER MOUNTPOINTzroot 20.3G 68.6G 96K /zrootzroot/ROOT 3.03G 68.6G 96K nonezroot/ROOT/default 3.03G 68.6G 3.03G /zroot/home 96K 68.6G 96K /homezroot/openshift 15.4G 68.6G 112K /zroot/openshiftzroot/openshift/pvc-5455ae63-a585-4865-a875-50f53e936a87 56K 68.6G 100K /zroot/openshift/pvc-5455ae63-a585-4865-a875-50f53e936a87zroot/openshift/pvc-de5515f3-8b0e-42ed-ba51-8ea7a347c815 15.4G 68.6G 15.4G /zroot/openshift/pvc-de5515f3-8b0e-42ed-ba51-8ea7a347c815zroot/openshift/pvc-de5515f3-8b0e-42ed-ba51-8ea7a347c815@snapshot-c96db74d-7ec2-420b-8fab-53a01119d4b7 68K - 100K -zroot/openshift/pvc-de5515f3-8b0e-42ed-ba51-8ea7a347c815@snapshot-4d5582dd-7e34-411f-924c-163487ba6160 68K - 9.77G -zroot/openshift/pvc-de5515f3-8b0e-42ed-ba51-8ea7a347c815@snapshot-1dc2ba05-668d-44dc-bd29-063600a85eaa 60K - 25.2G -zroot/snaps 96K 68.6G 96K /zroot/snapszroot/tmp 128K 68.6G 128K /tmpzroot/usr 1.88G 68.6G 96K /usrzroot/usr/ports 96K 68.6G 96K /usr/portszroot/usr/src 1.88G 68.6G 1.88G /usr/srczroot/var 1.41M 68.6G 96K /varzroot/var/audit 96K 68.6G 96K /var/auditzroot/var/crash 100K 68.6G 100K /var/crashzroot/var/log 880K 68.6G 880K /var/logzroot/var/mail 172K 68.6G 172K /var/mailzroot/var/tmp 96K 68.6G 96K /var/tmpfreebsd # find /zroot/openshift/zroot/openshift/zroot/openshift/pvc-de5515f3-8b0e-42ed-ba51-8ea7a347c815/zroot/openshift/pvc-de5515f3-8b0e-42ed-ba51-8ea7a347c815/test.txt/zroot/openshift-5455ae63-a585-4865-a875-50f53e936a87/zroot/openshift/pvc-5455ae63-a585-4865-a875-50f53e936a87/test.txtNot sure what else I should add here – as the topic is still fresh I will update and maybe add more things when they come.EOF