iSCSI Volume Claims

Persistent Volumes provide a means of connecting a service with storage. Local SSDs will, of course be uncontested in terms of latency, but iSCSI allows a container to be scheduled on any compute node.

	Local Disk	iSCSI	NFSv3
Volume Claim Required	yes	yes	no
Authenticated	–	yes	no
Schedule on any Node	no	yes	yes
Concurrent Access	yes	no	yes
Native Linux File System	yes	yes	no
Uses Buffer Cache	yes	yes	no

Target

Using ZFS, create a vdev for each target/lun

Following the FreeBSD Network Services guide

# /etc/ctl.conf
auth-group ag0 {
    chap postgresql ************
}

portal-group pg0 {
    discovery-auth-group no-authentication
    listen 192.168.0.9
}

target iqn.2026-04.lan.fs1:target0 {
    auth-group ag0
    portal-group pg0

    lun 0 {
        path /dev/zvol/zpool2/iscsi/t0-lun0
        blocksize 4096
        size 100G
    }

    lun 1 {
        path /dev/zvol/zpool2/iscsi/t0-lun1
        blocksize 4096
        size 100G
    }
}

Initial connection test on Linux (without authentication)

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>~/www/eradman.com/Untitled.xhtml</title>
<meta name="Generator" content="Vim/9.2" />
<meta name="plugin-version" content="vim9.2_v1" />
<meta name="syntax" content="nginx" />
<meta name="settings" content="no_progress,use_xhtml,use_css,no_foldcolumn,expand_tabs,pre_wrap,prevent_copy=,use_input_for_pc=none" />
<meta name="colorscheme" content="default" />
<style type="text/css">

pre { white-space: pre-wrap; font-family: monospace; color: #000000; background-color: #ffffff; }
body { font-family: monospace; color: #000000; background-color: #ffffff; }
* { font-size: 1em; }
a { color: inherit; }
.Comment { color: #0000c0; }
.Constant { color: #c00000; }
.Special { color: #c000c0; }
.Type { color: #008000; }

</style>
</head>
<body>
<pre id='vimCodeElement'>
t="iqn.2026-04.lan.fs1:target0"
p="fs1.lan:3260"
iscsiadm -m node --targetname $t --portal $p -o new     # add connection
iscsiadm -m node -L all                                 # log in
iscsiadm -m node                                        # list
iscsiadm -m node --targetname $t --portal $p -u         # log out
iscsiadm -m node --targetname $t --portal $p -o delete  # remove connection


<span class="Comment"># /etc/ctl.conf</span>
auth-group ag0 {
    chap postgresql <span class="Special">*</span>***********
}

portal-group pg0 {
    discovery-auth-group no-authentication
    <span class="Type">listen</span> <span class="Special">192.168.0.9</span>
}

target iqn.<span class="Constant">2026</span>-<span class="Constant">04</span>.lan.fs1:target0 {
    auth-group ag0
    portal-group pg0

    lun <span class="Constant">0</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun0
        blocksize 4096
        size 100G
    }

    lun <span class="Constant">1</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun1
        blocksize 4096
        size 100G
    }
}

</pre>
</body>
</html>
<!-- vim: set foldmethod=manual : -->

Initiator

Kubernetes needs the initiator service running on each node:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>~/www/eradman.com/Untitled.xhtml</title>
<meta name="Generator" content="Vim/9.2" />
<meta name="plugin-version" content="vim9.2_v1" />
<meta name="syntax" content="nginx" />
<meta name="settings" content="no_progress,use_xhtml,use_css,no_foldcolumn,expand_tabs,pre_wrap,prevent_copy=,use_input_for_pc=none" />
<meta name="colorscheme" content="default" />
<style type="text/css">

pre { white-space: pre-wrap; font-family: monospace; color: #000000; background-color: #ffffff; }
body { font-family: monospace; color: #000000; background-color: #ffffff; }
* { font-size: 1em; }
a { color: inherit; }
.Comment { color: #0000c0; }
.Constant { color: #c00000; }
.Special { color: #c000c0; }
.Type { color: #008000; }

</style>
</head>
<body>
<pre>
dnf -qy install iscsi-initiator-utils
systemctl enable --now iscsid


<span class="Comment"># /etc/ctl.conf</span>
auth-group ag0 {
    chap postgresql <span class="Special">*</span>***********
}

portal-group pg0 {
    discovery-auth-group no-authentication
    <span class="Type">listen</span> <span class="Special">192.168.0.9</span>
}

target iqn.<span class="Constant">2026</span>-<span class="Constant">04</span>.lan.fs1:target0 {
    auth-group ag0
    portal-group pg0

    lun <span class="Constant">0</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun0
        blocksize 4096
        size 100G
    }

    lun <span class="Constant">1</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun1
        blocksize 4096
        size 100G
    }
}

</pre>
</body>
</html>
<!-- vim: set foldmethod=manual : -->

The volume claim will exect a device formatted with a file system without GPT or MBR patitions. Since the iSCSI device is connected on the kubernetes node it will appear as local device

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>~/www/eradman.com/Untitled.xhtml</title>
<meta name="Generator" content="Vim/9.2" />
<meta name="plugin-version" content="vim9.2_v1" />
<meta name="syntax" content="nginx" />
<meta name="settings" content="no_progress,use_xhtml,use_css,no_foldcolumn,expand_tabs,pre_wrap,prevent_copy=,use_input_for_pc=none" />
<meta name="colorscheme" content="default" />
<style type="text/css">

pre { white-space: pre-wrap; font-family: monospace; color: #000000; background-color: #ffffff; }
body { font-family: monospace; color: #000000; background-color: #ffffff; }
* { font-size: 1em; }
a { color: inherit; }
.Comment { color: #0000c0; }
.Constant { color: #c00000; }
.Special { color: #c000c0; }
.Type { color: #008000; }

</style>
</head>
<body>
<pre>
$ mount | grep sda
/dev/sda on /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.0.9:3260-iqn.2026-04.lan.fs1:target0-lun-0 type ext4 (rw,relatime,stripe=4)
/dev/sda on /var/lib/kubelet/pods/bb8a3569-ceb8-4ee3-a6d6-308194f07ff8/volumes/kubernetes.io~iscsi/iscsi0 type ext4 (rw,relatime,stripe=4)


<span class="Comment"># /etc/ctl.conf</span>
auth-group ag0 {
    chap postgresql <span class="Special">*</span>***********
}

portal-group pg0 {
    discovery-auth-group no-authentication
    <span class="Type">listen</span> <span class="Special">192.168.0.9</span>
}

target iqn.<span class="Constant">2026</span>-<span class="Constant">04</span>.lan.fs1:target0 {
    auth-group ag0
    portal-group pg0

    lun <span class="Constant">0</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun0
        blocksize 4096
        size 100G
    }

    lun <span class="Constant">1</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun1
        blocksize 4096
        size 100G
    }
}

</pre>
</body>
</html>
<!-- vim: set foldmethod=manual : -->

Volume Claims

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>~/www/eradman.com/Untitled.xhtml</title>
<meta name="Generator" content="Vim/9.2" />
<meta name="plugin-version" content="vim9.2_v1" />
<meta name="syntax" content="nginx" />
<meta name="settings" content="no_progress,use_xhtml,use_css,no_foldcolumn,expand_tabs,pre_wrap,prevent_copy=,use_input_for_pc=none" />
<meta name="colorscheme" content="default" />
<style type="text/css">

pre { white-space: pre-wrap; font-family: monospace; color: #000000; background-color: #ffffff; }
body { font-family: monospace; color: #000000; background-color: #ffffff; }
* { font-size: 1em; }
a { color: inherit; }
.Comment { color: #0000c0; }
.Constant { color: #c00000; }
.Special { color: #c000c0; }
.Type { color: #008000; }

</style>
</head>
<body>
<pre>
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: iscsi0
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteOnce
  iscsi:
    targetPortal: 192.168.0.9:3260
    iqn: iqn.2026-04.lan.fs1:target0
    lun: 0
    fsType: ext4
    readOnly: false
    chapAuthSession: true
    secretRef:
      name: postgresql-data
  claimRef:
    namespace: default
    name: pg2-pvc
  storageClassName: fs-vdev


<span class="Comment"># /etc/ctl.conf</span>
auth-group ag0 {
    chap postgresql <span class="Special">*</span>***********
}

portal-group pg0 {
    discovery-auth-group no-authentication
    <span class="Type">listen</span> <span class="Special">192.168.0.9</span>
}

target iqn.<span class="Constant">2026</span>-<span class="Constant">04</span>.lan.fs1:target0 {
    auth-group ag0
    portal-group pg0

    lun <span class="Constant">0</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun0
        blocksize 4096
        size 100G
    }

    lun <span class="Constant">1</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun1
        blocksize 4096
        size 100G
    }
}

</pre>
</body>
</html>
<!-- vim: set foldmethod=manual : -->

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>~/www/eradman.com/Untitled.xhtml</title>
<meta name="Generator" content="Vim/9.2" />
<meta name="plugin-version" content="vim9.2_v1" />
<meta name="syntax" content="nginx" />
<meta name="settings" content="no_progress,use_xhtml,use_css,no_foldcolumn,expand_tabs,pre_wrap,prevent_copy=,use_input_for_pc=none" />
<meta name="colorscheme" content="default" />
<style type="text/css">

pre { white-space: pre-wrap; font-family: monospace; color: #000000; background-color: #ffffff; }
body { font-family: monospace; color: #000000; background-color: #ffffff; }
* { font-size: 1em; }
a { color: inherit; }
.Comment { color: #0000c0; }
.Constant { color: #c00000; }
.Special { color: #c000c0; }
.Type { color: #008000; }

</style>
</head>
<body>
<pre>
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pg2-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: fs-vdev


<span class="Comment"># /etc/ctl.conf</span>
auth-group ag0 {
    chap postgresql <span class="Special">*</span>***********
}

portal-group pg0 {
    discovery-auth-group no-authentication
    <span class="Type">listen</span> <span class="Special">192.168.0.9</span>
}

target iqn.<span class="Constant">2026</span>-<span class="Constant">04</span>.lan.fs1:target0 {
    auth-group ag0
    portal-group pg0

    lun <span class="Constant">0</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun0
        blocksize 4096
        size 100G
    }

    lun <span class="Constant">1</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun1
        blocksize 4096
        size 100G
    }
}

</pre>
</body>
</html>
<!-- vim: set foldmethod=manual : -->

If claimRef is not specified, a PersistentVolumeClaim will be assigned to a PersistantVolume dynamically. This may appear to be correct, but will likely result in a confused state in the future.

Authentication is provided using a Secret provider

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>~/www/eradman.com/Untitled.xhtml</title>
<meta name="Generator" content="Vim/9.2" />
<meta name="plugin-version" content="vim9.2_v1" />
<meta name="syntax" content="nginx" />
<meta name="settings" content="no_progress,use_xhtml,use_css,no_foldcolumn,expand_tabs,pre_wrap,prevent_copy=,use_input_for_pc=none" />
<meta name="colorscheme" content="default" />
<style type="text/css">

pre { white-space: pre-wrap; font-family: monospace; color: #000000; background-color: #ffffff; }
body { font-family: monospace; color: #000000; background-color: #ffffff; }
* { font-size: 1em; }
a { color: inherit; }
.Comment { color: #0000c0; }
.Constant { color: #c00000; }
.Special { color: #c000c0; }
.Type { color: #008000; }

</style>
</head>
<body>
<pre>
---
apiVersion: v1
kind: Secret
metadata:
  name: postgresql-data
type: "kubernetes.io/iscsi-chap"
stringData:
  node.session.auth.username: postgresql
  node.session.auth.password: ************


<span class="Comment"># /etc/ctl.conf</span>
auth-group ag0 {
    chap postgresql <span class="Special">*</span>***********
}

portal-group pg0 {
    discovery-auth-group no-authentication
    <span class="Type">listen</span> <span class="Special">192.168.0.9</span>
}

target iqn.<span class="Constant">2026</span>-<span class="Constant">04</span>.lan.fs1:target0 {
    auth-group ag0
    portal-group pg0

    lun <span class="Constant">0</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun0
        blocksize 4096
        size 100G
    }

    lun <span class="Constant">1</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun1
        blocksize 4096
        size 100G
    }
}

</pre>
</body>
</html>
<!-- vim: set foldmethod=manual : -->

Example Application

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>~/www/eradman.com/Untitled.xhtml</title>
<meta name="Generator" content="Vim/9.2" />
<meta name="plugin-version" content="vim9.2_v1" />
<meta name="syntax" content="nginx" />
<meta name="settings" content="no_progress,use_xhtml,use_css,no_foldcolumn,expand_tabs,pre_wrap,prevent_copy=,use_input_for_pc=none" />
<meta name="colorscheme" content="default" />
<style type="text/css">

pre { white-space: pre-wrap; font-family: monospace; color: #000000; background-color: #ffffff; }
body { font-family: monospace; color: #000000; background-color: #ffffff; }
* { font-size: 1em; }
a { color: inherit; }
.Comment { color: #0000c0; }
.Constant { color: #c00000; }
.Special { color: #c000c0; }
.Type { color: #008000; }

</style>
</head>
<body>
<pre>
---
apiVersion: v1
kind: Service
metadata:
  name: pg2
spec:
  clusterIP: None
  ports:
  - name: psql
    protocol: TCP
    port: 5432
    targetPort: 8123
  selector:
    app: pg2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pg2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pg2
  template:
    metadata:
      labels:
        app: pg2
    spec:
      containers:
      - name: pg2
        image: docker.io/postgres:18-alpine
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: pg2-data
          mountPath: "/var/lib/postgresql"
      volumes:
      - name: pg2-data
        persistentVolumeClaim:
          claimName: pg2-pvc

<span class="Comment"># /etc/ctl.conf</span>
auth-group ag0 {
    chap postgresql <span class="Special">*</span>***********
}

portal-group pg0 {
    discovery-auth-group no-authentication
    <span class="Type">listen</span> <span class="Special">192.168.0.9</span>
}

target iqn.<span class="Constant">2026</span>-<span class="Constant">04</span>.lan.fs1:target0 {
    auth-group ag0
    portal-group pg0

    lun <span class="Constant">0</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun0
        blocksize 4096
        size 100G
    }

    lun <span class="Constant">1</span> {
        path /dev/zvol/zpool2/iscsi/t0-lun1
        blocksize 4096
        size 100G
    }
}

</pre>
</body>
</html>
<!-- vim: set foldmethod=manual : -->

Eric Radman : a Journal

iSCSI Volume Claims

Target

Initiator

Volume Claims

Example Application