Blog/iSCSI target in user-space
iSCSI woes[edit | edit source]
I have been using LIO with fileio backing store for several years, but have found it unreliable when clients restart/reboot without first disconnecting properly. Eventually we wound some problems with the ErrorRecoveryLevel=2 setting, but unfortunately fixing it did not resolve the instability. The Linux kernel would error out with BUG ON messages or hard crashes. We did test several different Linux distros over the years, all with similar issues. To be fair, the problem is not very common - which made it extremely difficult to debug. Perhaps fileio isn't a very well tested scenario for iSCSI.
So finally, I decided to search for some alternative iSCSI target servers that were easy to manage and well supported on my current Linux distribution. I came across tgt, an iSCSI target implemented completely in user-space. This means it is not reliant on an in-kernel module for its operation, and if there are issues, the entire server should not go down.
What is iSCSI?[edit | edit source]
Internet Small Computer Systems Interface or iSCSI, is a protocol for accessing disk storage over standard TCP/IP networks such as a Local Area Network (LAN) or the Internet. iSCSI allows a target (server) to export local storage devices such as, HDD, SSD or NVMe, to clients (initiators). To the initiator, the storage device would appear as if it was locally attached.
There are several server implementations for Linux. The most common one is LIO, which is the Linux kernel built-in option since around 2011. Other software targets are SCST, tgt and IET (seems mostly inactive).
Linux target framework (tgt)[edit | edit source]
The Linux target framework (tgt) is a user-space SCSI target framework that supports the iSCSI and iSER (RDMA) transport protocols and that supports multiple methods for accessing block storage. Tgt consists of a user-space daemon and tools to configure it.
On Alpine linux, the scsi-tgt and scsi-tgt-scripts packages are provided in the testing repo. A big thanks to @psykose who created the Alpine packages for me :)
The manual is a little sparse, but if you read the provided examples you quickly get the grasp on how it works. Basically there are two parts; the tgtd daemon and the tgt-admin tool to configure the daemon via configuration files. It is possible to configure a running daemon without tgt-admin using tgtadm, which is useful for scripting and performing on-line changes.
tgtd daemon[edit | edit source]
Normally you do not need to provide any arguments to tgtd, and can simply run it on the command line. It will run in the background by default and listen on TCP/IP port 3260 on all interfaces. On Alpine linux you'd use rc-service tgtd start, which provides logging and service supervision. Extra command-line options can be provided in /etc/conf.d/tgtd.
# tgtd --help
Linux SCSI Target framework daemon, version 1.0.86 Usage: tgtd [OPTION] -f, --foreground make the program run in the foreground -D, --nodaemonize make the program run in the foreground with logger -C, --control-port NNNN use port NNNN for the mgmt channel -t, --nr_iothreads NNNN specify the number of I/O threads -p, --pid-file filename specify the pid file -d, --debug debuglevel print debugging information -V, --version print version and exit -h, --help display this help and exit
tgt-admin[edit | edit source]
tgt-admin is used to load and change configuration. Use the --execute switch to load the default configuration file from /etc/tgt/targets.conf and --show to display running configuration.
# tgt-admin --help
Usage:
tgt-admin [OPTION]...
This tool configures tgt targets.
-e, --execute read /etc/tgt/targets.conf and execute tgtadm commands
--delete <value> delete all or selected targets
(see "--delete help" for more info)
--offline <value> put all or selected targets in offline state
(see "--offline help" for more info)
--ready <value> put all or selected targets in ready state
(see "--ready help" for more info)
--update <value> update configuration for all or selected targets
(see "--update help" for more info)
-s, --show show all the targets
-C, --control-port <NNNN> specify the control port to connect to
-c, --conf <conf file> specify an alternative configuration file
--ignore-errors continue even if tgtadm exits with non-zero code
-f, --force force some operations even if the target is in use
-p, --pretend only print tgtadm options
--dump dump current tgtd configuration (note: does not
include detailed parameters, like write caching)
-v, --verbose increase verbosity (show tgtadm commands)
-h, --help show this help
Configuration example[edit | edit source]
The configuration is pretty straightforward once you understand the concepts. A target is defined as a <target>...</target> block with one or several devices (LUNs) exported via backing-store blocks.
A simple configuration without access control (open for anyone to connect) could look like this:
<target iqn.2008-09.com.example:server.target3>
backing-store /dev/LVM/somedevice1 # Becomes LUN 1
backing-store /media/target/somefile # Becomes LUN 2
backing-store /media/target/anotherfile # Becomes LUN 3
</target>
Here we have one LVM device and two files exported as three separate SCSI devices, also known as LUNs. All settings are default and LUN numbers are automatically chosen. The iSCSI naming convention for initiators and targets is defined in RFC3720.
If you want a bit more consistency and control we can expand the backing-store into <backing-store>..</backing-store blocks. This lets us define things like block-size and write-cache. I've also defined ErrorRecoveryLevel 2, which allows for transparent and automatic recovery after network interruptions or server restarts, if the initiator supports it.
<target iqn.2023-07.net.tnonline.wiki:server.target1>
## General settings
controller_tid 1
vendor_id Forza
## iSCSI features
ErrorRecoveryLevel 2
## Access control
initiator-address 192.168.0.10
incominguser user1 secretpass12
## iSCSI LUNs (exports)
<backing-store "/media/target/diskimage_1.img">
lun 1
block-size 4096
allow-in-use yes
write-cache on
scsi_sn 1001
product_id MediaFiles
params thin_provisioning=1
</backing-store>
</target>
Once configuration is complete, simply run tgt-admin -e to load it. Now the target is active and accepts connections from remote initiators.
We can see what initiators currently are connected using tgt-admin --show.
# tgt-admin --show
Target 1: iqn.2023-07.net.tnonline.wiki:server.target1
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 1
Initiator: iqn.1991-05.com.microsoft:studio alias: none
Connection: 1
IP Address: 192.168.0.10
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: 1001
Size: 107374 MB, Block size: 4096
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: Yes
Backing store type: rdwr
Backing store path: /media/target/diskimage_1.img
Backing store flags:
Account information:
user1
ACL information:
192.168.0.10
tgt performance[edit | edit source]
From a performance point of view, tgt has done really well compared to LIO and the issues with the Linux kernel are no more.
For the purpose of this Blog entry, I set up tgt in Alpine Linux inside a qemu virtual machine with a plain file exported via virtiofs ontop of a plain HDD with Btrfs filesystem. It's pretty amazing performance considering the many layers.
If you have RDMA capable network cards, it should be possible to saturate very high speed networks without any problems.
