Blog/iSCSI target in user-space

From Forza's ramblings

iSCSI woes[edit | edit source]

Screenshot of targetcli with 4 iSCSI LUNs on Linux LIO

I have been using LIO with fileio backing store for several years, but have found it unreliable when clients restart/reboot without first disconnecting properly. Eventually we wound some problems with the ErrorRecoveryLevel=2 setting, but unfortunately fixing it did not resolve the instability. The Linux kernel would error out with BUG ON messages or hard crashes. We did test several different Linux distros over the years, all with similar issues. To be fair, the problem is not very common - which made it extremely difficult to debug. Perhaps fileio isn't a very well tested scenario for iSCSI.

So finally, I decided to search for some alternative iSCSI target servers that were easy to manage and well supported on my current Linux distribution. I came across tgt, an iSCSI target implemented completely in user-space. This means it is not reliant on an in-kernel module for its operation, and if there are issues, the entire server should not go down.

What is iSCSI?[edit | edit source]

An iSCSI LUN looks like a locally attached disk

Internet Small Computer Systems Interface or iSCSI, is a protocol for accessing disk storage over standard TCP/IP networks such as a Local Area Network (LAN) or the Internet. iSCSI allows a target (server) to export local storage devices such as, HDD, SSD or NVMe, to clients (initiators). To the initiator, the storage device would appear as if it was locally attached.

There are several server implementations for Linux. The most common one is LIO, which is the Linux kernel built-in option since around 2011. Other software targets are SCST, tgt and IET (seems mostly inactive).

Linux target framework (tgt)[edit | edit source]

The Linux target framework (tgt) is a user-space SCSI target framework that supports the iSCSI and iSER (RDMA) transport protocols and that supports multiple methods for accessing block storage. Tgt consists of a user-space daemon and tools to configure it.

On Alpine linux, the scsi-tgt and scsi-tgt-scripts packages are provided in the testing repo. A big thanks to @psykose who created the Alpine packages for me :)

The manual is a little sparse, but if you read the provided examples you quickly get the grasp on how it works. Basically there are two parts; the tgtd daemon and the tgt-admin tool to configure the daemon via configuration files. It is possible to configure a running daemon without tgt-admin using tgtadm, which is useful for scripting and performing on-line changes.

tgtd daemon[edit | edit source]

Normally you do not need to provide any arguments to tgtd, and can simply run it on the command line. It will run in the background by default and listen on TCP/IP port 3260 on all interfaces. On Alpine linux you'd use rc-service tgtd start, which provides logging and service supervision. Extra command-line options can be provided in /etc/conf.d/tgtd.

# tgtd --help
Linux SCSI Target framework daemon, version 1.0.86

Usage: tgtd [OPTION]
-f, --foreground        make the program run in the foreground
-D, --nodaemonize       make the program run in the foreground with logger
-C, --control-port NNNN use port NNNN for the mgmt channel
-t, --nr_iothreads NNNN specify the number of I/O threads
-p, --pid-file filename specify the pid file
-d, --debug debuglevel  print debugging information
-V, --version           print version and exit
-h, --help              display this help and exit

tgt-admin[edit | edit source]

tgt-admin is used to load and change configuration. Use the --execute switch to load the default configuration file from /etc/tgt/targets.conf and --show to display running configuration.

# tgt-admin --help
tgt-admin [OPTION]...
This tool configures tgt targets.

  -e, --execute                 read /etc/tgt/targets.conf and execute tgtadm commands
      --delete <value>          delete all or selected targets
                                (see "--delete help" for more info)
      --offline <value>         put all or selected targets in offline state
                                (see "--offline help" for more info)
      --ready <value>           put all or selected targets in ready state
                                (see "--ready help" for more info)
      --update <value>          update configuration for all or selected targets
                                (see "--update help" for more info)
  -s, --show                    show all the targets
  -C, --control-port <NNNN>     specify the control port to connect to
  -c, --conf <conf file>        specify an alternative configuration file
      --ignore-errors           continue even if tgtadm exits with non-zero code
  -f, --force                   force some operations even if the target is in use
  -p, --pretend                 only print tgtadm options
      --dump                    dump current tgtd configuration (note: does not
                                include detailed parameters, like write caching)
  -v, --verbose                 increase verbosity (show tgtadm commands)
  -h, --help                    show this help

Configuration example[edit | edit source]

The configuration is pretty straightforward once you understand the concepts. A target is defined as a <target>...</target> block with one or several devices (LUNs) exported via backing-store blocks.

A simple configuration without access control (open for anyone to connect) could look like this:

    backing-store /dev/LVM/somedevice1      # Becomes LUN 1
    backing-store /media/target/somefile    # Becomes LUN 2
    backing-store /media/target/anotherfile # Becomes LUN 3

Here we have one LVM device and two files exported as three separate SCSI devices, also known as LUNs. All settings are default and LUN numbers are automatically chosen. The iSCSI naming convention for initiators and targets is defined in RFC3720.

Windows automatically reconnects if the target is restarted.

If you want a bit more consistency and control we can expand the backing-store into <backing-store>..</backing-store blocks. This lets us define things like block-size and write-cache. I've also defined ErrorRecoveryLevel 2, which allows for transparent and automatic recovery after network interruptions or server restarts, if the initiator supports it.

    ## General settings
    controller_tid 1
    vendor_id Forza

    ## iSCSI features
    ErrorRecoveryLevel 2

    ## Access control
    incominguser user1 secretpass12

    ## iSCSI LUNs (exports)
    <backing-store "/media/target/diskimage_1.img">
        lun 1
        block-size 4096
        allow-in-use yes
        write-cache on
        scsi_sn 1001
        product_id MediaFiles
        params thin_provisioning=1

Once configuration is complete, simply run tgt-admin -e to load it. Now the target is active and accepts connections from remote initiators.

Windows iSCSI initiator, connecting to a password protected tgt target

We can see what initiators currently are connected using tgt-admin --show.

# tgt-admin --show
Target 1:
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 1
            Initiator: alias: none
            Connection: 1
                IP Address:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: 1001
            Size: 107374 MB, Block size: 4096
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: Yes
            Backing store type: rdwr
            Backing store path: /media/target/diskimage_1.img
            Backing store flags:
    Account information:
    ACL information:

tgt performance[edit | edit source]

From a performance point of view, tgt has done really well compared to LIO and the issues with the Linux kernel are no more.

For the purpose of this Blog entry, I set up tgt in Alpine Linux inside a qemu virtual machine with a plain file exported via virtiofs ontop of a plain HDD with Btrfs filesystem. It's pretty amazing performance considering the many layers.

If you have RDMA capable network cards, it should be possible to saturate very high speed networks without any problems.

CrystalDiskMark benchmark running on a Windows iSCSI Initiator connected to the tgt Linux target over a 1Gbit/s LAN.