Btrfs/Adding and removing devices

From Forza's ramblings
(Redirected from Btrfs/Removing a device)

Btrfs Volume Management

Photograph of a harddrive with its cover removed. The actuator and induvial platters are visible.
A 2.5" hard disk device with Serial-ATA connections.

Part of what makes Btrfs so flexible is its volume management features. It allows Btrfs to utilise and combine the disk space on several devices and use it a single filesystem.

It is also possible to combine several devices using different types of data and metadata profiles, usually called RAID modes. Read more about this on the Btrfs/Profiles page.

Adding devices to an existing filesystem

As with most Btrfs operations, the filesystem needs to be mounted before modifying it.

Adding a device is as simple as using btrfs device add <device> <mountpoint>.

# btrfs device add /dev/loop2p1 /mnt/my-btrfs/

Depending on the profile of the existing filesystem it may be necessary to balance the filesystem to take full advantage of the added disk space or enabling extra resiliency. See the example below.

When balance is needed

Balancing is needed when...

  • Adding a device to a RAID0, RAID10 or RAID5/6 filesystem.
  • Converting from single to a RAID profile.
  • Converting from DUP to a RAID profile.
  • Adding a device to a filesystem using a RAID profile which was very full.

Balancing is not needed when the allocator can fill up the remaining unallocated space (including the new devices) without leaving unusable space. The goal with balancing is to ensure that there is enough unallocated space on each device to satisfy the requirements of the given profile.

See the chapter Btrfs/ENOSPC on the concequences (and remedies) if there isn't enough unallocated space available.

Removing devices from a filesystem

Use btrfs device remove <device-or-devid> <mountpoint> to remove a device.

device and devid can be found using btrfs filesystem usage <mountpoint> or btrfs filesystem show <mountpoint>.

# btrfs filesystem show /mnt/my-btrfs/
Label: 'my-btrfs'  uuid:  01dc0ff9-b1e1-4930-a51a-1b0207674cbe
    Total devices 3 FS bytes used 1.08GiB                                           
    devid    1 size 10.00GiB used 1.28GiB path /dev/loop0p1
    devid    2 size 10.00GiB used 1.28GiB path /dev/loop1p1
    devid    3 size 10.00GiB used 2.00GiB path /dev/loop2p1

Any existing data on the removed device will automatically moved over to other devices in the same filesystem. This can take a considerable amount of time. Use btrfs filesystem usage <mountpoint> to monitor the progress.

Limitations when removing devices

  • All of the data must fit on the remaining devices.
  • It is not possible to go below the minimum number of devices given the data or metadata profile used. For example on a 3 device RAID1 filesystem it is possible to remove 1 device but not 2.

See more about the limitations of different profiles on the Btrfs/Profiles page.

Complete example of adding a second device to single disk filesystem

Let's create an example filesystem to work with.

Preparing new devices

Btrfs supports using raw devices without partition tables, however it is usually better to create a partition table first as it makes future management easier.

You can use fdisk or cfdisk from the util-linux package or GNU parted to create a partition table and a partion to hold your new btrfs filesystem. Use a GUID Partition Table (GPT) instead of the old DOS MBR style partition table. GPT supports devices larger than 2TiB and also stores a backup copy at the end of the device.

If you have an NVME or SSD disk, it is good practice to empty it using blkdiscard. Discard tells the drive's firmware that the disk is empty and it improves it's performance and wear. Do this before you create any partition tables as it will erase everything of the disk.

# blkdiscard /dev/loop4 -v
/dev/loop4: Discarded 10737418240 bytes from the offset 0

Here we use GNU parted to create a partition that fills the whole device /dev/loop4

# parted /dev/loop4
GNU Parted 3.4
Using /dev/loop4
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
(parted) mkpart primary btrfs 4MiB 100%
(parted) print
Model: Loopback device (loopback)
Disk /dev/loop4: 10.7GB
Sector size (logical/physical): 4096B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  10.7GB  10.7GB  btrfs        primary

(parted) quit
Information: You may need to update /etc/fstab.

Now the device is ready to be added to the filesystem.

# btrfs device add /dev/nvme0n4p1 /mnt/my-btrfs/
Performing full device TRIM /dev/nvme0n4p1 (10.00GiB) ...

Create an initial filesystem on a single device

# mkfs.btrfs /dev/loop0p1 -L my-btrfs
# mount /dev/loop0p1 /mnt/my-btrfs/

By default Btrfs creates a filesystem with single data and DUP metadata profiles on single device filesystem. We can see details about this newly created filesystem using btrfs filesystem usage <mountpoint>. Adding -T displays the output in table mode which can be easier to read when there are multiple devices in the filesystem.

# btrfs filesystem usage -T /mnt/my-btrfs/
Overall:
    Device size:                  10.00GiB
    Device allocated:            536.00MiB
    Device unallocated:            9.47GiB
    Device missing:                  0.00B
    Used:                        288.00KiB
    Free (estimated):              9.48GiB      (min: 4.74GiB)
    Free (statfs, df):             9.48GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:                3.25MiB      (used: 0.00B)
    Multiple profiles:                  no

                Data    Metadata  System
Id Path         single  DUP       DUP      Unallocated
-- ------------ ------- --------- -------- -----------
 1 /dev/loop0p1 8.00MiB 512.00MiB 16.00MiB     9.47GiB
-- ------------ ------- --------- -------- -----------
   Total        8.00MiB 256.00MiB  8.00MiB     9.47GiB
   Used           0.00B 128.00KiB 16.00KiB

We can see that there is only one device /dev/loop0 in this filesystem. We can also see that single data and DUP metadata profiles are used.

Let's add some data to the filesystem to see how it looks like.

# btrfs filesystem usage -T /mnt/my-btrfs/
Overall:
    Device size:                  10.00GiB
    Device allocated:              1.52GiB
    Device unallocated:            8.47GiB
    Device missing:                  0.00B
    Used:                        100.55MiB
    Free (estimated):              9.38GiB      (min: 5.15GiB)
    Free (statfs, df):             9.38GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:                3.25MiB      (used: 0.00B)
    Multiple profiles:                  no

                Data      Metadata  System
Id Path         single    DUP       DUP      Unallocated
-- ------------ --------- --------- -------- -----------
 1 /dev/loop0p1   1.01GiB 512.00MiB 16.00MiB     8.47GiB
-- ------------ --------- --------- -------- -----------
   Total          1.01GiB 256.00MiB  8.00MiB     8.47GiB
   Used         100.02MiB 256.00KiB 16.00KiB

Now we can see that a block group of 1GiB was created and of that, 100MiB of data is used.

Adding a second device to the filesystem

Adding a second device is done using btrfs device add <device> <mountpoint>

# btrfs device add /dev/loop1p1 /mnt/my-btrfs/
Performing full device TRIM /dev/loop1p1 (10.00GiB) ...
# btrfs filesystem usage /mnt/my-btrfs/
# btrfs filesystem usage -T /mnt/my-btrfs/
Overall:
    Device size:                  19.99GiB
    Device allocated:              1.52GiB
    Device unallocated:           18.47GiB
    Device missing:                  0.00B
    Used:                        100.55MiB
    Free (estimated):             19.38GiB      (min: 10.14GiB)
    Free (statfs, df):            19.37GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:                3.25MiB      (used: 0.00B)
    Multiple profiles:                  no

                Data      Metadata  System
Id Path         single    DUP       DUP      Unallocated
-- ------------ --------- --------- -------- -----------
 1 /dev/loop0p1   1.00GiB 512.00MiB 16.00MiB     8.48GiB
 2 /dev/loop1p1         -         -        -    10.00GiB
-- ------------ --------- --------- -------- -----------
   Total          1.00GiB 256.00MiB  8.00MiB    18.47GiB
   Used         100.02MiB 256.00KiB 16.00KiB

The second device is added but no data or metadata is used on it yet.

As more data is added to the filesystem, Btrfs will start allocating block groups on both devices. Btrfs allocation algorithm allocates new block groups on the device with most available unallocated space.

# btrfs filesystem usage -T /mnt/my-btrfs/
Overall:
    Device size:                  19.99GiB
    Device allocated:              2.52GiB
    Device unallocated:           17.47GiB
    Device missing:                  0.00B
    Used:                          1.08GiB
    Free (estimated):             18.40GiB      (min: 9.66GiB)
    Free (statfs, df):            18.40GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:                3.25MiB      (used: 0.00B)
    Multiple profiles:                  no

                Data    Metadata  System
Id Path         single  DUP       DUP      Unallocated
-- ------------ ------- --------- -------- -----------
 1 /dev/loop0p1 1.00GiB 512.00MiB 16.00MiB     8.48GiB
 2 /dev/loop1p1 1.00GiB         -        -     9.00GiB
-- ------------ ------- --------- -------- -----------
   Total        2.00GiB 256.00MiB  8.00MiB    17.47GiB
   Used         1.07GiB   1.58MiB 16.00KiB
NOTE: The metadata profile will remain as DUP after adding a device. It is important to convert the profile to RAID1 to take advantage of better resilience and healing ability of this mode. See Btrfs/Profiles for more information.

Converting DUP metadata profile to RAID1

When Btrfs is used on multiple devices, the recommended profile for metadata is RAID1. By default, Btrfs is using RAID1 metadata profile when using mkfs.btrfs on multiple devices, and DUP metadata profile on single devices.

As we can see from the example used earlier, the metadata profile remains as DUP after adding the second device. Btrfs supports conversion between profiles using the btrfs balance command.

# btrfs balance start -mconvert=raid1 /mnt/my-btrfs/
Done, had to relocate 2 out of 4 chunks
# btrfs filesystem usage -T /mnt/my-btrfs/
Overall:
    Device size:                  19.99GiB
    Device allocated:              2.56GiB
    Device unallocated:           17.43GiB
    Device missing:                  0.00B
    Used:                          1.08GiB
    Free (estimated):             18.35GiB      (min: 9.64GiB)
    Free (statfs, df):            18.35GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:                3.25MiB      (used: 0.00B)
    Multiple profiles:                  no

                Data    Metadata  System
Id Path         single  RAID1     RAID1    Unallocated
-- ------------ ------- --------- -------- -----------
 1 /dev/loop0p1 1.00GiB 256.00MiB 32.00MiB     8.71GiB
 2 /dev/loop1p1 1.00GiB 256.00MiB 32.00MiB     8.71GiB
-- ------------ ------- --------- -------- -----------
   Total        2.00GiB 256.00MiB 32.00MiB    17.43GiB
   Used         1.07GiB   1.58MiB 16.00KiB