Btrfs/Adding and removing devices
Btrfs Volume Management[edit | edit source]
Part of what makes Btrfs so flexible is its volume management features. It allows Btrfs to utilise and combine the disk space on several devices and use it a single filesystem. Btrfs allows adding and removing devices from mounted filesystems without down-time, and it is possible to combine devices of different size. This makes it possible to gradually increase the filesystem size by adding additional devices over time.
For example, you have a 256GB SSD as root filesystem that is starting to get full. Btrfs makes it possible to simply add another SSD and combine the space of both devices, without having to re-create the filesystem.
Btrfs supports different types of profiles (usually called RAID modes), when combining devices. It is also possible to use different profiles for METADATA and DATA chunks. See the Btrfs/Profiles page.
Adding devices to an existing filesystem[edit | edit source]
As with most Btrfs operations, the filesystem needs to be mounted before modifying it.
Adding a device is as simple as using btrfs device add <device> <mountpoint>.
# btrfs device add /dev/loop2p1 /mnt/my-btrfs/
Depending on the profile of the existing filesystem it may be necessary to balance the filesystem to take full advantage of the added disk space or enabling extra resiliency. See the example below.
When balance is needed[edit | edit source]
When making changes to the disk layout by adding or removing devices, it is usually necessary to balance the filesystem so that block groups are spread out evenly, utilising all available space. Balancing is not needed when the allocator can fill up the remaining unallocated space (including the new devices) without leaving unusable space. The goal with balancing is to ensure that there is enough unallocated space on each device to satisfy the requirements of the given profile.
Balancing is needed when...[edit | edit source]
- Adding a device to a RAID0, RAID10 or RAID5/6 filesystem.
- Converting from single to a RAID profile.
- Converting from DUP to a RAID profile.
- Adding a device to a filesystem using a RAID profile which was very full.
One example is if you have 2 device RAID1 filesystem and add a 3rd device. A balance s needed to spread out the block groups evenly among the three devices. Otherwise the filesystem may end up with no free disk space (Btrfs/ENOSPC) error even though there is available space on the new device. This is because RAID1 requires free space on two devices for each new block group allocation.
Another example if you have a 2 device RAID0 filesystem and add a 3rd device. A balance is needed to convert from a 2-striped filesystem to a 3-striped filesystem. Without balancing, old data will still only be striped over the two old devices.
If you have a single device filesystem and simply want to add another device, no balance is needed for normal operation. However, it is recommended to convert METADATA to RAID1 instead of DUP. This will increase resiliency against metadata corruption for free as RAID1 does not cost any extra disk space than DUP.
Removing devices from a filesystem[edit | edit source]
Use btrfs device remove <device-or-devid> <mountpoint> to remove a device.
device and devid can be found using btrfs filesystem usage <mountpoint> or btrfs filesystem show <mountpoint>.
# btrfs filesystem show /mnt/my-btrfs/
Label: 'my-btrfs' uuid: 01dc0ff9-b1e1-4930-a51a-1b0207674cbe
Total devices 3 FS bytes used 1.08GiB
devid 1 size 10.00GiB used 1.28GiB path /dev/loop0p1
devid 2 size 10.00GiB used 1.28GiB path /dev/loop1p1
devid 3 size 10.00GiB used 2.00GiB path /dev/loop2p1
Any existing data on the removed device will automatically moved over to other devices in the same filesystem. This can take a considerable amount of time. Use btrfs filesystem usage <mountpoint> to monitor the progress.
Limitations when removing devices[edit | edit source]
- All of the data must fit on the remaining devices.
- It is not possible to go below the minimum number of devices given the data or metadata profile used. For example on a 3 device RAID1 filesystem it is possible to remove 1 device but not 2.
See more about the limitations of different profiles on the Btrfs/Profiles page.
Replacing an existing device[edit | edit source]
Btrfs has a dedicated method to replace a device. Instead of adding a new device and then remove the old, btrfs replace should be used. It is several times faster and can also handle corruptions and problems better than btrfs device remove.
Btrfs replace works with all profiles and can be used with single device and multi device filesystems.
There is a dedicated page for Btrfs replace with more information.
Complete example of adding a second device to single disk filesystem[edit | edit source]
Let's create an example filesystem to work with.
Preparing new devices[edit | edit source]
Btrfs supports using raw devices without partition tables, however it is usually better to create a partition table first as it makes future management easier.
You can use fdisk or cfdisk from the util-linux package or GNU parted to create a partition table and a partion to hold your new btrfs filesystem. Use a GUID Partition Table (GPT) instead of the old DOS MBR style partition table. GPT supports devices larger than 2TiB and also stores a backup copy at the end of the device.
If you have an NVME or SSD disk, it is good practice to empty it using blkdiscard. Discard tells the drive's firmware that the disk is empty and it improves it's performance and wear. Do this before you create any partition tables as it will erase everything of the disk.
# blkdiscard /dev/loop4 -v
/dev/loop4: Discarded 10737418240 bytes from the offset 0
Here we use GNU parted to create a partition that fills the whole device /dev/loop4
# parted /dev/loop4
GNU Parted 3.4 Using /dev/loop4 Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt (parted) mkpart primary btrfs 4MiB 100% (parted) print Model: Loopback device (loopback) Disk /dev/loop4: 10.7GB Sector size (logical/physical): 4096B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 4194kB 10.7GB 10.7GB btrfs primary (parted) quit Information: You may need to update /etc/fstab.
Now the device is ready to be added to the filesystem.
# btrfs device add /dev/nvme0n4p1 /mnt/my-btrfs/
Performing full device TRIM /dev/nvme0n4p1 (10.00GiB) ...
Create an initial filesystem on a single device[edit | edit source]
# mkfs.btrfs /dev/loop0p1 -L my-btrfs # mount /dev/loop0p1 /mnt/my-btrfs/
By default Btrfs creates a filesystem with single data and DUP metadata profiles on single device filesystem. We can see details about this newly created filesystem using btrfs filesystem usage <mountpoint>. Adding -T displays the output in table mode which can be easier to read when there are multiple devices in the filesystem.
# btrfs filesystem usage -T /mnt/my-btrfs/
Overall:
Device size: 10.00GiB
Device allocated: 536.00MiB
Device unallocated: 9.47GiB
Device missing: 0.00B
Used: 288.00KiB
Free (estimated): 9.48GiB (min: 4.74GiB)
Free (statfs, df): 9.48GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 3.25MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path single DUP DUP Unallocated
-- ------------ ------- --------- -------- -----------
1 /dev/loop0p1 8.00MiB 512.00MiB 16.00MiB 9.47GiB
-- ------------ ------- --------- -------- -----------
Total 8.00MiB 256.00MiB 8.00MiB 9.47GiB
Used 0.00B 128.00KiB 16.00KiB
We can see that there is only one device /dev/loop0 in this filesystem. We can also see that single data and DUP metadata profiles are used.
Let's add some data to the filesystem to see how it looks like.
# btrfs filesystem usage -T /mnt/my-btrfs/
Overall:
Device size: 10.00GiB
Device allocated: 1.52GiB
Device unallocated: 8.47GiB
Device missing: 0.00B
Used: 100.55MiB
Free (estimated): 9.38GiB (min: 5.15GiB)
Free (statfs, df): 9.38GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 3.25MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path single DUP DUP Unallocated
-- ------------ --------- --------- -------- -----------
1 /dev/loop0p1 1.01GiB 512.00MiB 16.00MiB 8.47GiB
-- ------------ --------- --------- -------- -----------
Total 1.01GiB 256.00MiB 8.00MiB 8.47GiB
Used 100.02MiB 256.00KiB 16.00KiB
Now we can see that a block group of 1GiB was created and of that, 100MiB of data is used.
Adding a second device to the filesystem[edit | edit source]
Adding a second device is done using btrfs device add <device> <mountpoint>
# btrfs device add /dev/loop1p1 /mnt/my-btrfs/
Performing full device TRIM /dev/loop1p1 (10.00GiB) ...
# btrfs filesystem usage /mnt/my-btrfs/
# btrfs filesystem usage -T /mnt/my-btrfs/
Overall:
Device size: 19.99GiB
Device allocated: 1.52GiB
Device unallocated: 18.47GiB
Device missing: 0.00B
Used: 100.55MiB
Free (estimated): 19.38GiB (min: 10.14GiB)
Free (statfs, df): 19.37GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 3.25MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path single DUP DUP Unallocated
-- ------------ --------- --------- -------- -----------
1 /dev/loop0p1 1.00GiB 512.00MiB 16.00MiB 8.48GiB
2 /dev/loop1p1 - - - 10.00GiB
-- ------------ --------- --------- -------- -----------
Total 1.00GiB 256.00MiB 8.00MiB 18.47GiB
Used 100.02MiB 256.00KiB 16.00KiB
The second device is added but no data or metadata is used on it yet.
As more data is added to the filesystem, Btrfs will start allocating block groups on both devices. Btrfs allocation algorithm allocates new block groups on the device with most available unallocated space.
# btrfs filesystem usage -T /mnt/my-btrfs/
Overall:
Device size: 19.99GiB
Device allocated: 2.52GiB
Device unallocated: 17.47GiB
Device missing: 0.00B
Used: 1.08GiB
Free (estimated): 18.40GiB (min: 9.66GiB)
Free (statfs, df): 18.40GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 3.25MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path single DUP DUP Unallocated
-- ------------ ------- --------- -------- -----------
1 /dev/loop0p1 1.00GiB 512.00MiB 16.00MiB 8.48GiB
2 /dev/loop1p1 1.00GiB - - 9.00GiB
-- ------------ ------- --------- -------- -----------
Total 2.00GiB 256.00MiB 8.00MiB 17.47GiB
Used 1.07GiB 1.58MiB 16.00KiB
metadata profile will remain as DUP after adding a device. It is important to convert the profile to RAID1 to take advantage of better resilience and healing ability of this mode. See Btrfs/Profiles for more information.Converting DUP metadata profile to RAID1[edit | edit source]
When Btrfs is used on multiple devices, the recommended profile for metadata is RAID1. By default, Btrfs is using RAID1 metadata profile when using mkfs.btrfs on multiple devices, and DUP metadata profile on single devices.
As we can see from the example used earlier, the metadata profile remains as DUP after adding the second device. Btrfs supports conversion between profiles using the btrfs balance command.
# btrfs balance start -mconvert=raid1 /mnt/my-btrfs/
Done, had to relocate 2 out of 4 chunks
# btrfs filesystem usage -T /mnt/my-btrfs/
Overall:
Device size: 19.99GiB
Device allocated: 2.56GiB
Device unallocated: 17.43GiB
Device missing: 0.00B
Used: 1.08GiB
Free (estimated): 18.35GiB (min: 9.64GiB)
Free (statfs, df): 18.35GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 3.25MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path single RAID1 RAID1 Unallocated
-- ------------ ------- --------- -------- -----------
1 /dev/loop0p1 1.00GiB 256.00MiB 32.00MiB 8.71GiB
2 /dev/loop1p1 1.00GiB 256.00MiB 32.00MiB 8.71GiB
-- ------------ ------- --------- -------- -----------
Total 2.00GiB 256.00MiB 32.00MiB 17.43GiB
Used 1.07GiB 1.58MiB 16.00KiB
