Btrfs/ENOSPC

From Forza's ramblings
(Redirected from Btrfs/No Diskspace Left)

ENOSPC - Out of disk space[edit | edit source]

Graph showing btrfs usage divided by block groups.

Unlike most conventional filesystems, BTRFS uses a two-stage allocator. The first stage allocates large regions of space known as chunks for specific types of data, then the second stage allocates blocks like a regular (old-fashioned) filesystem within these larger regions.

Btrfs combines chunks into three types of block groups:

Type Description
DATA Stores normal user file data
METADATA Stores internal metadata. Small files can also stored inline
SYSTEM Stores mapping between physical devices and the logical space representing the filesystem
UNALLOCATED Any unallocated space
Only the type of data that the chunk is allocated for can be stored in that block group.

The most common case these days when you get a -ENOSPC error on BTRFS is that the filesystem has run out of room for data or metadata in existing block groups, and that there is not enough unallocated space to allocate a new block group of correct type.

You can verify that this is the case by running btrfs filesystem usage on the filesystem that threw the error. If the Data or Metadata line shows a Total value that is significantly different from the Used value, then this is probably the cause.

# btrfs fi us /mnt/btrfs_vol
Overall:
    Device size:          14.01GiB
    Device allocated:      1.38GiB
    Device unallocated:   12.63GiB
    Device missing:        0.00B
    Used:                901.58MiB
    Free (estimated)      12.64GiB    (min: 6.33GiB)
    Data ratio:            1.00
    Metadata ratio:        2.00 
    Global reserve:        3.25MiB    (used: 0.00B)
    Multiple profiles:       no

Data,RAID0: Size:890.00MiB, Used:870.08MiB (97.76%)
   /dev/sdb1     445.00MiB
   /dev/sdc1     445.00MiB

Metadata,RAID1: Size:256.00MiB, Used:15.73MiB (6.15%)
   /dev/sdb1     256.00MiB
   /dev/sdc1     256.00MiB

System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/sdb1       8.00MiB
   /dev/sdc1       8.00MiB

Unallocated:
   /dev/sdb1      21.00MiB  <== Not enough space for another chunk of data or metatada in RAID profile.
   /dev/sdc1      12.61GiB

Preventing ENOSPC - Btrfs Balance[edit | edit source]

What btrfs balance does is to send things back through the allocator, which results in space usage in the chunks being compacted. For example, if you have two metadata chunks that are both 40% full, a balance will result in them becoming one metadata chunk that's 80% full. By compacting space usage like this, the balance operation is then able to delete the now empty chunks, and thus frees up room for the allocation of new chunks.

It is important to run btrfs balance before you run out of unallocated space. A common way is to set up balance as a scheduled maintenance task.

Only balance DATA chunks, never METADATA chunks.

Fixing ENOSPC errors[edit | edit source]

There have a few options to correct ENOSPC errors, depending on why they were caused.

  • ENOSPC caused by full filesystem.
  • ENOSPC caused by btrfs balance.
  • ENOSPC caused by an unbalanced filesystem.

Always do recovery options from a root shell, not from a GUI, file manager or similar.

Recovering from a full filesystem[edit | edit source]

Even if you balance data regularly, you can end up with a full disk by simply writing to many files to it. Writing files requires additional metadata and Btrfs may need to allocate additional metadata chunks to do that, which is not possible on a full filesystem.

When ENOSPC happens, Btrfs will change your filesystem to read-only to protect itself. This may seem counter-intuitive as it prevents you from deleting stuff.

  1. First, unmount your filesystem. We cannot operate on a read-only filesystem.
  2. Mount your filesystem again. It should now be read-write.
  3. Now you should delete enough files to create at least 1GiB free space for metadata allocation.

Please note that deleting snapshots, deleting ref-linked files and de-duped files can cause additional metadata allocation, which brings you back to ENOSPC again.

When you can't delete files without going ENOSPC[edit | edit source]

Only way to recover now is to add additional devices to your Btrfs filesystem.

Same steps as before:

  1. First, unmount your filesystem. We cannot operate on a read-only filesystem.
  2. Mount your filesystem again. It should now be read-write.

Now you need to find another device that you can add to your filesystem for a short period, for example a USB-stick.

Here, I assume your USB device is /dev/sdc1

  1. Add the USB stick to your filesystem: btrfs device add /dev/sdc1 /mnt/your-btrfs-mountpoint
  2. Now you should be able to delete files. Remove several GiB of files.
  3. When you have removed several GiB of files you can remove the USB-stick by doing btrfs device delete /dev/sdc1 /mnt/your-btrfs-mountpoint.

Important:

If you have a RAID1 filesystem, you will need to add two new devices.
If you have a RAID10 filesystem, you will need to add four new devices.

This is because data and metadata chunks are allocated according to the profile used.

Recovering from balance[edit | edit source]

Recovering from an unbalanced filesystem[edit | edit source]