Btrfs/Deduplication

From Forza's ramblings

Reflinks[edit | edit source]

Valves
Valves at the switch house east at Landschaftspark Duisburg-Nord, Duisburg, North Rhine-Westphalia, Germany (2020)

A rather unique feature of Btrfs is the concept of cloning files in an atomic way. This usually called a reflink.

This allows the user to make an instant copy of a file, even if it's several Gib large. It is similar to a hard link, but with a big advantage that the two files remain separate. When the original file or the copy is modified, Copy-on-Write (CoW), ensures that the files remain unique from each other.

File cloning (reflink, copy-on-write) via cp:

cp --reflink <source file> <destination file>
Tip: Put an alias in your .bash_profile or /etc/profile.d/ for cp to always do reflinks. Blog/Bash Aliases.

Reflinking files can cross subvolumes on the same filesystem but not cross between different filesystems.

A limitation with the Linux VFS layer is that in order for reflinks to work between subvolumes, they have to be visible from the same mount point.

Below is a list of of mount-points. The mounts in /media/xxx are subvolumes of the same filesystem on /dev/sdb2.

List of mount points
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       230G   61G  160G  28% /
/dev/sdb2        19T   11T  7.6T  59% /media/filehistory
/dev/sdb2        19T   11T  7.6T  59% /media/downloads
/dev/sdb2        19T   11T  7.6T  59% /media/userData
/dev/sdb2        19T   11T  7.6T  59% /media/vm
/dev/sdb2        19T   11T  7.6T  59% /mnt/archive
# cp --reflink=always /media/download/archlinux.iso /media/userData/Linux
This will fail because the subvolumes are not directly visible in a common Parent. The directories / and /media are from another filesystem.
# cp --reflink=always /mnt/archive/volume/download/archlinix.iso /mnt/archive/volume/userData/Linux
 This will work because they have a parent /mnt/archive on the same filesystem.

Deduplication[edit | edit source]

Deduplication means to two take two or more files and join equal parts as reflinked copies. If one of the files is changed, CoW makes sure that the file remain unique from each other. Smallest possible part of a file to deduplicate is 4KiB.

There are several tools that can do deduplication on Btrfs. They either operate on whole files or on individual filesystem blocks. Block based deduplication can be more efficient as it can match parts of files that are equal, but the downside is that the deduplication process can be slower.

Name File-based Block-based Incremental Notes
duperemove Yes No Yes Sqlite database for csum. Runs by extent boundary by default, but has an option to more carefully compare.
bees No Yes Yes Runs continuously as a daemon. Uses a database to track deduplication, useful for large storage like backup servers.
dduper Yes Yes No Uses built-in BTRFS csum-tree, so is extremely fast and lightweight. Requires BTRFS-PROGS patch for csum access. Be careful of hash-collisions on large filesystems!
rmlint Yes No Partial A duplicate file finder with btrfs support.
dduper Yes No No A fork of fdupes which includes support for BTRFS deduplication when it identifies duplicate files.

Bees[edit | edit source]

Bees is perhaps the most specific Btrfs tool as it is specifically made to work only with Btrfs filesystems. It is different in several ways:

  1. It uses a very lightweight database (hash table) to keep track of deduplication progress. You can stop and restart and it continues from where it left off.
  2. Memory usage is fixed and never grows beyond database size and a small amount for the bees runtime, whereas most other dedup tools grow to multi gigabyte RAM usage)
  3. It runs continuously in the background, deduplicating any newly written data.
  4. It runs only on whole filesystems, not on a limited set of files.
  5. Bees can split extents to achieve better deduplication results.

The manual is a little awkward for non-bees people. Have a look at Btrfs/Deduplication/Bees for a quick user-guide to get started.

Duperemove[edit | edit source]

Duperemove works on indivitual file extents similar to Bees, but it operates on individual files instead of the whole filesystem. It scans each file, caluclates block hashes and submits duplicates to the Linux kernel for deduplication.

One big advantage is that duperemove can store calculated hashes in a database. This makes it possible to re-run duperemove without having to scan all old files, while it will detect new files and add to the database. This greatly reduces the time for each run.

Duperemove can also take input from the fdupes program, avoiding the need to do any hash checking at all.

For some examples on how to use Duperemove you can head over to my user-guide at Btrfs/Deduplication/Duperemove

The official web page for Duperemove is at http://markfasheh.github.io/duperemove/