Deduplication with Bees
Bees is perhaps the most unique Deduplication tool as it is specifically made to work only with Btrfs filesystems. It is different in several ways:
- It uses a very lightweight database (hash table) to keep track of deduplication progress. You can stop and restart and it continues from where it left off.
- Memory usage is fixed and never grows beyond database size and a small amount for the bees runtime, whereas most other dedup tools grow to multi gigabyte RAM usage)
- It runs continuously in the background, deduplicating any newly written data.
- It runs only on whole filesystems, not on a limited set of files.
- Bees can split extents to achieve better deduplication results.
Make sure you have a recent kernel. Some kernel versions has bugs triggered by Bees, so please check https://github.com/Zygo/bees/blob/master/docs/btrfs-kernel.md before continuing.
Installation instruction can be found on their Github repository here: https://github.com/Zygo/bees/blob/master/docs/install.md.
Please note that at the time of writing, the last stable version (v0.7) has some issues. Before compiling the program, please read the known issues section.
First you need to determine the UUID of the filesystem you want to run Bees on. Use
btrfs filesystem show to list all Btrfs filesystems with their UUID's.
# btrfs filesystem show
Label: 'btrfs-root' uuid: 446d32cb-a6da-45f0-9246-1483ad3420e0 Total devices 1 FS bytes used 35.04GiB devid 1 size 229.47GiB used 79.03GiB path /dev/sda3 Label: '6TB' uuid: fe0a1142-51ab-4181-b635-adbf9f4ea6e6 Total devices 2 FS bytes used 3.48TiB devid 2 size 2.72TiB used 2.21TiB path /dev/sdc2 devid 3 size 1.82TiB used 1.30TiB path /dev/sdb2 Label: 'btrfs-boot' uuid: 1128e72e-b00f-4c2a-a1e1-afa89f3c11cc Total devices 1 FS bytes used 70.55MiB devid 1 size 1.00GiB used 256.00MiB path /dev/sda2 Label: 'usb-backup' uuid: df68a30d-d26e-4b9c-9606-a130e66ce63d Total devices 1 FS bytes used 581.47GiB devid 1 size 927.51GiB used 591.02GiB path /dev/sdd1
We'll use the
fe0a1142-51ab-4181-b635-adbf9f4ea6e6 in this guide.
Now we can create the Bees configuration file
/etc/bees/6TB.conf directly, or copy the sample configuration file located at
/etc/bees/beesd.conf.sample and edit it. You can use any name on the file with a
.conf file extension.
# UUID of the filesystem UUID=fe0a1142-51ab-4181-b635-adbf9f4ea6e6 # Specify the bees database size. It has to be a multiple of 128KiB DB_SIZE=$((256*1024*1024)) # 256MiB in bytes
The database size determines how efficient Bees will be on your filesystem. The database has to fit in RAM, so make sure you have enough RAM available to run Bees with the selected database size.
|Unique data size||Database size||Average dedupe extent size|
|1TiB||128MiB||128KiB <- recommended|
Once you have the configuration set up we can run the Bees daemon
# beesd fe0a1142-51ab-4181-b635-adbf9f4ea6e6
I recommend to run bees inside a screen terminal or as a daemon. Bees comes with a systemd unit file: https://github.com/Zygo/bees/tree/master/scripts
# systemctl enable --now email@example.com
Created symlink /firstname.lastname@example.org → /lib/systemd/system/beesd@.service.
Bees will store various statistics and other files:
|/run/bees/uuid.status||Current running stats of bees.|
|/run/bees/mnt/uuid/.beeshome/beescrawl.dat||Bees crawler stats|
|/run/bees/mnt/uuid/.beeshome/beesstats.txt||Bees statistics, database usage.|
Systemd service broken
The last stable version (v0.7) have an error inside the systemd unit file, preventing the service from starting. You can fix it easily by editing the unit file by adding
RuntimeDirectory=bees on the
[Service] section. It can be done easily with one of the two next command.
# sed -i '/^Restart=on-abnormal*/a RuntimeDirectory=bees' scripts/beesd@.service
# sed -i '/^Restart=on-abnormal*/a RuntimeDirectory=bees' $(pkg-config systemd --variable=systemdsystemunitdir)/beesd@.service && systemctl daemon-reload
Bees version not showed
The package for the last stable version (v0.7), didn't return its version number at the compilation, leading the binary to return an error instead of its version.
You can fix it by manually set it before compilation with:
# sed -i 's/BEES_VERSION ?=.*/BEES_VERSION ?= v0.7/' ./Makefile
Too much information is logged
By default, beesd is configured to be the much verbose by default, which is not ideal in some situation. It can be changed by launching beesd with
-v 6 (warning level), or set in the config file.
# UUID of the filesystem UUID=fe0a1142-51ab-4181-b635-adbf9f4ea6e6 # Specify the bees database size. It has to be a multiple of 128KiB DB_SIZE=$((256*1024*1024)) # 256MiB in bytes ## Options to apply, see `beesd --help` for details OPTIONS="-P -v 6"