A few months ago, I migrated a bunch of media from a ZFS mirror on my Mac to my home server VM. I used mismatched drives that I happened to have on hand, and while it works fine, I might as well replace the disks with the ones from the ZFS pool that I’m not using any more.
While ZFS works reasonably well, it carries a ton of technical and political debt, particularly on Linux. Since btrfs has matured significantly in the last few years, I’m using it on my Linux servers now. I like btrfs a lot. It has good performance, and offers advanced features like data and metadata checksumming, snapshots, RAID, and compression.
What follows is mostly for my own notes, but maybe it’ll help someone.
Note that /var/lib/plex/data is just one of the subvolumes that
belongs to the btrfs filesystem I’m working on.
First, let’s identify the existing devices in the filesystem:
root@proteus:~# btrfs filesystem show /var/lib/plex/data
Label: 'Media' uuid: 290c5b77-9bff-47d5-9f82-3d7576f134f5
Total devices 2 FS bytes used 1.62TiB
devid 1 size 5.46TiB used 1.62TiB path /dev/sdd2
devid 2 size 3.64TiB used 1.62TiB path /dev/sdc2
/dev/sdd2, which btrfs regards as devid 2, is the disk I’ll
replace first. I’ve already partitioned a replacement disk, /dev/sde2
Let’s start the replacement:
root@proteus:~# btrfs replace start -B 2 /dev/sde2 /var/lib/plex/data
The -B option makes the command block until it’s completed, so
it’s completely optional. 2 is the devid we’re replacing. The
filesystem will remain online and read/write during the migration.
Btrfs doesn’t appear to prioritize regular filesystem I/O during
the migration, so expect performance to suffer until the migration
is done.
It’s also possible to monitor the status in a blocking way:
root@proteus:~# btrfs replace status /var/lib/plex/data
7.4% done, 0 write errs, 0 uncorr. read errs
With the migration in progress, you can also see that the mirror has three devices attached:
root@proteus:~# btrfs filesystem show /var/lib/plex/data/
Label: 'Media' uuid: 290c5b77-9bff-47d5-9f82-3d7576f134f5
Total devices 3 FS bytes used 1.63TiB
devid 0 size 3.64TiB used 1.62TiB path /dev/sde2
devid 1 size 5.46TiB used 1.64TiB path /dev/sdd2
devid 2 size 3.64TiB used 1.64TiB path /dev/sdc2
The new disk, devid 0, is only reporting 3.64TiB, even though
it’s an 8TB disk. We’ll fix that once the migration is done.
Hours later, the process is done:
root@proteus:~# btrfs filesystem show /var/lib/plex/data/
Label: 'Media' uuid: 290c5b77-9bff-47d5-9f82-3d7576f134f5
Total devices 2 FS bytes used 1.62TiB
devid 1 size 5.46TiB used 1.63TiB path /dev/sdd2
devid 2 size 3.64TiB used 1.63TiB path /dev/sde2
Note how devid 2 is the new disk on /dev/sde2, but it still
shows the wrong capacity. It’s unclear why resizing is not automatic,
but it can be done manually:
root@proteus:~# btrfs filesystem resize 2:max /var/lib/plex/data
Resize '/var/lib/plex/data' of '2:max'
The resize operation completes almost instantly, and now we have:
root@proteus:~# btrfs filesystem show /var/lib/plex/data/
Label: 'Media' uuid: 290c5b77-9bff-47d5-9f82-3d7576f134f5
Total devices 2 FS bytes used 1.62TiB
devid 1 size 5.46TiB used 1.63TiB path /dev/sdd2
devid 2 size 7.28TiB used 1.63TiB path /dev/sde2
Finally, we can see that full redundancy is maintained:
root@proteus:~# btrfs device usage /var/lib/plex/data
/dev/sdd2, ID: 1
Device size: 5.46TiB
Device slack: 0.00B
Data,RAID1: 1.63TiB
Metadata,RAID1: 3.00GiB
System,RAID1: 64.00MiB
Unallocated: 3.83TiB
/dev/sde2, ID: 2
Device size: 7.28TiB
Device slack: 3.50KiB
Data,RAID1: 1.63TiB
Metadata,RAID1: 3.00GiB
System,RAID1: 64.00MiB
Unallocated: 5.65TiB
There are some relevant messages in the kernel log:
[1987714.818696] BTRFS info (device sdd2): dev_replace from /dev/sdc2 (devid 2) to /dev/sde2 started
[1998011.747252] BTRFS info (device sdd2): dev_replace from /dev/sdc2 (devid 2) to /dev/sde2 finished
[1999100.104284] BTRFS info (device sdd2): resizing devid 2
[1999100.104289] BTRFS info (device sdd2): new size for /dev/sde2 is 8001352437760
Now /dev/sdc can be removed from the system.
root@proteus:~# lsblk -S /dev/sdc
NAME HCTL TYPE VENDOR MODEL REV TRAN
sdc 4:0:0:0 disk Mercury Elite Pro Quad C 0 usb
The above command shows that the disk occupies bay C in the disk
enclosure.
After removing the disk and replacing it with another 8TB disk, the entire process was repeated to replace /dev/sdd2.