UPDATED 02/Jan/2023, ChangeLog:
- 02/Jan/2023: documented that the new version of grub-btrfs is now an official package (you still have to install another package: inotify-tools);
- 02/Dec/2022: documented the new version of grub-btrfs and its new grub-btrfsd daemon; the configuration for Timeshift is much simpler, but you have to install another package: inotify-tools.
After looking at the very nice videos of Stephen’s Tech Talks, in particular, this one https://www.youtube.com/watch?v=6wUtRkEWBwE, I decided to try to set up Timeshift, Timeshift-autosnap, and grub-btrfs in my Linux Arch installation, where I’m using BTRFS as the filesystem. These three packages allow a timeshift snapshot to be automatically created each time you update your system; moreover, a new grub entry is automatically generated to boot into a specific snapshot.
The video mentioned above is handy, but unfortunately, some recent changes in Timeshift itself broke the behavior of the two other packages. In this post, I’ll try to show how to fix the problem and go back to a working behavior. I’ll also show an experiment using the snapshots so that, hopefully, it’s clear what’s going on in the presence of such snapshots and how to use them in case you want to revert your system.
Install timeshift and timeshift-autosnap
First of all, let’s install timeshift and timeshift-autosnap (the latter depends on the former, and they are both available from AUR; I’m using the yay AUR helper here):
1 |
yay -S timeshift-autosnap |
The programs will be installed from sources; thus, they will be compiled (it might take some time).
Let’s create a new Timeshift snapshot to make sure it works (the first time, you will have to configure Timeshift; of course, it is crucial that you choose “BTRFS”).
You can configure timeshift-autosnap with the number of snapshots to keep (in this example, I specify 10):
1 2 3 4 5 6 |
sudo nano /etc/timeshift-autosnap.conf ... # maxSnapshots defines how much old snapshots script should left. # Only positive whole numbers can be used. # Default value is 3. maxSnapshots=10 |
Install grub-btrfs (new version, installation from the official repository, 02/Jan/2023)
The new version of grub-btrfs is now available as an official package (Please remove the old AUR version if you still have it installed):
1 2 |
community/grub-btrfs 4.12-2 (33.8 KiB 98.6 KiB) (Installed) Include btrfs snapshots in GRUB boot options |
So you can simply install it with pacman:
1 |
sudo pacman -S grub-btrfs |
Now, let’s make sure grub-btrfs can find Timeshift’s snapshots (remember, we’ve just created one). So let’s update the grub configuration, and we should see in the end something like the following output:
1 2 3 4 5 6 7 8 |
$ sudo grub-mkconfig -o /boot/grub/grub.cfg ... Adding boot menu entry for UEFI Firmware Settings ... Detecting snapshots ... Found snapshot: 2022-11-19 15:08:51 | timeshift-btrfs/snapshots/2022-11-19_15-08-51/@ | ondemand | N/A | Found 1 snapshot(s) Unmount /tmp/grub-btrfs.shzoN5Pcoo .. Success done |
The last lines prove that grub-btrfs can detect snapshots.
We now need to configure that to monitor the Timeshift snapshot directory instead of the default one (/.snapshots).
Automatically update the grub menu upon snapshot creation or deletion (2 December 2022)
What follows is based on the new version of grub-btrfs. At the bottom of the post, there are still the old instructions, which are to be considered stale and left there only for “historical reasons”.
Grub-btrfs provides a daemon watching the snapshot directory and updates the grub menu automatically every time a snapshot is created or deleted.
Important: This daemon requires an additional package:
1 |
sudo pacman -S inotify-tools |
By default, this daemon watches the directory “/.snapshots” (the default directory for Snapper). Since Timeshift uses a different directory, we have to tweak the configuration for the daemon.
Let’s run:
1 |
sudo systemctl edit --full grub-btrfsd |
We must change the line
1 |
ExecStart=/usr/bin/grub-btrfsd --syslog /.snapshots |
into
1 |
ExecStart=/usr/bin/grub-btrfsd --syslog --timeshift-auto |
This is required for Timeshift version 22.06 and later because Timeshift creates a new directory named after their process ID in /run/timeshift every time they are started. Since the PID will be different every time, also the directory will be different. Grub-btrfs provides the command line argument –timeshift-auto to correctly detect the current snapshot directory (In previous versions of grub-btrfs, we had to tweak /etc/fstab to deal with that, as shown later in the old section).
Let’s start the daemon:
1 |
sudo systemctl start grub-btrfsd |
In the journalctl log, we should see something like (where the date and time have been stripped off):
1 2 |
grub-btrfsd[9965]: grub-btrfsd starting up... grub-btrfsd[9989]: Watching /run/timeshift for timeshift to start |
Let’s start Timeshift. In the journalctl log, we should see something like this:
1 2 3 4 |
grub-btrfsd[10307]: detected Timeshift startup, PID is: 10232 grub-btrfsd[10611]: Grub submenu recreated systemd[1]: tmp-grub\x2dbtrfs.e6hSsOzTFf.mount: Deactivated successfully. grub-btrfsd[10612]: Watching /run/timeshift/10232/backup/timeshift-btrfs/snapshots for new snapshots... |
Let’s verify that if we create a new snapshot, grub-btrfs automatically updates the GRUB menu: in a terminal window, run “journalctl -f” to look at the log, then create a new snapshot in Timeshift. In the log, you should see something like the following lines:
1 2 3 4 5 6 7 8 9 |
timeshift-gtk.desktop[10232]: Created directory: /run/timeshift/10232/backup/timeshift-btrfs/snapshots/2022-11-20_17-31-59 grub-btrfsd[10667]: Detected snapshot creation/ deletion, recreating Grub menu timeshift-gtk.desktop[10232]: Created subvolume snapshot: /run/timeshift/10232/backup/timeshift-btrfs/snapshots/2022-11-20_17-31-59/@ timeshift-gtk.desktop[10232]: Created control file: /run/timeshift/10232/backup/timeshift-btrfs/snapshots/2022-11-20_17-31-59/info.json timeshift-gtk.desktop[10232]: BTRFS Snapshot saved successfully (0s) timeshift-gtk.desktop[10232]: Tagged snapshot '2022-11-20_17-31-59': ondemand timeshift-gtk.desktop[10232]: ------------------------------------------------------------------------------ grub-btrfsd[11020]: Grub submenu recreated systemd[1]: tmp-grub\x2dbtrfs.OV56bNx3Ax.mount: Deactivated successfully. |
Similarly, if we delete an existing snapshot, we should see something similar in the log.
Remember that it takes a few seconds for grub-btrfs to recreate the grub menu.
Once we’re sure everything works, we can enable the daemon to always start at boot:
1 |
sudo systemctl enable grub-btrfsd |
The next time we boot, our grub menu will also show a submenu to boot snapshots.
Concerning doing some experiments booting a snapshot and restoring it, please look at the next section.
IMPORTANT: If you have several Linux distributions on your computer and you use a multiboot system like the one I blogged about, and this distribution is not the main one, you will have to manually tweak the entry in your main distribution’s GRUB menu. See the linked blog post near the end.
Some experiments
Let’s do some experiments with this configuration.
Here’s the kernel I’m currently running:
1 2 |
> uname -a Linux lg-arch 5.18.7-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 25 Jun 2022 20:22:01 +0000 x86_64 GNU/Linux |
I’m updating the system (I’m skipping some output below, and you can ignore the “stale mount” errors):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 |
> yay :: Synchronizing package databases... core 156,5 KiB 855 KiB/s 00:00 [----------------------] 100% extra 1721,1 KiB 3,43 MiB/s 00:00 [----------------------] 100% community 6,7 MiB 4,35 MiB/s 00:02 [----------------------] 100% :: Starting full system upgrade... resolving dependencies... looking for conflicting packages... warning: insufficient columns available for table display Packages (20) alsa-card-profiles-1:0.3.53-2 binutils-2.38-6 docker-compose-2.6.1-1 electron17-17.4.9-1 evolution-3.44.3-1 evolution-data-server-3.44.3-1 geocode-glib-3.26.3-1 glib2-2.72.3-1 glib2-docs-2.72.3-1 gnome-calculator-42.2-1 gnome-control-center-42.3-1 gnome-software-42.3-1 gnome-software-packagekit-plugin-42.3-1 gst-plugin-pipewire-1:0.3.53-2 liblangtag-0.6.4-1 libsoup3-3.0.7-1 linux-5.18.8.arch1-1 pipewire-1:0.3.53-2 wayland-1.21.0-1 xapp-2.2.12-1 Total Download Size: 274,61 MiB Total Installed Size: 627,21 MiB Net Upgrade Size: 0,79 MiB :: Proceed with installation? [Y/n] :: Retrieving packages... ... :: Running pre-transaction hooks... (1/2) Creating Timeshift snapshot before upgrade... Using system disk as snapshot device for creating snapshots in BTRFS mode Mounted '/dev/nvme0n1p13' at '/run/timeshift/2945/backup' Creating new backup...(BTRFS) Saving to device: /dev/nvme0n1p13, mounted at path: /run/timeshift/2945/backup Created directory: /run/timeshift/2945/backup/timeshift-btrfs/snapshots/2022-07-02_15-35-53 Created subvolume snapshot: /run/timeshift/2945/backup/timeshift-btrfs/snapshots/2022-07-02_15-35-53/@ Created control file: /run/timeshift/2945/backup/timeshift-btrfs/snapshots/2022-07-02_15-35-53/info.json BTRFS Snapshot saved successfully (0s) Tagged snapshot '2022-07-02_15-35-53': ondemand ------------------------------------------------------------------------------ Found stale mount for device '/dev/nvme0n1p13' at path '/run/timeshift/2945/backup' Unmounted successfully E: Failed to remove directory Ret=256 Mounted '/dev/nvme0n1p13' at '/run/timeshift/3244/backup' ------------------------------------------------------------------------------ Removing snapshot: 2022-06-30_18-06-17 Deleting subvolume: @ (Id:263) Deleted subvolume: @ (Id:263) Deleted directory: /run/timeshift/3244/backup/timeshift-btrfs/snapshots/2022-06-30_18-06-17 Removed snapshot: 2022-06-30_18-06-17 ------------------------------------------------------------------------------ Found stale mount for device '/dev/nvme0n1p13' at path '/run/timeshift/3244/backup' Unmounted successfully E: Failed to remove directory Ret=256 Generating grub configuration file ... Found linux image: /boot/vmlinuz-linux Found initrd image: /boot/intel-ucode.img /boot/amd-ucode.img /boot/initramfs-linux.img Found fallback initrd image(s) in /boot: intel-ucode.img amd-ucode.img initramfs-linux-fallback.img Adding boot menu entry for UEFI Firmware Settings ... Detecting snapshots ... Found snapshot: 2022-07-02 15:35:53 | timeshift-btrfs/snapshots/2022-07-02_15-35-53/@ | ondemand | {timeshift-autosnap} {created before upgrade} | Found snapshot: 2022-06-30 19:03:38 | timeshift-btrfs/snapshots/2022-06-30_19-03-38/@ | ondemand | {timeshift-autosnap} {created before upgrade} | Found snapshot: 2022-06-30 18:07:08 | timeshift-btrfs/snapshots/2022-06-30_18-07-08/@ | ondemand | {timeshift-autosnap} {created before upgrade} | Found 3 snapshot(s) Unmount /tmp/grub-btrfs.zrVA9quPSF .. Success done ... ==> Generating module dependencies ==> Creating xz-compressed initcpio image: /boot/initramfs-linux-fallback.img |
So it created a snapshot before updating the system (in particular, it installed a new kernel version). Let’s reboot and verify we are running the new kernel (5.18.8 instead of 5.18.7):
1 2 |
> uname -a Linux lg-arch 5.18.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 29 Jun 2022 23:03:08 +0000 x86_64 GNU/Linux |
Let’s reboot and select from GRUB the latest snapshot (remember, the one before applying the upgrade), so timeshift-btrfs/snapshots/2022-07-02_15-35-53 (snapshots are presented in the grub submenu from the most recent to the oldest one). We do that by pretending that the update broke the system (it’s not the case), and we want to get back to a working system before the update we have just performed.
You see that the “Authentication Required” dialog greets us, and in the background, you can see the notification that we “booted into Timeshift Snapshot, please restore the snapshot”:
The password is required because it’s trying to run Timeshift:
In the screenshot, you can see that we are now using the older kernel since we booted in that snapshot, where the update has not yet been performed. We have to restore the snapshot manually; otherwise, on the next boot, we’ll get back to the updated system version and not in the snapshot anymore.
So, let’s restore the snapshot:
You see, Timeshift has created another snapshot ([LIVE]). We now reboot normally (that is, using the main grub entry, NOT the snapshot entries).
Once rebooted normally, we can verify again that we are running the old kernel:
1 2 |
> uname -a Linux lg-arch 5.18.7-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 25 Jun 2022 20:22:01 +0000 x86_64 GNU/Linux |
Let’s have a look at Timeshift, and we can see the last snapshot is an effective one, not a LIVE one:
Yes, we are now in a system where the update above has never been applied.
Let’s try to rerun the update command (we don’t effectively execute the update, it’s just an experiment):
1 2 3 |
> yay -> /var/lib/pacman/db.lck is present. -> There may be another Pacman instance running. Waiting... |
Why? Because the snapshot had been created automatically by timeshift-autosnap before applying the updates while the package manager was running, its lock is still there.
Let’s remove the lock and try to rerun the update:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
> sudo rm /var/lib/pacman/db.lck > yay :: Synchronizing package databases... core is up to date extra is up to date community is up to date :: Starting full system upgrade... resolving dependencies... looking for conflicting packages... warning: insufficient columns available for table display Packages (20) alsa-card-profiles-1:0.3.53-2 binutils-2.38-6 docker-compose-2.6.1-1 electron17-17.4.9-1 evolution-3.44.3-1 evolution-data-server-3.44.3-1 geocode-glib-3.26.3-1 glib2-2.72.3-1 glib2-docs-2.72.3-1 gnome-calculator-42.2-1 gnome-control-center-42.3-1 gnome-software-42.3-1 gnome-software-packagekit-plugin-42.3-1 gst-plugin-pipewire-1:0.3.53-2 liblangtag-0.6.4-1 libsoup3-3.0.7-1 linux-5.18.8.arch1-1 pipewire-1:0.3.53-2 wayland-1.21.0-1 xapp-2.2.12-1 Total Installed Size: 627,21 MiB Net Upgrade Size: 0,79 MiB |
The output is similar to the one shown above (unless there are even more new updates in the meantime, which might happen in a rolling release), but something is missing:
1 |
Total Download Size: 274,61 MiB |
Why? Because the downloaded packages in the cache are NOT part of the saved snapshot, they are still present in the current system, even though we restored the snapshot. Why are the cached packages still there, but the lock has been restored with the snapshot? That’s due to the way subvolumes are specified in the /etc/fstab:
1 2 |
UUID=... /var/cache btrfs subvol=/@cache,defaults,noatime,compress=zstd:1,ssd 0 0 UUID=... /var/log btrfs subvol=/@log,defaults,noatime,compress=zstd:1,ssd 0 0 |
You see, the cache of downloaded packages and the logs are NOT part of the snapshots, while /var/lib (including the pacman lock) is part of the snapshots.
Let’s now revert the snapshot: we select the one with “Before restoring…”.
Again, we are now in a LIVE situation, and Timeshift tells us again to reboot to make it effective.
Let’s reboot (by using the main grub entry).
We’re back to the updated system, and there’s nothing to update (again, unless new updates have been made available in the meantime):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
> uname -a Linux lg-arch 5.18.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 29 Jun 2022 23:03:08 +0000 x86_64 GNU/Linux > yay [sudo] password for bettini: :: Synchronizing package databases... core is up to date extra is up to date community is up to date :: Starting full system upgrade... there is nothing to do :: Searching databases for updates... :: Searching AUR for updates... -> Missing AUR Debug Packages: gnome-terminal-transparency-debug gnome-text-editor-debug there is nothing to do |
If we’re happy with the updated system, we can also remove those two snapshots (remember that grub-btrfs monitors the snapshots so that it will update its grub submenu entries):
I hope you find this blog post helpful, and I hope it complements the beautiful video of Stephen’s Tech Talks mentioned above.
Old version (with old release 4.11 of grub-btrfs)
UPDATE 02/Dec/2022: These are the older instructions for the previous version of grub-btrfs, where there was no “grub-btrfsd.service” and there was another systemd program (“grub-btrfs.path”).
I leave these instructions here just for “historical reasons”.
The first problem is that timeshift has recently changed the strategy for creating snapshots. Instead of creating them in /run/timeshift/backup/timeshift-btrfs/snapshots, it now creates them in /run/timeshift/<PID>/backup/timeshift-btrfs/snapshots, where <PID> is the PID of the Timeshift process. Each time you run Timeshift, the directory will be different, breaking grub-btrfs (which expects to find the snapshots always in the same directory).
Fortunately, there’s a workaround: we add an entry to /etc/fstab in order to mount explicitly the path /run/timeshift/backup/timeshift-btrfs/snapshots:
1 |
UUID=<UUID> /run/timeshift/backup btrfs defaults,noatime 0 0 |
where, of course, <UUID> has to be replaced with the same UUID of the physical disk partition.
Reboot, and then Timeshift will also put the snapshot in that directory (besides the one with the PID, as mentioned above). You can try to create a snapshot to verify that (this also allows us to use the Timeshift wizard so that we specify to create BTRFS snapshots).
Let’s make sure the mount point is active (and note the unit name)
1 2 3 4 5 |
> systemctl list-units -t mount UNIT LOAD ACTIVE SUB DESCRIPTION ... run-timeshift-backup.mount loaded active mounted /run/timeshift/backup ... |
Let’s now install grub-btrfs
1 |
sudo pacman -S grub-btrfs |
We need to configure that to monitor the Timeshift snapshot directory instead of the default one (/.snapshots).
1 |
sudo systemctl edit --full grub-btrfs.path |
The file contents
1 2 3 4 5 6 7 8 9 10 11 12 |
[Unit] Description=Monitors for new snapshots DefaultDependencies=no Requires=\x2esnapshots.mount After=\x2esnapshots.mount BindsTo=\x2esnapshots.mount [Path] PathModified=.snapshots [Install] WantedBy=\x2esnapshots.mount |
should be replaced with
1 2 3 4 5 6 7 8 9 10 11 12 |
[Unit] Description=Monitors for new snapshots DefaultDependencies=no Requires=run-timeshift-backup.mount After=run-timeshift-backup.mount BindsTo=run-timeshift-backup.mount [Path] PathModified=/run/timeshift/backup/timeshift-btrfs/snapshots [Install] WantedBy=run-timeshift-backup.mount |
Let’s reload and re-enable the monitoring service:
1 2 |
sudo systemctl reenable grub-btrfs.path sudo systemctl start grub-btrfs.path |
If we have already created a few snapshots, we can run update-grub (or, if you have not installed the package update-grub, use the command “grub-mkconfig -o /boot/grub/grub.cfg”) and verify that new grub entries are created for the found snapshots:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
> sudo update-grub Generating grub configuration file ... Found linux image: /boot/vmlinuz-linux Found initrd image: /boot/intel-ucode.img /boot/amd-ucode.img /boot/initramfs-linux.img Found fallback initrd image(s) in /boot: intel-ucode.img amd-ucode.img initramfs-linux-fallback.img ... Detecting snapshots ... Found snapshot: 2022-06-30 18:07:08 | timeshift-btrfs/snapshots/2022-06-30_18-07-08/@ | ondemand | N/A Found snapshot: 2022-06-30 18:06:17 | timeshift-btrfs/snapshots/2022-06-30_18-06-17/@ | ondemand | N/A Found snapshot: 2022-06-30 16:00:01 | timeshift-btrfs/snapshots/2022-06-30_16-00-01/@ | daily | N/A | Found 3 snapshot(s) Unmount /tmp/grub-btrfs.3u9wTHZNNW .. Success done |
We can also restart the system and prove that we can access the GRUB submenu with the generated entries for the snapshots.
thanks for the tutorial. I’m really helped by that.
Pingback: Linux EndeavourOS Cassini Review | Lorenzo Bettini
Nice tutorial. Tks. Worked fine on Arch
“sudo pacman install grub-btrfs” should be “sudo pacman -S grub-btrfs”
Right! Thank you, I’ve just updated the post accordingly.
Hello Lorenzo, What a great blog. Thank you for posting these detailed instructions, I thought I was missing something until you pointed to the fact that grub-btrfs had been updated and required a different configuration. How you managed to work it out, I have no idea but I am glad you did.
I’m glad the blog helped 🙂
Hello Lorenzo, I’m using Endeavour. After a recent update when I run systemctl –failed grub-btrfsd.service loaded failed failed Regenerate grub-btrfs.cfg I have tried look on the internet but with on luck. Can help by pointing the right place to look.
Without further error details, it’s hard to understand what’s going on.
Some time ago, IIRC, an update to btrfs-progs broke grub-btrfs. At that time, it was enough to downgrade btrfs-progs and block the upgrade til they fixed the problem in that package.
Maybe that’s the same.
It’s better to ask the EndeavourOS forum or, even better, create an issue in grub-btrfs.
For the time being, it’s better to restore a working snapshot.