Introduction ============ Why another tool for performing ZFS-related tasks? -------------------------------------------------- ZFS is useful in many different ways. It does not restrict the user to a specific workflow. Instead, it provides an impressive collection of basic tools, commands, APIs and "building blocks" for doing just about anything imaginable when it comes to storage while being actually incredible reliable. Next to ZFS' reliability, ZFS's ``send`` & ``receive`` commands are really fast operations, no matter how many small files or fragments are being dealt with. The mentioned commands can also handle sending and receiving compressed datasets without extra decompression/compression steps which is rather common in other, similar tools. ZFS' stream or pipe-based approach, tightly integrating into Unix-like environments, is easily extensible and a true invitation for writing advanced, custom tools on top. In stark contrast, while searching for a management, replication and backup tool for ZFS around 2016, one gap stood out in this part of the ecosystem: A lack of quality, limited reliability especially in a mathematical sense and basically no meaningful form of code structure, e.g. object orientation. Many tools were written in shell languages such as e.g. ``bash`` or, even worse, in languages like ``PHP``. Virtually all of them came without proper test suites. Most of them fell apart when using them in an "unusual manner" just a tiny little bit. All of them made implicit assumptions about the structure and usage of zpools and/or surrounding computer networks, which, if not followed precisely to the letter, could cause data loss in worst-case-scenarios. None of the tools made an effort to model zpools or their workflows in any form, making them extra hard to debug or understand and ultimately impossible to extend. As a less problematic but still annoying side note, most existing tools were lacking any form of proper user interface. The CLI interfaces commonly found required a ton of experience and were not forgiving mistakes. Native GUIs were virtually non-existing. Web UIs intended for system administrators and narrow use cases were usually the only half-decent user interfaces available. Design goals ------------ The fundamental idea is to **synchronize portions of two zpools** on potentially, but not necessarily, two different physical computers. This may be for backups but also for moving "projects" or virtual machines around between different workstations or servers. The overall approach should be as robust as possible, allowing for fine-grained tests, easy maintenance and good extensibility. Under no circumstances, even if misconfigured, should the tool allow data loss to occur, ideally making it just as reliable as the ZFS filesystem underneath. It should be possible to mathematically proof the latter. Two of the requirements that led to ``abgleich`` being developed was as-fast-as-possible transfers while also allowing to optimize for the data volume being transferred during backups in low-bandwidth-environments. By default, ``abgleich`` generates a sequence of atomic transactions which are presented to the user for confirmation before acting on them. In this regard, it behaves rather similar to partitioning tools such as *parted* or *GParted*. While the tool can be used on headless servers, it is explicitly also aiming at laptops, desktop computers and workstations running on ZFS. Proper command line and graphical user interfaces for "end users" are therefore required. A single API is driving the different types of user interfaces. ZFS' ability to create and clone snapshots allows to mimic workflows similar to what is possible with ``git``. Just like ``git`` "commits", the idea is to only create snapshots if a dataset/volume actually contains changes. In the ZFS world, this is a rather unusual design choice as it is common practise to generate snapshot in fixed time intervals regardless of changes in the dataset/volume. ``abgleich`` implements special logic to safely and quickly determine if a dataset/volume contains "uncommitted" changes. With regards to similarities to ``git``, ``abgleich`` aims to provide the ability to synchronize "projects" (datasets/volumes) across multiple computers in addition to providing backup functionality. Projects can be "moved" or "cloned" from one machine to the next, allowing a user to always work on them efficiently and locally. Besides, synchronization operations can happen both in a pull and a push configuration. Video: Discussion and demo of `abgleich` (in German) ---------------------------------------------------- A discussion and "live" demo of ``abgleich`` (in German) can be found `here`_. .. _here: https://www.youtube.com/watch?v=BjZJmoHnK3Q Alternative ZFS-based Tools --------------------------- The one and only, the classic, the "bash-nightmare": `zfs-auto-snapshot`_. .. _zfs-auto-snapshot: https://github.com/zfsonlinux/zfs-auto-snapshot Noteworthy alternative sync and backup tools -------------------------------------------- It has been attempted to build similar tools on top of `BTRFS`_, which did not take off as much as initially expected despite being seen as a better or purer version of ZFS, i.e. less licensing issues. Noteworthy examples include `btrfs-sxbackup`_ and `btrbk`_. .. _BTRFS: https://btrfs.wiki.kernel.org/index.php/Main_Page .. _btrfs-sxbackup: https://github.com/masc3d/btrfs-sxbackup .. _btrbk: https://github.com/digint/btrbk Besides, there is a rather mature family of similar tools based on `rsync`_, which primarily suffer from the significant performance overhead of having to go through all of the filesystem's layers where ZFS-based tools in comparison can operate on a rather low block-level. .. _rsync: https://en.wikipedia.org/wiki/Rsync Another noteworthy example offering similar capabilities but operating on top of the filesystem as well is `Borg`_. .. _Borg: https://www.borgbackup.org/ Although ``git`` itself is not intended to manage large binary files or simply very high numbers of files at once, it can be extended with plugins such as `git-lfs`_ to provide capabilities very similar to ``abgleich``. Unfortunately, any ``git``-based approach will suffer from filesystem-overhead just like ``rsync``, next to other issues. Other approaches based on ``git`` have been developed by Microsoft for dealing with very large mono-repos. Examples include `VFS for Git`_ (first generation, abandoned), `Scalar`_ (second generation, abandoned) and `microsoft/git`_ (third and current generation). .. _git-lfs: https://git-lfs.github.com/ .. _VFS for Git: https://github.com/microsoft/VFSForGit .. _Scalar: https://github.com/microsoft/scalar .. _microsoft/git: https://github.com/microsoft/git