versioning without storing multiple full copies?

[ItsCubeTime]

Hi, is versioning without storing multiple full copies possible?

If not, are there any open (preferably) source alternatives that can?

I wish to backup some local hard drives onto an internal HDD dedicated for backups, I want the backup drive to store all data necessary to restore files if any of my other drives fail (so I want to be able to unplug my drive, put it in a new system & be able to extract my data, its fine if I need some external software to actually retrieve the data however as longs as its there).

Now, Im wondering if its possible to do this with versioning as well in any efficient way so that I can go back to one of my drives at a specific point in time? I have about 5TB worth of data that I wish to backup onto a 6TB drive. Its okay if available time points are fairly limited, say that I can restore a backup from 1 day ago + 1 week ago + 1 month ago - but nowhere inbetween (however if possible it would be even more awesome if it kept track of all file changes and could generate a version at any point in time based on that).

I guess Im kind of asking for git (but preferably something faster since I will be backing up several TBs).

Thanks for reading 💖 And Happy new year!

[RonnyPfannschmidt]

Borg is a backup tool, not a redundancy supplement

[RonnyPfannschmidt]

However if reading your request differently, borg already deduplicates between backups

[ThomasWaldmann]

borg deduplicates between backup archives and it can also compress data, so it is possible to store multiple backup archives of a 5TB data set onto a 6TB drive.

How many you will be able to store depends on the amount of changes between your backups. Typically, there is little change if there is little time between 2 backup archives and more change of there is more time in between - but how much this is specifically of course depends a lot on what you are doing on that system. So, for borg prune, you could just start cautiously with keeping only a few archives and increase the amount if you see how it fits. Avoid running out of space by all means (see docs).

borg stores all files you give to it as it reads them from the filesystem and logically it always does a full backup (== store all files into a backup archive). it is just not stupid and does not store the same piece of data multiple times internally.

if your filesystem is totally idle, this can be considered a snapshot of the state you have at the time of the backup. if the fs is active, you may want to rather backup a fs-level snapshot to get better consistency. also, you might need additional actions if you want to backup databases.

so, guess this can be considered what you refer to as "version".

note: borg does not "life" monitor file system changes, so only versions / states are available for which you have created borg archives by running "borg create".

borg is a backup tool and not a vcs, so the amount of control and the kind of available operations differ.

[ThomasWaldmann]

about "versioning without storing multiple full copies":

borg does chunk based deduplication which is better than full file based deduplication.

so, if a big file only changes a little internally, borg only stores the changed pieces (chunks) additionally and re-uses (re-references) the unchanged chunks.

there is a lower threshold for the chunk size though, so files below that only generate 1 chunk and thus, if the file changes (and still only generates 1 chunk), the full changed file (== that 1 chunk) is stored. this is to avoid too much management overhead. the chunking parameters have sane defaults, but can be configured to match specific use cases.

[ItsCubeTime]

Thanks or the responses, sounds like exactly what Im looking for, can I browse & extract individual files from a backup as well? Or do I need to restore everything into a 2nd hard drive to be able to access my files?

[ThomasWaldmann]

Yes, you can use borg mount to comfortably browse, search for or extract files.

You can also use borg extract to extract a few files (if you know the exact paths).

[ItsCubeTime]

ah right, I just saw the software doesnt seem to support Windows (which is the platform I use), such a shame :l

Thanks for the answers still

[ThomasWaldmann]

There is not native windows support, but you can try cygwin or WSL.