Advanced filtering for borg prune

[knutov]

What is the best way to do filtering of snapshots to prune with borg prune based on comments, suffix, dates without times?

Example snapshot list

snapshot (of files) name	snapshot comment	snapshot timestamp
files-2021-01-18-144409	folder only ./some/path	2021-01-18 14:44:19
files-2021-01-18-144006	folder only ./path2/files	2021-01-18 14:40:17
files-2021-01-18-143709	folder only ./dir1	2021-01-18 14:37:19
files-2021-01-18-143115	folder only ./dir2	2021-01-18 14:31:28

snapshot (from --stdin)	timestamp
mysql-db1	2021-02-28 02:49:08
mysql-db2	2021-02-28 02:49:06
mysql-db3	2021-02-28 02:49:03

Real scenarios I'd like to implement with snapshots:

1. prune all snapshots, except last N days, based on dates without times (ex: I have multiple snapshots per day with multiple prefixes/suffixes, and I want to save all snapshots in this saved days, not the last snapshot from this day, so all snapshot from 2021-01-18 must stay, not the last one)
2. prune with some --weekly --days etc rules snapshot only with comment containing './dir1'
3. prune with some --weekly --days etc rules only snapshot with some suffix and/or prefix
4. combination of all above

Are any better ways then borg prune --dry-run ... | grep ... | xargs borg delete for scenario 2 and 3 and do manual delete in some script based on borg list for the first scenario ?

[ThomasWaldmann]

Doing something based on comments seems bad, in general.

At some other places, borg supports selection based on archive name prefix or name match.
For prune, borg already supports name prefix (not sure about name match, if we do not have that, that could be added).

Keeping all of a day: would make the time based code of prune more complex to implement and also more hard to understand how it works - not a fan of that considering we get issue reports all the time from people suspecting that it doesn't work correctly.

Why would you like to keep all of one day?

[knutov]

Doing something based on comments seems bad, in general.

I understand, but I'm really have this scenario. I do sometimes partial backup with only selected folders, and I need for them common-style snapshot names becouse of my backup restore scenario/scripts, so the only place to store additional meta data is comments.

I do backup multiple mysql databases (via --stdin) to one borg archive, so they have different suffixes and different time. When I do borg prune borg prunes all snapshots from the day excluding the last snapshot made in this day, so I have to run multiple times borg prune with filtering via regex by suffix. The problem is borg do compact every time (as I'm on stable branch with borg 1.1.15), so it takes too much time.

[ThomasWaldmann]

I would not like "comments as metadata". If you want more metadata, open a separate issue about allowing tags or custom k/v metadata. This is rather something for a future moin release.

For moin 1.1.x I can see us implementing something along the lines what we already have elsewhere in borg, so no totally new concepts. Not sure if we have suffix somewhere, but iirc we have pattern match on archive names.

[ThomasWaldmann]

If you just use a different PREFIX for your multiple mysql backups, you can already handle this with existing borg prune featureset.

[ThomasWaldmann]

The "implicit compact_segments takes time" issue might soon be solvable, after borg 1.2 is released (2021).

[knutov]

Yes, "comments as metadata" is not good, but it works already now. Any additional changes will take time, and may be I'm the only person who need this.

It's ok to filter suffix via regexp to me, but it will be good to have it as commandline parameter too. Should I open additional issue for that?

Do you plan to release stable 1.2 release soon? I mean few weeks or it can take more time?

[ThomasWaldmann]

suffix would be a new feature (IIRC), feel free to open a separate issue to discuss it.

1.2rc1 will come in the next weeks, see there: https://github.com/borgbackup/borg/milestone/75

[knutov]

One more reason to have ability to filter snapshots for prune by some additional metadata:

on https://borgbackup.readthedocs.io/en/stable/quickstart.html there are examples with --prefix '{hostname}-', the problem is hostname (or anything else) can contain symbol - and mostly any other symbol, so there can be partial match: example my.domain.com and my.domain.com-net.org. In case of borg prune with prefix my.domain.com- the second prefix will also be affected.

In real life it is not possible reliable use any symbols in snapsot name as a separator.

[ThomasWaldmann]

See #8379 about a new feature "backup series" - always using the same name and doing a precise match on that name, e.g. for prune.

[ThomasWaldmann]

See #8425 about tags.