My QRecall archive covers four disks, each replacing the prior disk as it nearly failed or I scaled up:
Disk 1: 12/21/13 (290 GB) to 10/20/17 (925 GB)
Disk 2: 10/20/17 (925 GB)
Disk 3: 10/20/17 (925 GB) to 11/7/19 (10 GB)
Disk 4: 4/24/22 (855 GB)

The 10/20/17 backups are all the same size with the same number of items.

For some Very Bad Reasons I stopped regularly backing up my drives in late 2019. I used a rolling merge over those years that kept several days, weeks, fortnight, month, and year layers.

I want to resume my better habits and today created a new backup (Disk 4 that replaced Disk 3 back in 2019) using Capture (took about 23 hours).

My archive file is now 1.4 TB.

I?d like to clean things up and get started again with my previous setup of rolling merge actions and I?m really not sure what my options are.

If feels, FWIW, that having the archive span four files is inefficient.
Can I (or even should I) combine these individual disk backups into one?

Hoping to get some good thoughts on how best to move forward that might also result in a smaller more efficient archive.

Thanks!

Mark

Mark,

Glad to hear you're doing a little spring cleaning.

Your current situation isn't grossly inefficient because all four volumes still share the same data. So any file captured in Disk 4 share its data with the same files in Disk 3, 2, and 1.

But it is a little inefficient. Having multiple copies of the same volume means there's an initial layer for each volume with a complete copy of your entire directory system (essentially all of the file and folder metadata). But since metadata is typically only 1% to 2% of an archive, this isn't a big deal.

It also prevents the merge from discarding the oldest versions of files in Disks 1, 2, and 3 because they're not part of the Disk 4 history.

And finally, it makes it hard to find a really old file because you have to search for it across four different volumes.

Since all four volumes are essentially the same volume, I would recommend combining them. Then the rolling merge, compact, and search will all work the way they're supposed to.

But before you begin, I would examine your rolling merge and see how far into the past it preserves layers. If it's 5 years or less, there's no point in keeping Disk 1 and 2 at all, since the next rolling merge will merge those layers with the layers of Disk 3 (essentially removing the older volumes). If this is the case, I'd recommend you start by deleting volumes Disk 1 and Disk 2 from the archive, and then combine the remaining Disk 3 and Disk 4.

If your rolling merge does go back more than 5 years, and you really want to keep all of that history, then just merge all of the volumes. And it's important to merge them all at once; don't to it piecemeal or you may not be able to merge some of them. Navigate to the root of the archive, select all of the volumes, and choose Archive > Combine Items.

If the combine is successful (there are some obscure technical reasons why it can't be done), the history of all four volumes will be combined and you'll be left with a single volume (Disk 4) with a unified history. If the volumes can't be combined, try combining just Disk 2 through 4, or just 3 and 4, and then decide how long you want to keep the history in Disk 1 (and 2).

In the end, your storage should be slightly more efficient and the next compact action will probably reduce the save of your archive.

Good luck!

Yes, currently doing Spring cleaning 2020 for my computer. Way ahead of where I am for the apartment which may require organizing an archeological dig.

And this is interesting:
It looks like the decision has been taken from my hands. I still have all four disks but they now show as:
Disk 1: 11/17/19 (1.55 TB) 2,041,696 items
Disk 2: 11/17/19 (1.55 TB) 2,041,696 items
Disk 3: 11/17/19 (1.55 TB) 2,041,696 items
Disk 4: 04/24/22 (855 GB) 559,260 items (layer 1), 04/25/22 (9.5 GB) 669 items (layer 2)

Disks 1?3 indicate only one layer each (down from quite a few layers in each that I saw a couple days ago); there are two layers for Disk 4.

I'm pretty sure that my rolling merge from years ago maxed out at 2 or three years. So I think what probably happened was I must have activated the schedule without realizing it and a merge of some sort was performed after yesterday's capture. I'm interpreting this new state as there is only one layer duplicated three times.
Does that make sense given what I see now?

A quick look in one of those disks' ~/Documents folder shows I have some files going back to The Early Days so I guess everything is flattened but safe. I imagine it's what I would have done anyway but would have preferred agonizing over the choice for a few days. My mistake.

Given all this, should I delete Disks 1 and 2 as duplicates of Disk 3 and move forward with just Disks 3 and 4?

I should clarify that all four disks I'm referring to are listed under one owner/volume. I can only select one at at time and the Archive>Combine Items? menu is grayed out. Is combining still an option or is it moot since I guess I can select the root volume and search there?

When I select the Volume, I see only three layers (1: 1.55 TB; 2: 855 GB; 3: 9.5 GB). That's confusing to me given the individual disks listed. Can I actually delete Disks 1 and 2?

And then, as you wrote, my next compact will drastically reduce the size of the archive and reduce the time spent during capture/merge actions. Hope I have that right.

Mark Gerber wrote:I'm pretty sure that my rolling merge from years ago maxed out at 2 or three years. So I think what probably happened was I must have activated the schedule without realizing it and a merge of some sort was performed after yesterday's capture.

You think correctly. The rolling merge has combined all of the history of Disk 1, Disk 2, and Disk 3 into a single layer, keeping only the last captured items in each volume.

However, these are not duplicates. The single layer contains all three volumes. The contents of those volume are separate from one another, and you'd have to look at the capture date of the volume (or any item in the volume) to tell how far back it goes.

I'm interpreting this new state as there is only one layer duplicated three times.

There is a single layer that contains three separate volumes.

A quick look in one of those disks' ~/Documents folder shows I have some files going back to The Early Days so I guess everything is flattened but safe. I imagine it's what I would have done anyway but would have preferred agonizing over the choice for a few days.

Correct and correct.

Given all this, should I delete Disks 1 and 2 as duplicates of Disk 3 and move forward with just Disks 3 and 4?

Given that your rolling merge only maintains about three years of history, there's no point in trying to combine Disk 1, 2, or 3 with anything. As soon as you combine these volumes with Disk 4, the next merge will delete them (because they're too old).

I suggest you simply select Disk 1, Disk 2, and Disk 3 and delete them (Archive >Delete Items). It will be a lot faster than combining them, only to have the next merge discard them.

I should clarify that all four disks I'm referring to are listed under one owner/volume. I can only select one at at time and the Archive>Combine Items? menu is grayed out.

An archive contains owners. Owners contain volumes (disks). Volumes contain files and folders. If you open an owner in the browser and are looking at a list of volumes, you should absolutely be able to select more than one using Shift+click or Command+Click. Try a different view (list view, for example) if you're having problems.

I suggest switching to list view, select Disk 1, then while holding the Command key, click-select Disk 2 and Disk 3. Now you can select Archive > Delete Items to remove all three at once.

And then, as you wrote, my next compact will drastically reduce the size of the archive and reduce the time spent during capture/merge actions. Hope I have that right.

Absolutely correct, with the possible exception of excluding the word "drastically".

QRecall's data de-duplication means that the same file captured in Disk 4, Disk 3, Disk 2, and Disk 1 is only stored once, and deleting three of those references doesn't remove that data. It does reduce the meta data for the other three references, but metadata records are typically only about 1%-2% of an archive. Capture won't go much faster because capture is only comparing new files with the volume being recaptured (Disk 4). The contents of Disk 3, 2, and 1 are irrelevant and are not consulted during the recapture.

The actions that will be substantially faster are merge, compact, and verify, since you will have removed hundreds of millions of file and folder records that no longer need to be considered.