David,

Thanks for the diagnostic report.

Your logs indicate that there's a corrupt value in the layers.index file. This is an auxiliary index file that summarizes the directory structure of the entire archive.

A reindex of the archive should fix it.

Your logs also show that you started a repair on 11-16 a 12:25. It was canceled about 50 minutes layer, and it promptly stopped at 13:11.

I suspect that starting another repair, and letting it finish, will fix the problem.

If not, please send another diagnostic report after the repair.

David,

Sorry to hear you're having problems. I'm sure there are going to be some edge cases to work out, but I have several computers here running Big Sur and they're all capturing files as I type this.

Next step is to send a diagnostic report and take a look at exactly what those errors are.

Everyone safe and healthy here! I hope everyone else reading this is too.

FileVault and QRecall play just fine together ... now.

In the very early days of OS X, FileVault was implemented as a special mount point, backed by a hidden disk image file that handed the encryption. This complicated things in numerous ways for QRecall. First, QRecall captures volumes, and your home folder was now on a different volume. You couldn't capture files for another user if they weren't logged in. And if you restored your files, FileVault might make them all disappear when you logged in again. *sigh*

Mac OS X Lion (10.7) introduced FileVault 2 (officially known as "FileVault," with the earlier version being referred to as "Legacy FileVault"). FileVault 2 encrypts the entire volume at the block level and is completely transparent to the filesystem and QRecall.

If you want the same level of protection for your QRecall archive, you can enable encryption. (Make sure you make a backup copy of your encryption keys and keep them safe.)

Yes, this year has not been great.

However, we've made a tremendous amount of progress on QRecall 3.0. The main thing slowing it down has just been the massive number of new features and changes.

So far, we've got completely re-engineered archive file management to take advantage of APFS features. Record optimization and new multi-threaded record processing for faster database access. And we're updating the code base to get ready for ARM (Apple Silicon) computers.

We've added support for capturing APFS volumes, including the ability to capture and restore sparse files.

But the huge new feature, and the one taking all of the time, is Stacks.

A stack is a remote, incremental, copy of your archive, organized into "slices." Essentially a backup of your backup. But unlike the archive, which is an interactive database, a stack is written in serial, compact, slices suitable for efficient cloud storage. We have basic document stacks working now, and would like to have at least one cloud service ready for the beta. Ultimately, we will offer several different cloud services (Amazon S3, Dropbox, SFTP, ...) to choose from.

Still working on the code that allows you to recover/repair an archive from a stack, but most of the stack functionality is already up and running, and we're using it in-house.

So the year hasn't been a total loss

The principle reason to merge layers is to trim/simplify the history of incremental backups in your archive. This frees up the space occupied by those earlier layers, which becomes available for new captures. So instead of growing forever, your archive becomes a conveyer belt: capturing new changes while letting ancient changes fall of the other end.

If you have sufficient disk space, you don't care how big your archive gets, and you don't mind wading through hundreds of layers to find an old version of something, then you don't have to merge layers.

The only difference between a "merge" and a "rolling merge" is that the rolling merge automatically selects groups of layers to merge based on a rolling calendar. Other than that, there's no difference. Or said another way, a simple merge will merge a range of layers based on a formula that you give it, while a rolling merge picks sets of layers to merge based on calendar intervals you choose.

If you want the traditional "rolling backups" provided by conventional backup solutions, the rolling merge automates that for you. But if you want absolute control on what gets merged and when, then the simple merge action (or doing it manually in the QRecall app, or rolling your own logic using the command-line tool) is at your disposal.

Finally, if you have a merge action, see if it has conditions. For example, if you tell the capture assist that you want to keep as much history as possible, it will create a merge action that merges the oldest layers in the archive, but only if the free space on the disk is below a certain threshold. The end result is a merge action that never runs until the disk starts to fill up. It then runs every day until the size of the archive is reduced enough to to stop it running again.

I hope that helps!

Jon,

Got your diagnostic report. And yes, QRecall cannot start its privileged helper. You also have a lot of "not found" errors on your archives, but we'll need to straighten out the helper first.

This problem is always an installation problem. To gain elevated privileges, QRecall must request that macOS install a "privileged helper" on its behalf. Sometimes (and I'm not always sure why) this doesn't happen or it gets messed up. Once messed up, a new helper can't be installed and QRecall can't un-install itself (because un-installing requires the privileged helper). A true catch-22.

The fix is simply to manually uninstall QRecall. This takes a little bit of work, but I'll give you a shortcut.

The steps are in the help under QRecall > Help > Guide > Advanced > Uninstall > Uninstall QRecall (the hard way). There's also a copy on the web.

As a shortcut, start by performing just steps 3 and 4. (I suspect this is the root of the problem.) After restarting, launch QRecall. It should prompt for you admin credentials and reinstall QRecall. If it doesn't prompt you, make sure QRecall > Preferences > Authorization > "Capture and recall using administrative privileges" setting is turned on.

If that doesn't solve the problem, perform the entire un-install procedure (make sure you skip the ? "dagger" steps and step 9, as you don't want to start over when you re-install). Restart again and then launch QRecall.

If this still doesn't solve the "can't launch privileged helper" problem, please send another diagnostic report (QRecall > Help > Send Report) and we'll dig deeper.

What's most likely is that the B1 volume got reformatted/resized/whatever and appears to QRecall as two different volumes. You can confirm this by looking at the timeline (or info sidebar) and see that all of the captures of the first B1 occur before the second B1.

If you want to keep all of this history, simply select both B1 icons and choose Archive > Combine Items. This command takes two volumes, or two owners, that are actually the same volume/owner and stitches them together to form a single item with one time lime.

Alternatively, if you're not interested in that much history, find the older B1 (again, use the info sidebar) and delete it (Archive > Delete Item).

Working backwards...

Yes, redundancy increases the size of the archive. If you selected the default redundancy level (1:8) then every eight K of data is accompanied by an additional one K of redundant information, making everything you store 12.5% bigger.

There are also meta data, index files, and other database related information in the archive which makes it bigger than the actual file data. But that typically doesn't account for more than a couple of percentage points.

The "insufficient free space" message is the compact action trying not to waste time and endanger your data by moving it around pointlessly. The compact action won't physically compact the archive unless at least 12% of the archive's database file is empty space. This prevents the compact action from moving hundreds of Gigabytes of data, only to recover 4K. You can adjust this threshold in the advanced settings ("Compact Free Space Ratio Minimum"), or you can run the compact from the QRecall menu: open the archive in the browser, then choose Archive > Compact. When run from the menu, it ignores the free space ratio.

But while you have the archive open, make sure you navigate to the root of the archive to make sure you don't have any other owners or volumes that might contain older versions of data. If you do, consider deleting the older owners/volumes (before compacting).

Finally, if you want the smallest possible archive size for files you're keeping for posterity, consider turning on compression in the archive's settings. If you up the compact compression level, then compact the archive, the entire archive will be compressed. (Note it only does this once, and once compressed it won't uncompress or recompress the data). If you still need even more space back, consider reducing the redundancy level.

I hope that helps!

It's hard to say exactly what was going on, but I wouldn't say this behavior is surprising.

When a volume fills up, it becomes harder and harder for the filesystem to locate free clusters needed while writing, and files become increasingly fragmented. If the volume is really really full, this slowdown can be excruciating; like 1,000 or more times slower than normal. In other words, an operation that would normally have finished in a 1/10 of a second could take a minute or more.

The first compact after a capture performs garbage collection, which results in rewriting several quanta index files. If this was happening on a nearly full drive, I can imagine abysmal performance.

Your clean up of 300 MB was probably enough to save the volume from fragmentation purgatory, and as soon as the compact was stopped it would have deleted its own temporary index files, freeing more space, and probably restoring much of the filesystem's typical performance.

Looking at the code, the layer size should read "Unknown" for a repaired layer. Not sure why you're seeing Zero KB, but I'll dig a little deeper into it. Probably a bug.

Following a repair, QRecall makes the following guarantees:

All items (unless you set some of the special repair options) in a layer have been verified to be undamaged and can be recalled.

Layers that are not marked as "repaired" are complete and aren't missing any items

The inverse implication is that layered marked as "repaired" might be missing items?there's no way for QRecall to tell?but the items that are there are OK.

Now to cover some of the minutia of the repair process...

A "Zero KB" just mean that QRecall couldn't (easily) reconstruct the size of that layer. Like the Finder, QRecall would have to scan every folder and item in a layer to fully reconstruct its size, and repair won't do this more than one folder level deep. Maybe that's a bug. Maybe it's just "hey, the layer was damaged; there's stuff about it we don't know anymore, sorry." So those layers aren't empty and you should find items in them if you browse them on their own (move both layer shades so only that layer is visible).

About the dates and layer order. If layer records are missing, the repair can reconstruct most of the information about that layer from the individual folder and file records. Every folder record has a "layer ID", so by scanning the folder records it's possible to put them back into their original position in the layers. But if the layer records were damaged or missing, some information (like that date that layer was captured/merged), has to be inferred from the dates of the individual folders. It should to be close, but it won't be exact.

Finally, if you merge a repaired layer with a subsequent layer and that subsequent layer completely captured all the same items, that repaired layer will disappear. If not, then the merged layer will still have remnants of the repaired layer and will also be marked as "repaired".

Let me know if that helps, or confuses the matter even more.

Nico Dudek wrote:Hi James, First I hope you are healthy and well!

Everyone here is well, thank you!

After copying the archive I was asked to reindex it. Is this normal and worth to do? Or would you recommend a new archive instead of moving the old one?

That is not normal. Simply copying an archive should not corrupt it. However, if some other action were to modify the archive while it was being copied, then that will mess up the copy. Before copying/moving an archive, I suggest you hold all scheduled actions to make sure nothing starts during the copy.

You can try a reindex. A reindex simply rebuilds the various index files QRecall needs to access the archive data more efficiently, but it does not in any way try to repair that data. So if a reindex is successful, your archive is intact and undamaged. If the reindex fails then the archive data was corrupted during the copy. If you don't still have the original, it's easiest to just start over. If you do have the original, verify it and then try copying it again (making sure no other actions are running).

As a rule, you can freely relocate and rename archives as you please. After moving it, open up an action for that archive, click on the archive action button, pick "Choose Archive", and select the relocated archive. That action now refers to the archive at its new location. Save and close that action. If you have other actions, QRecall will offer to relocate those for you.

Let us know if you have any other questions.

Prey Lawn wrote:From where can I purchase a key?

Identity keys are available on the QRecall website. See the "One Key" section to purchase a key.

Without seeing the errors that you're getting, I'm going to take a wild guess and suggest this is probably a keychain problem.

QRecall will attempt to mount networked volumes before it starts an action.

Most networked volume require authentication (typically an account name and password) to connect. To mount them automatically, that authentication information can be stored on your keychain.

Therefore, the requirements of automatically mounting a volume require that your keychain be open, which implies that your user is logged in and active. If you've logged out or you have security set up so your keychain gets closed after a period of inactivity, QRecall can't access the credentials necessary to mount the volume.

If that's the problem, the simple solutions are (a) change your security settings so your keychain doesn't get locked automatically or (b) pre-mount the networked volume so QRecall doesn't need to mount it when the action starts.

Let us know if that helps.

Norbert Karls wrote:Hi again,

Hello!

QRecallHelper will not exceed its allowance of »wired« or »reserved« memory, but it will take on more and more »virtual« memory over time

Note that QRecall uses a lot of "mapped" files. So most of its virtual memory is really "virtual." In other words, it's just reserving address space; it doesn't mean any actual memory is involved (although some will, eventually) and it is separate from the VM backing store.

For example, when a process "maps" a 500MB file to memory what happens is the system simply reserves a span of logical memory addresses (say, 0x0006000c6a0000000 through 0x0006000c6bfffffff) and sets up the contents of that file to correspond to those addresses. The "virtual" memory of the process immediately increases by 0.5 GB, but nothing has really happened; no actually memory has been allocated or used at this point. The real measure of a process' impact is how much of the VM store and heap space it has consumed, which you can't tell from a casual look at its virtual memory footprint. That's why the latest macOS and Activity Monitor app has a "memory pressure" metric that takes these kinds of things into account.

The logs don't mention any reason for the process ending, there's just the same chatter on and on for hours and then it just stops. A caught error leading to termination would be logged and could then be found near the end of the logfile; this is not the case, though.

That would indicate that the process is crashing for some reason, and it appears to be repeatable. That information would be in a system crash log. If you send a diagnostic report from QRecall (Help > Send Report...) those crash logs will be sent to use and could indicate what's failing.

I then put my hopes into trying an incremental approach

The repair (and index) is really an "all or nothing" proposition. It has to finish, or it has to start over.

Finally I looked around the various options to the commands, hoping to find a way to have it just extract individual files or layers or some way to force reindexing ignoring errors to create an index that would allow me to get something instead of nothing, but without any luck.

That's literally what the repair command does, so we need to figure out why it's crashing. (Note that a program can ignore errors, it can't ignore a crash.)

Here's a thing that came to my mind: A repair action will first dig through the entirety of an archive analyzing it before then starting to take action (with the exception of a reindex, but it refuses that here). Couldn't we have some kind of scavenge mode, especially with the --recover option, that will start writing a new archive immediately, adding files and layers at the pace they are discovered in the source archive?

That's pretty much what it's already doing .. the problem is the crash. However, it did occur to me that some aspects of the repair, like dealing with damaged records, could be made more incremental and I'll add that to the wish list.

Do stay healthy

We're in this together!