Looking at the code, the layer size should read "Unknown" for a repaired layer. Not sure why you're seeing Zero KB, but I'll dig a little deeper into it. Probably a bug.

Following a repair, QRecall makes the following guarantees:

All items (unless you set some of the special repair options) in a layer have been verified to be undamaged and can be recalled.

Layers that are not marked as "repaired" are complete and aren't missing any items

The inverse implication is that layered marked as "repaired" might be missing items?there's no way for QRecall to tell?but the items that are there are OK.

Now to cover some of the minutia of the repair process...

A "Zero KB" just mean that QRecall couldn't (easily) reconstruct the size of that layer. Like the Finder, QRecall would have to scan every folder and item in a layer to fully reconstruct its size, and repair won't do this more than one folder level deep. Maybe that's a bug. Maybe it's just "hey, the layer was damaged; there's stuff about it we don't know anymore, sorry." So those layers aren't empty and you should find items in them if you browse them on their own (move both layer shades so only that layer is visible).

About the dates and layer order. If layer records are missing, the repair can reconstruct most of the information about that layer from the individual folder and file records. Every folder record has a "layer ID", so by scanning the folder records it's possible to put them back into their original position in the layers. But if the layer records were damaged or missing, some information (like that date that layer was captured/merged), has to be inferred from the dates of the individual folders. It should to be close, but it won't be exact.

Finally, if you merge a repaired layer with a subsequent layer and that subsequent layer completely captured all the same items, that repaired layer will disappear. If not, then the merged layer will still have remnants of the repaired layer and will also be marked as "repaired".

Let me know if that helps, or confuses the matter even more.

Nico Dudek wrote:Hi James, First I hope you are healthy and well!

Everyone here is well, thank you!

After copying the archive I was asked to reindex it. Is this normal and worth to do? Or would you recommend a new archive instead of moving the old one?

That is not normal. Simply copying an archive should not corrupt it. However, if some other action were to modify the archive while it was being copied, then that will mess up the copy. Before copying/moving an archive, I suggest you hold all scheduled actions to make sure nothing starts during the copy.

You can try a reindex. A reindex simply rebuilds the various index files QRecall needs to access the archive data more efficiently, but it does not in any way try to repair that data. So if a reindex is successful, your archive is intact and undamaged. If the reindex fails then the archive data was corrupted during the copy. If you don't still have the original, it's easiest to just start over. If you do have the original, verify it and then try copying it again (making sure no other actions are running).

As a rule, you can freely relocate and rename archives as you please. After moving it, open up an action for that archive, click on the archive action button, pick "Choose Archive", and select the relocated archive. That action now refers to the archive at its new location. Save and close that action. If you have other actions, QRecall will offer to relocate those for you.

Let us know if you have any other questions.

Prey Lawn wrote:From where can I purchase a key?

Identity keys are available on the QRecall website. See the "One Key" section to purchase a key.

Without seeing the errors that you're getting, I'm going to take a wild guess and suggest this is probably a keychain problem.

QRecall will attempt to mount networked volumes before it starts an action.

Most networked volume require authentication (typically an account name and password) to connect. To mount them automatically, that authentication information can be stored on your keychain.

Therefore, the requirements of automatically mounting a volume require that your keychain be open, which implies that your user is logged in and active. If you've logged out or you have security set up so your keychain gets closed after a period of inactivity, QRecall can't access the credentials necessary to mount the volume.

If that's the problem, the simple solutions are (a) change your security settings so your keychain doesn't get locked automatically or (b) pre-mount the networked volume so QRecall doesn't need to mount it when the action starts.

Let us know if that helps.

Norbert Karls wrote:Hi again,

Hello!

QRecallHelper will not exceed its allowance of »wired« or »reserved« memory, but it will take on more and more »virtual« memory over time

Note that QRecall uses a lot of "mapped" files. So most of its virtual memory is really "virtual." In other words, it's just reserving address space; it doesn't mean any actual memory is involved (although some will, eventually) and it is separate from the VM backing store.

For example, when a process "maps" a 500MB file to memory what happens is the system simply reserves a span of logical memory addresses (say, 0x0006000c6a0000000 through 0x0006000c6bfffffff) and sets up the contents of that file to correspond to those addresses. The "virtual" memory of the process immediately increases by 0.5 GB, but nothing has really happened; no actually memory has been allocated or used at this point. The real measure of a process' impact is how much of the VM store and heap space it has consumed, which you can't tell from a casual look at its virtual memory footprint. That's why the latest macOS and Activity Monitor app has a "memory pressure" metric that takes these kinds of things into account.

The logs don't mention any reason for the process ending, there's just the same chatter on and on for hours and then it just stops. A caught error leading to termination would be logged and could then be found near the end of the logfile; this is not the case, though.

That would indicate that the process is crashing for some reason, and it appears to be repeatable. That information would be in a system crash log. If you send a diagnostic report from QRecall (Help > Send Report...) those crash logs will be sent to use and could indicate what's failing.

I then put my hopes into trying an incremental approach

The repair (and index) is really an "all or nothing" proposition. It has to finish, or it has to start over.

Finally I looked around the various options to the commands, hoping to find a way to have it just extract individual files or layers or some way to force reindexing ignoring errors to create an index that would allow me to get something instead of nothing, but without any luck.

That's literally what the repair command does, so we need to figure out why it's crashing. (Note that a program can ignore errors, it can't ignore a crash.)

Here's a thing that came to my mind: A repair action will first dig through the entirety of an archive analyzing it before then starting to take action (with the exception of a reindex, but it refuses that here). Couldn't we have some kind of scavenge mode, especially with the --recover option, that will start writing a new archive immediately, adding files and layers at the pace they are discovered in the source archive?

That's pretty much what it's already doing .. the problem is the crash. However, it did occur to me that some aspects of the repair, like dealing with damaged records, could be made more incremental and I'll add that to the wish list.

Do stay healthy

We're in this together!

Steven,

I'm afraid you'll have to ignore this one.

The com.apple.TCC file contains sensitive information that Apple doesn't want malware to have access to. QRecall works around most file restrictions with special privileges, but the com.apple.TCC directory seems to be different. Also note that it's very inconsistent; some users report problems reading this file, while most systems capture it every day just fine.

The folder contains a database that stores, among other things, the security preferences and allowances defined for various apps. Obviously, macOS is super paranoid about letting a bad actor modify (or even read) this information.

The dvp database is similarly protected.

I would suggest adding an exclusion to your archive that skips these items. Should you need to perform a complete restore of your start-up volume, it just means you'll need to re-grant apps access to your private information (contacts, external volumes, etc). Which is probably a good thing to review from time to time anyway.

That is correct. When you change identity keys, you are a new "owner" so all of your items will be captured for the first time.

Of course, since the archive you're using has already captured these items, most (if not all) of the data is duplicate. What's taking time is reading each block of every file, looking that up that block in the archive's database, and comparing it to make sure it's a duplicate. That's a lot of work.

If you're not concerned with the (short) history of changes you already have, then it won't hurt to simply wipe the archive and start over: choose File > New Archive and overwrite your existing archive with an empty one. Then restart the capture.

If you do want to keep your history, let the capture finish (or stop the one in progress). Open the archive, navigate to the root, choose both owners, and choose Archive > Combine Items.... This will change the owner of all of your previously captured items so they now belong to your new owner (identity). From there on out, it will be as if you had started with your permanent identity key from the beginning.

I hope that helps.

That's the "next action" from the scheduler. You could try restarting the scheduler, but it should (eventually) just go away.

Is it an action that's running? (Or one that has stopped with an error?) The activity window only shows the progress of running actions, the results of failed ones, and the next scheduled action. Deleting an action document, archive, etc. won't affect the display of a process that's already started/finished.

If you think it's cosmetic issue, you can try restarting the monitor (Quit/SIGTERM the QRecall Monitor process).

maxbraketorque wrote:For the second question, there are no orphaned actions listed in the Actions window.

Oh, I think you might be referring to the Status window, not the actions or activity window.

The Status window shows the state of all known, and previously known, archives. To make QRecall forget about the status of an archive, click on the action (gear) button, or Right/Control-click anywhere in that pane, then choose Forget.

QRecall doesn't track files. It doesn't matter if you delete a file and create a new one, or move the file, or rename the file, or swap two files, or .... well you get it. All QRecall knows is that a file that was there yesterday no longer exists, and a file that wasn't there yesterday now exists.

The disappearance of the old file will be noted, and the new file will be captured, de-duplicating all of its data against what has been previously captured.

So no new data gets added to the archive (except a little meta-data), but it does take time to perform all of that de-duplication.

If fast captures during the day are important and you have plenty of extra disk space, there is a feature just for this. In the capture action there's a "defer de-duplication" option. If you set that, no data de-duplication is performed during the capture. All new file data (~80GB) is simply appended to the archive. The next compact action that runs first begins by de-duplicating all of that deferred data. Note that this is slower than if the data had been de-duplicated during the capture, but it does give you the option of performing the de-duplication when it is more convenient.

maxbraketorque wrote:1) I decided to break out each of my four external drives into their own automated QR archive using the QR Assistant. If I want all four archive backups to run back-to-back, can I set their action times to all be say 15 minutes apart, e.g., start actions for backup1 at 8pm, actions for backup2 at 8:10 pm, etc? Will sets of actions for each archive run serially with sets of actions from other archives, or will each set of actions run in parallel if the prior set of actions for the archive before it haven't finished?

Overlapping actions that modify an archive will run sequentially (only one action can be modifying an archive at a time). Overlapping actions that just read an archive (recall, verify) and actions on other archives will all run concurrently.

If that introduces performance or resource issues there are two settings in the QRecall Scheduler preferences that can help:

Maximum concurrent actions and Maximum actions per volume.

If you set Maximum concurrent actions to 1, only one scheduled action will run at a time.

(*)These have no effect on commands initiated from the UI or the command-line.

2) When I was first experimenting with using QR, I created a few archives that I subsequently manually renamed and then subsequently deleted. Some actions for these deleted archives seem to be persisting in the QR monitor window. How do I find and remove these actions?

Open the Window > Actions window and find the actions that no longer have archives. Select them. Delete them.

See QRecall Help > Guide > Troubleshooting > Problems > Can't Open Archive

... and let us know if that helps.

QRecall does not copy files. It breaks them into blocks of data and adds the unique blocks to a massive database.

As the archive grows, the corpus of previously captured data grows. Every new block of data must query the captured set to determine if it's unique. This involves several layers of hash tables and indexes, and many of these tests will require data to be read from the archive, usually in a very random manner.

So I/O performance must be less than what you'd see if you simply wrote the files. There will be occasional reading of the archive during a capture, and the drive's seek time becomes an important performance metric. In short, it's a lot of work.