That would, absolutely, work too.

Snapshots help, but don't totally solve, this problem.

An APFS snapshot is just that: an instantaneous copy of the entire volume, frozen at a point in time. (Naturally, the filesystem doesn't actually copy the entire volume—it's all just metadata slight-of-hand—but the effect is identical.) Any further modifications to the volume (writing new data, creating new files, deleting files, and so on) happen on the "live" version of the volume and does not alter the snapshot.

This helps make the backup process much more stable and it doesn't have to worry about race conditions like files being added or deleted to a folder after QRecall has read the directory to find out what files it contains.

It does not, however, solve the fundamental problem of data that hasn't been written yet. The issue with virtual machine files (and databases, and large media files, and so on) is that they are still being written to while the capture is in progress. The backup copy is then a half-written file that is unlikely to be usable. Snapshots don't really change this equation, it just changes the point in time that the captured data is out of sync with the file as it will be written.

Snapshots can, however, make it easier to live within these constraints. The best solution QRecall can offer is a slightly modified version of the solution you already have.

You're already doing the right thing—closing all of your VMs before starting the capture. QRecall can automate this if you schedule the VM capture action to start as soon as your VM software application quits. This way, you don't have to remember to do anything; whenever you quit your VM software a capture will start immediately.

The beauty of APFS snapshots means you don't have to wait for the capture to complete to get back to work. As soon as the capture starts, it will have made a snapshot of all of your VM files (in their closed, pristine, state). You are now free to launch your VM software and start working again; new changes to the VM won't be part of snapshot or the capture that's in progress, no matter how long it takes.

I wouldn't recommend creating a new user. After entering your identity key, if you have problems capturing just try restarting your system.

This is a long-standing issue with the way macOS now shares (or doesn't share, to be more precise) preference changes between different applications.

Darryl Tooth wrote:That didn?t work... As soon as I launched QRecall and then went to change the preferences the app just locked up.

Also be aware that when trying to fix installation issues, QRecall my fail to connect with (deleted) services still registered with macOS. This may result in stalls and timeouts, so be patient. During startup or when changing the service configuration, give QRecall a minute or two to fix the problem.

If you just want to rename them (in place), it shouldn't be a problem.

Things get a tad more complicated if you move the archive and rename it, or relocate it to a different volume. But QRecall shouldn't be tricked by simply renaming it.

The worst that can happen is that your actions fail to locate the renamed/moved archive. To rectify that, simply open an action, choose the new archive, and save it again.

Darryl,

Thanks for letting us know about this. First, send a diagnostic report: in the QRecall application choose Help > Send Report…. Screen shots of the log are great, but there's a lot more details behind the scene that the report will capture.

But even before I see the report, I'm going to guess that QRecall is having problems installing its privileged helper. This is a privilege escalation service managed by macOS, and sometimes it get "stuck" for unknown reasons. The result is that macOS doesn't install the new service, yet the service won't run.

When you turn off Capture and recall items using administrative privileges what you're doing is telling QRecall to run without privileges, so it doesn't try (and doesn't get any errors). But without privileges, it also can't capture and restore your system files, take snapshots, make power management requests, and so on.

Try this:

Perform a a quick uninstall of QRecall: In the QRecall app, hold down the Option+Shift keys and choose QRecall > Quit and Uninstall.

Delete the file /Library/PrivilegedHelperTools/com.qrecall.hoist

Delete the file /Library/LaunchDaemons/com.qrecall.hoist.plist

Restart your computer.

Launch QRecall and reinstall it; make sure you've turned Capture and recall items using administrative privileges back on.

If either of those two files existed, then I suspect that was your problem. If that doesn't fix it, we'll know more when your diagnostic report arrives.

Finally, admin privileges and the volume format or protocol used for you archive volume are largely unrelated.

Good news: we're getting closer

The problem was that you weren't running 2.1.14(3). All of the action logs up to now were still running 2.1.14(1). When you launched QRecall to send this report, that was 2.1.14(3) ... which should now be installed.

I suggest

Launch QRecall again and use QRecall > About QRecall… to verify that you are, indeed, still running 2.1.14(3).

Deleting all other copies of the QRecall application you have lying around. It's OK to keep the installer disk images for earlier versions, as long as they're not open.

Continue running 2.1.14(3) to see if it fixes the problem.

Send another report in about a week if you don't see any problems.

Greetings,

As a rule, QRecall isn't designed to protect against changes to your source files, because source files change all the time.

QRecall protects against data corruption by preserving older versions of your files for as long as possible. Its block-level data de-duplication and compression help in that mission by storing your files in as little space as possible, allowing you to keep more history. So if you do discover that a file is corrupted, you have a better chance of recovering an earlier, valid, copy.

There are utilities that will calculate checksums of your file. You can then periodically use that information to detect if any data in your files has changed.

On the other side of the equation, it's a completely different story. QRecall archives have multiple layers of interlocking data integrity checks, so that even a single corrupt bit of data in an archive will be detected. These integrity checks are verified during every operation. It also has the option of storing redundant data so that modest amounts of data corruption can be corrected and recovered.

I hope that answers your questions.

David Ramsey wrote:I'm surprised to see "Frankenstein"; that was the name of my Hackintosh. This backup is running on a Mac Pro 6,1 called "Computronium".

"Frankenstein" is the name of your identity key. When you first install QRecall, your identity key's name defaults to the name of your system. It will remain that until you change it.

To change your identity key, go to QRecall > Preferences > Identity Key and edit the name.

The full syntax for specifying an item in an archive is <owner's name>:<volume name>/dir/.../dir/item, essentially the path you see at the bottom of the archive browser window.

The convenient * syntax means "the one and only owner" or the "one and only volume". But you have more than one volume, so the * syntax can't be used.

There are two easy ways to solve this.

First, you can simply delete the "CCC Backup" volume. You were trying to solve the problem stopping captures of your "CCC Backup" volume, so this seems like the preferable choice. Once you're back down to a single volume, the * volume name will work again. In the browser, navigate to the owner level ("Frankenstein"), select the "CCC Backup" volume, and choose Archive > Delete Items….

Alternatively, you can specify the volume in the item path so there's no ambiguity: qrecall ls '/Volumes/QBackup/Mac Pro Backup' '*rimary/Users/dramsey/Library/Preferences/QRecall/Actions'

It isn't obvious what's wrong, so go through each level one at a time to see where the break is:

qrecall ls '/Volumes/QBackup/Mac Pro Backup'
qrecall ls '/Volumes/QBackup/Mac Pro Backup' '*'
qrecall ls '/Volumes/QBackup/Mac Pro Backup' '*:*'
qrecall ls '/Volumes/QBackup/Mac Pro Backup' '*:*/Users'
qrecall ls '/Volumes/QBackup/Mac Pro Backup' '*:*/Users/dramsey'
qrecall ls '/Volumes/QBackup/Mac Pro Backup' '*:*/Users/dramsey/Library'
qrecall ls '/Volumes/QBackup/Mac Pro Backup' '*:*/Users/dramsey/Library/Preferences'
qrecall ls '/Volumes/QBackup/Mac Pro Backup' '*:*/Users/dramsey/Library/Preferences/QRecall'

The first two should only list one owner and one volume, respectively; if they don't, that's probably the issue.

David Ramsey wrote:* restore process failed: folder not found in selected layers

Hmm, it works here for me...

Basically, the command is saying that /Users/dramsey/Library/Preferences/QRecall/Actions doesn't exist in the latest layer. Are you also sure you only have a single volume and owner? Is /Users/dramsey on a different filesystem?

What happens if you try this:
qrecall ls '/Volumes/QBackup/Mac Pro Backup' '*:*/Users/dramsey/Library/Preferences/QRecall/Actions'

You can try shorter versions of this (i.e. qrecall ls '/Volumes/QBackup/Mac Pro Backup' '*:*/Users') to try and narrow down where the missing piece is.

Lazy is good. Getting computers to do our work for us is how we got this far

You can use the qrecall command tool, which can be added to a Terminal/shell script, executed from an AppleScript, or about a thousand other ways.

qrecall restore <path_to_archive> '*:*/Users/<you>/Library/Preferences/QRecall/Actions'

Note: the shorthand syntax '*:*' (or just ':') works when you have only one owner with one volume in the archive. If you have more than one, of either, you'll need to see man qrecall section on <item> syntax for details about specifying the owner and volume you want to restore from.

David Ramsey wrote:It has the minor drawback of needing to recreate the actions if I ever restore from the backup volume, ...

But those action documents would still be captured in your QRecall archive, so you could just restore them from that backup.

David,

Interesting problem.

QRecall uses "bookmarks" (the modern macOS replacement for aliases), and you are correct; if you create a bookmark of your startup volume, or an item on that startup volume, and later resolve that bookmark when booted from a different startup volume, it will find that item (first) relative to your new startup volume, not the original volume (even if it's online).

I believe you can exclude items from your CCC Backup. So the simplest solution would be to exclude the contents of the ~/Library/Preferences/QRecall/Actions folder. Then when you boot from your external drive, QRecall won't have any actions to run. (And don't forget to delete any old actions on your CCC Backup volume.)

Sorry for the delay in addressing this; it took a little time to reproduce exactly what was happening.

I'm pretty sure this pre-release version will fix it: QRecall 2.1.14(3)

Same drill as before: install 2.1.14(3) and let it run overnight.

David,

Thanks for sending a diagnostic report. It was just want I needed to determine ...

... QRecall is totally at fault!



So here's what's happening,

You have a capture action set to run at 3:00 each day. That action has a condition that holds the action until the archive volume is attached. So the scheduler starts the action at 3:00 (something that shouldn't be happening because Apple's power nap is not supposed to run third-party software when your laptop is closed, but that's another story), but then immediately places the action on hold.

Now here's what's going wrong. We recently made a change to the scheduler so that it prevents your system from going into idle sleep because a few users were having an issue with actions starting at the same time the computer wanted to go to sleep; the action would start, connect to a file server, then immediately go to sleep, which would result in network errors when the computer woke up again.

In your situation, however, this is a disaster. Once your action starts, the scheduler is now preventing the system from going to sleep since actions are "running." But your capture action isn't really running yet because it's held by a condition. So the laptop stays awake until the battery is dead.

I think I've fixed this and you can try this pre-release of QRecall version 2.1.14: This version only treats actions that have started but are NOT being held as "running", for the purposes of determining if the system should be allowed to go to sleep. If you do try this version, let it run for a few days and then send another diagnostic report.

Another solution that would work with the current version is to remove the "hold while no archive" condition and change the schedule to an event schedule that runs the action when the "archive volume connects". Now, the action won't attempt to start until you connect your external drive. If you connect and disconnect your backup drive once a day, that's all you need to do. If you have occasion to leave the drive connected for more than a day, consider adding a "Repeat again" interval so that capture runs again the next day.