Mike,

Corruption during a single copy is highly unlikely. My advice would be to simply copy the archive(s) from A to C and then verify the archive(s) on C. Verify checks the validity of every byte of data in the archive. If any data was corrupted during the copy, verify will discover it.

I would also suggest suspending your A1 action schedule (or suspend all schedules) for the duration of the transfer.

Once the archives are copied, open any A action and change its archive location to the new archive on the C drive. Save the action. QRecall will ask if you want to relocate all other actions that use the same archive to the new location; choose "yes." Repeat for A2 and A3.

Unmount A and then un-suspend your action schedules.

All really good questions. I wish I had more answers.

From what little testing I've been able to do[1], here's what I surmise.

If the Time Machine (or QRecall) backup is macOS 10.12, then even if you use the recovery partition you'll end up restoring a macOS 10.12 installation, which must be on an HFS+ volume?10.12 can't read APFS volumes, so it could never boot.

So the only way to directly restore an APFS volume with macOS 10.13 is to first install 10.13, make a backup, then use the recovery partition to restore it.

What I suggested earlier should work, either (a) restore from a 10.12 system backup and then run the 10.13 installer over it or (b) install a virgin copy of 10.13, then use your 10.12 backup to recover just your applications and user documents.

[1] I've had very little luck creating an APFS volume on a mechanical HD for testing. I've made five attempts, and the only thing I've managed to do so far is render two hard drives, partitioned with different versions of the OS for testing, unreadable. I'd have had to completely wipe and reparation the drives twice now.

High Sierra seems to work just fine on HFS+, but I'm beginning to side with Apple that APFS on HHDs isn't ready for prime time.

Steven,

As far as I've been able to tell, you shouldn't have any problems restoring your apps and user documents.

Here's what I know so far:

APFS doesn't introduce any new filesystem types so everything that was supported before (the various file types, file metadata, extended attributes, compressed files, access control lists, and so on) remain the same. So you shouldn't have any problems using QRecall to capture, and later restore, any of your user-land files.

The same is probably true for the system files, but this needs more testing.

What I do know is that, as of today, Apple has made it clear that Apple's macOS installer is the only tool that can create a bootable APFS volume. In other words, you can format a new volume using APFS, you can convert an existing volume from HFS+ to APFS, and you can copy all of the system files to it, but you can't bless it and it will never boot.

Apple has indicated they intend to correct this limitation. But until they do, the only way to restore a High Sierra startup volume is to format it using APFS, use QRecall (or any other utility) to copy all of the files back on it, and then run the macOS installer (either from a second volume or using the remote installer) to install the OS files over the files you recovered.

Good question.

I've been so consumed with trying to get the next version of QRecall working, I haven't had a chance to run comprehensive tests on QRecall and macOS 10.13.

On the other hand, QRecall usage statistics collected by the server show at least 40+ customers running QRecall 2.0.8 on the beta of macOS 10.13 and I've yet to receive a single bug report, so I'm continuing under the assumption that it's not a complete disaster.

As soon as I've had a chance to do some tests, I'll update this thread.

Postscript

One more thing: if you really want to see what the effect of using two identity keys are, get two trail keys from the website. Use them to create individual archives for each computer, then create a single archive and capture both computer to that. It may take a couple of days to test these out (four full, first-time, captures over a networked storage device), but it will definitively show how much storage savings you'll get.

Tom,

It's great that you're looking at QRecall for your data integrity needs.

The biggest advantage of capturing multiple computers to a single archive is storage efficiency. Any common data you both back up (example: you both keep a copy of the same 100 GB of music/movies on your computers) will be consolidated into a single copy in the archive. A minor advantage is that regular maintenance can be relegated to one computer, usually the one with the fastest connection.

The downside to sharing a single archive is that only one computer can update the archive at a time, so regular capture operations have to be performed serially which might not be convenient. And as archive size grows, the speed of operations decreases as the complexity of duplicate data analysis increases exponentially with size. So a 2TB archive will be slower to use than two 1.2TB archives.

Honestly, since you're only talking 2 computers and I doubt you have huge amounts of duplicate data between you, I would suggest starting with two archives and seeing how it goes. If you start to bump up against the storage limits of your device you can revisit this arrangement.

Nothing much as changed on this front.

I've contemplated what it would take to transfer items from one archive to another. While I can see some appeal, the complexities (and code) are significant.

I'm not saying this will never happen, but there's still a lot of features on the to-do list with better benefit/cost ratios.

John,

Each action is stored in a .qraction document inside the ~/Library/Preferences/QRecall/Actions folder. Simply copy these from your old system (or restore them from an archive).

If you alter the contents of this folder manually, you'll need to restart the QRecallScheduler process if you want QRecall to see the changes you made immediately. A system restart is the simplest solution, but you can also Quit the scheduler using Activity Monitor / Terminal and then quit and relaunch the QRecall application.

Note: each QRecall action identifies its archive using a bookmark (the modern equivalent of a Finder alias). Moving the archive or the bookmark to a new system might break the bookmark. If that happens, update the bookmark by opening each action and reselecting its archive.

First, try simply restarting your system.

There's a (now) long standing bug in macOS where an updated preference settings belonging to one process (the QRecall application) can't be read by another process (the QRecallHelper tool).

If that's the problem, a simple restart will usually break the deadlock.

Mike M wrote:Getting errors related to "identity key", some "lost process" too

The "identity key" and "lost process" errors are both bugs in macOS. We're hard at work on a new version of QRecall that should address these issues, but looking at your logs it appears that all actions (except the "identity key" ones) all ran and completed successfully.

If you have a question about the success of an action, the definitive resource is the log. If an action logs "<Some Action> complete" then that action ran successfully to completion. Each action runs as a separate process and the the only time it logs that message when it has started, accomplished its task without any major errors, and closed the archive. So if an actions logs "Capture complete", that capture was successful, no matter what else happened in the mean time.

The "identity key" errors are the result of a long standing, and rather annoying, bug in macOS. It has to do deadlocks that can occur when multiple tasks try to access the same preferences file. The Core Foundation framework is supposed to handle this, but starting around OS X 10.9 it sometimes gets stuck and never recovers. Some people have reported that sending a TERM signal to the cfprefsd daemon will sometime fix it, but a restart of your system will almost always resolve it. You encountered these on two days, and after restarting your system the next day the errors went away.

Note: I have six archives total : 3 archives on Disk 1 and 3 similar archives on Disk 2. I rotate Disk 1 and Disk 2 every week to an offsite storage area. This means that every week, a large number of actions are marked "waiting for archive" ... for instance while Disk 1 is offsite, every action for Disk 1 is marked "waiting for archive," and the next week same with Disk 2.

I do not know if this "offsite rotation scheme" is causing problems.

They aren't causing problems, but you'd probably be better off with a slightly different configuration of actions.

All of your actions have a "hold if no archive" condition. I assume this was left over from using the capture assistant to create them.

The "hold if no archive" is really intended for situations where you are capturing to a single external drive which is not always connected—a common case for external drives used with laptops. This condition starts the action but then places it on "hold" until the drive is mounted; it acts as a prompt that QRecall is waiting for the archive media and you should plug it in now.

Your situation is a little different. You have a set of actions that are expected not to run when their external drive isn't connected. I would suggest removing the "hold if no archive" condition and and replacing it with an "ignore if no archive" condition. This condition simply skips an action that targets an archive which is currently off-line.

Mike,

Thanks for the diagnostic report.

It's a bug.

The action terminated with an uncaught exception when trying to obtain exclusive access to the archive.

It's a random thing, unlikely to happen again, and doesn't involve any data loss. The code has been fixed and should appear in the next release.

QRecall is telling you that you (or specifically, your user account) doesn't have permission to create/write a document in the root directory of that volume.

A QRecall archive is much like any document: you must have read, write, and execute permission in the directory you want to create it.

There are a couple of easy ways to accomplish this:

Ignore ownership

If you plan to use the entire volume for backup, it might be easier to simply turn off permissions and ownership for that volume. Select the volume in the Finder, choose Get Info, and check the "Ignore ownership" option. Once you do this, macOS will let you read and write anything on the volume.

Fix the permissions

Change the ownerships and/or permissions of the directory you want to use. If you can't change the root directory, create a sub folder and make yourself its owner. Note that any other computers/users that require access the same archive will also need appropriate permissions. (This is why the "ignore ownership" option is the simplest.)

Final note: while QRecall makes every effort to support all major filesystems for hosting archives, it's happiest (and most efficient) when your archive volume is formatted for macOS. Many external drives ship pre-formatted for Windows, so a quick erase in Disk Utility can improve your performance and reliability.

You can't remove specific revisions of an item, but you can delete all revisions of an item.

In your Downloads example, you started off capturing items in your download and then decided to exclude that folder from your archive. But the earlier layers still contain the item originally captured when that folder wasn't excluded.

Open the archive, rewind until you find a captured instance of the Downloads folder (or turn on Show Deleted Items). Select the folder (or its contents) and choose Archive ➤ Delete Items…. QRecall will remove every instance of those items from your archive, as if they had never been captured. And since they are now excluded from the archive, they won't be recaptured.

If you delete a lot of data, follow up with the Archive ➤ Compact command to recover the unused space.

I've considered making a variant of the delete items command that would act on a range of layers, but that gets really complicated (due to QRecall's cross-integrity checks and the way it structures directory deltas) and there simply hasn't been enough demand.

Mike M wrote:Is that correct?

Basically, yes.

The need to keep different histories for your system and user documents means you need two archives. Whether these archives contain overlapping data is up to you, and your needs.

Create a "System" archive:

Capture your startup volume once a day

Keep the past 14 days

Excludes the contents of your home folder(s) (optional)

Create a "User" archive:

Capture your home folder(s) every two hours or so

Create additional actions to capture important items the moment they change or are safe to capture (optional)

Keep the past week of hourly changes, then keep daily or weekly snapshots going back as far as you desire

Exclude the ~/Downloads (and any other extraneous) folders

If you exclude the contents of your home folder from the "System" archive, then you won't have any duplicate effort. The drawback is that a full system restore will become a two-step process; you'll first have to restore your startup volume (from "System") and then restore the latest user folder (from "User").

James, as an aside, regarding the problem I reported, I am not trying to accomplish that task any more, and rather am experimenting with a new backup drive and setting up some backup along the lines I describe above. So there is no rush. If you discover something deeply broken, however, let me know.

The only real "problem" I saw had to do with your launch services database, which I suggest you reset. Not only will it make QRecall's job easier, but it will avoid random crashes in other apps too.

Looking at the log files that have transferred so far, it would appear that your capture issues are due to a corrupted launch services database in combination with a bug in QRecall.

The launch services database has had a long-standing problem where requests for some metadata (icon, localized display name, and so forth) can cause the requesting application to crash. To protect against this, QRecall launches a second process (called the metadata helper) that gathers file metadata on behalf of the capture action process. The idea is that if the metadata helper crashes, the capture process can keep running.

There was a recovery bug in 2.0.7 where, depending on the timing, the capture process would still terminate abnormally if the metadata helper process crashes. This has been (knock wood) fixed in 2.0.8, which you recently updated to.

I suspect that upgrading to 2.0.8 is what allowed your capture action to continue running. However, your launch service database is still corrupted and the metadata helper is still crashing. While this is no longer fatal, it's better for all of your apps, not just QRecall, if your launch services database isn't causing crashes.

The fix for your Launch Services database is usually just to reset your database. Close all the apps you're using, open the Terminal, and execute this command:

/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -kill -r -domain local -domain system -domain user

When the command finishes (it may take a minute), immediately restart your computer.