David Ramsey wrote:The proximate problem appears to be that the "iconservicesagent" process is hogging a tremendous amount of CPU time, up past 100%.

This is the culprit. Apple seems to have consolidated icon lookup/management/generation into this new background service. The bad news is that this service is really, really, slow—at least at the onset.

I don't know if it's the 10.13 upgrade, the APFS conversion, or some combination of both, but after installing High Sierra this process was driving me nuts. QRecall captures were glacially slow, with iconservicesagent maxed at 100% CPU while everything else crawled along. I have a folder of about 1,200 Photoshop files, and it was excruciating to open. In icon view, I'd scroll a page and the Finder would freeze for 30 seconds, scroll down a little more, and another 30 second freeze, and so on.

I suspect that the icon information gets cached somewhere, because over time things have improved dramatically. Or possibly some of the performance problems have been addressed in 10.13.1 and 10.13.2. Anyway, about a month ago I stopped noticing iconservicesagent being a problem. I can still catch it on the top list, but it's now unusual, rather than the norm. My 1,200 Photoshop files scroll about as fast I can page down and captures are back to their regular speed.

So with any luck, this might eventually take care of itself. Remember that all captures are incremental. Just let the capture run, maybe overnight, and stop if it hasn't finished. It will pick up again where it left off when you start it again. Once you're caught up, merge the incomplete layers with the final one.

Hello Matthias,

This could be any number of problems. Often it's just an installation issue.

First, you might simply try reinstalling QRecall. Open the QRecall application. While holding down the Option and Shift key, choose QRecall > Quit and Uninstall. Restart your system, launch QRecall again, and re-authorize it.

Whether this fixes the problem or not, please send a diagnostic report, so that we can look at the details. Open QRecall and choose Help > Send Report…. That will upload the details of what is, or was, going wrong.

Ralph Strauch wrote:Once Qrecall opens an archive, the only way to unmount that drive (without just pulling the plug) is to shut down the computer. Eject won't work, it just produces a notice that something else is using the file. This persists even when Qrecall and all other running apps have been quit, and even when I've logged out of my account and try to eject the drive from a different account. The only way to disconnect the drive seems to be Shutdown. The problem does not occur with files opened by other apps.

If you believe the problem preventing your volume from unmounting is an open file or files, there's an obscure command line tool that might help track the culprit.

The lsof (list open files) tool will list all of then open file descriptors in your system. Use the +D option to filter the list to only those items contained within a specific directory/volume, like this:

sudo lsof +D /Volumes/Stuck

will list the files/directories that are open on the volume Stuck. More importantly, the list includes the name of the process that opened the file and the user account it's running under.

To answer your question, QRecall can be uninstalled casually by holding down the Option and Shift keys and choosing QRecall > Quit and Uninstall command. For a thorough uninstall, see Help > QRecall Help > Guide > Advanced > Uninstall > Uninstall (the hard way).

However, I don't think uninstalling QRecall is going to fix anything.

Looking at the diagnostic report you sent, the QRecall app is hanging when it tries to open the archive file(s). Specifically, the call to open the file hangs deep in the filesystem. You can see this in the .hang file where QRecall calls open(), which eventually get stuck in check_file_access() > lck_mtx_sleep_deadline() > thread_block_reason().

Also, several actions are getting stuck obtaining write access to the archive files. Finally, one action crashed trying to close a mapped file in the archive.

All of these symptoms point to a problem with the drive or server that contains the archive. If it's a server or NAS, I'd first suggest restarting it; the server might have stale file semaphores that are causing problems. It might be connectivity problem. It also might indicate hard drive issues. It's hard to tease out a culprit from the logs, but that's the direction I'd look in.

Sorry about that. QRecall is (currently) a bit literal about UIDs and GIDs.

(An earlier version used to capture and lookup the UIDs based on the current account info, but there were other problems with that. I'll make a note to try and resurrect that option in a future version.)

The workaround is to just recall all of the files belonging to sjg (which will belong to 503), then simply change the ownership of all of your files back to 501 (the new and improved sjg). Here's how:

I would first create an second admin account for this. It will be *much* easier to do if you're not logged in as sjg at the same time. Log out of sjg and log into the new admin account. (I keep an admin account named "Maintenance" around just for silliness like this.)

Alternative: If you've already recalled these file and have them sitting somewhere, skip the new account setup and jump to the terminal stuff.

Use QRecall to restore your sjg home folder. This will overwrite your new sjg home folder with the captured version. At this point sjg is useless as all of the files belong to 503 and sjg is 501, so you can't even log it at this point.

Open a Terminal window and issue this command:

sudo -s

The terminal will prompt you for the admin account's password. After sudo -s returns, you'll be running a shell with root privileges. Now run these commands:

find -x /Users/sjg -user 503 -print0 | xargs -0 chown 501

find -x /Users/sjg -group 503 -print0 | xargs -0 chown :501

These command will find every file and directory in /Users/sjg and pass it to the chown command, which will change the ownership of the file/dir to 501. The second command does the same with the group ownership. These may take awhile to run, depending on how many files you have.

When these commands finish, everything owned by 503 in /Users/sjg will now be owned by 501. Issue the exit command in the terminal, log out, and back into your home account.

Note: you might have other files, outside of your home folder, that still belong to 503. You'll just have to deal with those on an ad hoc basis, but these commands can be reused with any directory, just change the path of the find command to whatever directory you want to fix.

Boy, I can't slip anything past you guys.

Yes, as mentioned in the release notes for 2.0.9, the certificate used to digitally sign the application and distribution disk image file have changed and are now correct. The previous version of QRecall was accidentally signed with my Mac Developer signature?a signature used to sign apps distributed via the App Store?instead of my Developer ID.

Mike,

Corruption during a single copy is highly unlikely. My advice would be to simply copy the archive(s) from A to C and then verify the archive(s) on C. Verify checks the validity of every byte of data in the archive. If any data was corrupted during the copy, verify will discover it.

I would also suggest suspending your A1 action schedule (or suspend all schedules) for the duration of the transfer.

Once the archives are copied, open any A action and change its archive location to the new archive on the C drive. Save the action. QRecall will ask if you want to relocate all other actions that use the same archive to the new location; choose "yes." Repeat for A2 and A3.

Unmount A and then un-suspend your action schedules.

All really good questions. I wish I had more answers.

From what little testing I've been able to do[1], here's what I surmise.

If the Time Machine (or QRecall) backup is macOS 10.12, then even if you use the recovery partition you'll end up restoring a macOS 10.12 installation, which must be on an HFS+ volume?10.12 can't read APFS volumes, so it could never boot.

So the only way to directly restore an APFS volume with macOS 10.13 is to first install 10.13, make a backup, then use the recovery partition to restore it.

What I suggested earlier should work, either (a) restore from a 10.12 system backup and then run the 10.13 installer over it or (b) install a virgin copy of 10.13, then use your 10.12 backup to recover just your applications and user documents.

[1] I've had very little luck creating an APFS volume on a mechanical HD for testing. I've made five attempts, and the only thing I've managed to do so far is render two hard drives, partitioned with different versions of the OS for testing, unreadable. I'd have had to completely wipe and reparation the drives twice now.

High Sierra seems to work just fine on HFS+, but I'm beginning to side with Apple that APFS on HHDs isn't ready for prime time.

Steven,

As far as I've been able to tell, you shouldn't have any problems restoring your apps and user documents.

Here's what I know so far:

APFS doesn't introduce any new filesystem types so everything that was supported before (the various file types, file metadata, extended attributes, compressed files, access control lists, and so on) remain the same. So you shouldn't have any problems using QRecall to capture, and later restore, any of your user-land files.

The same is probably true for the system files, but this needs more testing.

What I do know is that, as of today, Apple has made it clear that Apple's macOS installer is the only tool that can create a bootable APFS volume. In other words, you can format a new volume using APFS, you can convert an existing volume from HFS+ to APFS, and you can copy all of the system files to it, but you can't bless it and it will never boot.

Apple has indicated they intend to correct this limitation. But until they do, the only way to restore a High Sierra startup volume is to format it using APFS, use QRecall (or any other utility) to copy all of the files back on it, and then run the macOS installer (either from a second volume or using the remote installer) to install the OS files over the files you recovered.

Good question.

I've been so consumed with trying to get the next version of QRecall working, I haven't had a chance to run comprehensive tests on QRecall and macOS 10.13.

On the other hand, QRecall usage statistics collected by the server show at least 40+ customers running QRecall 2.0.8 on the beta of macOS 10.13 and I've yet to receive a single bug report, so I'm continuing under the assumption that it's not a complete disaster.

As soon as I've had a chance to do some tests, I'll update this thread.

Postscript

One more thing: if you really want to see what the effect of using two identity keys are, get two trail keys from the website. Use them to create individual archives for each computer, then create a single archive and capture both computer to that. It may take a couple of days to test these out (four full, first-time, captures over a networked storage device), but it will definitively show how much storage savings you'll get.

Tom,

It's great that you're looking at QRecall for your data integrity needs.

The biggest advantage of capturing multiple computers to a single archive is storage efficiency. Any common data you both back up (example: you both keep a copy of the same 100 GB of music/movies on your computers) will be consolidated into a single copy in the archive. A minor advantage is that regular maintenance can be relegated to one computer, usually the one with the fastest connection.

The downside to sharing a single archive is that only one computer can update the archive at a time, so regular capture operations have to be performed serially which might not be convenient. And as archive size grows, the speed of operations decreases as the complexity of duplicate data analysis increases exponentially with size. So a 2TB archive will be slower to use than two 1.2TB archives.

Honestly, since you're only talking 2 computers and I doubt you have huge amounts of duplicate data between you, I would suggest starting with two archives and seeing how it goes. If you start to bump up against the storage limits of your device you can revisit this arrangement.

Nothing much as changed on this front.

I've contemplated what it would take to transfer items from one archive to another. While I can see some appeal, the complexities (and code) are significant.

I'm not saying this will never happen, but there's still a lot of features on the to-do list with better benefit/cost ratios.

John,

Each action is stored in a .qraction document inside the ~/Library/Preferences/QRecall/Actions folder. Simply copy these from your old system (or restore them from an archive).

If you alter the contents of this folder manually, you'll need to restart the QRecallScheduler process if you want QRecall to see the changes you made immediately. A system restart is the simplest solution, but you can also Quit the scheduler using Activity Monitor / Terminal and then quit and relaunch the QRecall application.

Note: each QRecall action identifies its archive using a bookmark (the modern equivalent of a Finder alias). Moving the archive or the bookmark to a new system might break the bookmark. If that happens, update the bookmark by opening each action and reselecting its archive.

First, try simply restarting your system.

There's a (now) long standing bug in macOS where an updated preference settings belonging to one process (the QRecall application) can't be read by another process (the QRecallHelper tool).

If that's the problem, a simple restart will usually break the deadlock.