Mike M wrote:Getting errors related to "identity key", some "lost process" too

The "identity key" and "lost process" errors are both bugs in macOS. We're hard at work on a new version of QRecall that should address these issues, but looking at your logs it appears that all actions (except the "identity key" ones) all ran and completed successfully.

If you have a question about the success of an action, the definitive resource is the log. If an action logs "<Some Action> complete" then that action ran successfully to completion. Each action runs as a separate process and the the only time it logs that message when it has started, accomplished its task without any major errors, and closed the archive. So if an actions logs "Capture complete", that capture was successful, no matter what else happened in the mean time.

The "identity key" errors are the result of a long standing, and rather annoying, bug in macOS. It has to do deadlocks that can occur when multiple tasks try to access the same preferences file. The Core Foundation framework is supposed to handle this, but starting around OS X 10.9 it sometimes gets stuck and never recovers. Some people have reported that sending a TERM signal to the cfprefsd daemon will sometime fix it, but a restart of your system will almost always resolve it. You encountered these on two days, and after restarting your system the next day the errors went away.

Note: I have six archives total : 3 archives on Disk 1 and 3 similar archives on Disk 2. I rotate Disk 1 and Disk 2 every week to an offsite storage area. This means that every week, a large number of actions are marked "waiting for archive" ... for instance while Disk 1 is offsite, every action for Disk 1 is marked "waiting for archive," and the next week same with Disk 2.

I do not know if this "offsite rotation scheme" is causing problems.

They aren't causing problems, but you'd probably be better off with a slightly different configuration of actions.

All of your actions have a "hold if no archive" condition. I assume this was left over from using the capture assistant to create them.

The "hold if no archive" is really intended for situations where you are capturing to a single external drive which is not always connected—a common case for external drives used with laptops. This condition starts the action but then places it on "hold" until the drive is mounted; it acts as a prompt that QRecall is waiting for the archive media and you should plug it in now.

Your situation is a little different. You have a set of actions that are expected not to run when their external drive isn't connected. I would suggest removing the "hold if no archive" condition and and replacing it with an "ignore if no archive" condition. This condition simply skips an action that targets an archive which is currently off-line.

Mike,

Thanks for the diagnostic report.

It's a bug.

The action terminated with an uncaught exception when trying to obtain exclusive access to the archive.

It's a random thing, unlikely to happen again, and doesn't involve any data loss. The code has been fixed and should appear in the next release.

QRecall is telling you that you (or specifically, your user account) doesn't have permission to create/write a document in the root directory of that volume.

A QRecall archive is much like any document: you must have read, write, and execute permission in the directory you want to create it.

There are a couple of easy ways to accomplish this:

Ignore ownership

If you plan to use the entire volume for backup, it might be easier to simply turn off permissions and ownership for that volume. Select the volume in the Finder, choose Get Info, and check the "Ignore ownership" option. Once you do this, macOS will let you read and write anything on the volume.

Fix the permissions

Change the ownerships and/or permissions of the directory you want to use. If you can't change the root directory, create a sub folder and make yourself its owner. Note that any other computers/users that require access the same archive will also need appropriate permissions. (This is why the "ignore ownership" option is the simplest.)

Final note: while QRecall makes every effort to support all major filesystems for hosting archives, it's happiest (and most efficient) when your archive volume is formatted for macOS. Many external drives ship pre-formatted for Windows, so a quick erase in Disk Utility can improve your performance and reliability.

You can't remove specific revisions of an item, but you can delete all revisions of an item.

In your Downloads example, you started off capturing items in your download and then decided to exclude that folder from your archive. But the earlier layers still contain the item originally captured when that folder wasn't excluded.

Open the archive, rewind until you find a captured instance of the Downloads folder (or turn on Show Deleted Items). Select the folder (or its contents) and choose Archive ➤ Delete Items…. QRecall will remove every instance of those items from your archive, as if they had never been captured. And since they are now excluded from the archive, they won't be recaptured.

If you delete a lot of data, follow up with the Archive ➤ Compact command to recover the unused space.

I've considered making a variant of the delete items command that would act on a range of layers, but that gets really complicated (due to QRecall's cross-integrity checks and the way it structures directory deltas) and there simply hasn't been enough demand.

Mike M wrote:Is that correct?

Basically, yes.

The need to keep different histories for your system and user documents means you need two archives. Whether these archives contain overlapping data is up to you, and your needs.

Create a "System" archive:

Capture your startup volume once a day

Keep the past 14 days

Excludes the contents of your home folder(s) (optional)

Create a "User" archive:

Capture your home folder(s) every two hours or so

Create additional actions to capture important items the moment they change or are safe to capture (optional)

Keep the past week of hourly changes, then keep daily or weekly snapshots going back as far as you desire

Exclude the ~/Downloads (and any other extraneous) folders

If you exclude the contents of your home folder from the "System" archive, then you won't have any duplicate effort. The drawback is that a full system restore will become a two-step process; you'll first have to restore your startup volume (from "System") and then restore the latest user folder (from "User").

James, as an aside, regarding the problem I reported, I am not trying to accomplish that task any more, and rather am experimenting with a new backup drive and setting up some backup along the lines I describe above. So there is no rush. If you discover something deeply broken, however, let me know.

The only real "problem" I saw had to do with your launch services database, which I suggest you reset. Not only will it make QRecall's job easier, but it will avoid random crashes in other apps too.

Looking at the log files that have transferred so far, it would appear that your capture issues are due to a corrupted launch services database in combination with a bug in QRecall.

The launch services database has had a long-standing problem where requests for some metadata (icon, localized display name, and so forth) can cause the requesting application to crash. To protect against this, QRecall launches a second process (called the metadata helper) that gathers file metadata on behalf of the capture action process. The idea is that if the metadata helper crashes, the capture process can keep running.

There was a recovery bug in 2.0.7 where, depending on the timing, the capture process would still terminate abnormally if the metadata helper process crashes. This has been (knock wood) fixed in 2.0.8, which you recently updated to.

I suspect that upgrading to 2.0.8 is what allowed your capture action to continue running. However, your launch service database is still corrupted and the metadata helper is still crashing. While this is no longer fatal, it's better for all of your apps, not just QRecall, if your launch services database isn't causing crashes.

The fix for your Launch Services database is usually just to reset your database. Close all the apps you're using, open the Terminal, and execute this command:

/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -kill -r -domain local -domain system -domain user

When the command finishes (it may take a minute), immediately restart your computer.

Mike,

Thanks for bringing this to our attention. We definitely have not received any recent diagnostic reports from you, so there could be something wrong there too.

Let's start by getting a copy of some of your log files and see if we can't figure out what's going on, or why your diagnostic reports aren't getting through.

I'm sending you a DropBox upload request. Use it to upload your recent log files (in your <home>/Library/Logs/QRecall folder) and any QRecall related .crash files you find in <home>/Library/Logs/DiagnosticReports or /Library/Logs/DiagnosticReports.

Tomas,

Good question!

In QRecall 2.0, the exclusion list has moved to archive settings or you can exclude items individually from all archives. You no longer have to keep the exclusion list for all of your capture actions in synchronization. Any excluded items from 1.x actions should have been automatically migrated to your per-archive settings when you upgraded to 2.0.

To exclude items from one archive, open the archive and choose Archive ➤ Settings…. Add the excluded items to the list.

To exclude an item from all archives, choose the item in the Finder, right/control+click and choose Services ➤ QRecall Capture Preferences…. (If this doesn't appear in the menu, you many need to enable QRecall services. See the "No QRecall Services?" sidebar in the Desktop Integration help section.) In the Capture Preferences window, choose one of the Exclude options.

You can also set per-item preferences using the command-line tool.

The easiest place to start would be for you to send a diagnostic report (in the QRecall app, choose Help ➤ Send Report…), which will include the specifics about those errors.

Byron,

Thanks for your interest in QRecall.

The ability to filter files during a capture is an oft requested feature, and it's already pretty high on the "to do" list. It is, unfortunately, held up for a couple of practical reasons (there are other features higher on the "to do" list) and for technical reasons, but I anticipate this feature will appear in QRecall 2.2.

QRecall 2.0 does have the ability to exclude individual files and folders using capture preferences. These can be set in the Finder (using the Capture Preferences service) or from the command line. Using the command line tool, you can kind-of-sort-of-almost get the effect you're looking for by setting the "exclude" preference en masse.

For example, if you want to exclude all of your *.class files you could use something like this:

find ~/Development -type f -name "*.class" -print0 | xargs -0 qrecall captureprefs exclude

The limitation with the exclude preference is that it is attached to the file. If your compiler deletes a .class file and then creates a new one, the exclude preference gets discarded along with the original file.

For situations like that, a more persistent setting would be to exclude the contents of the folders containing those .class files, either surgically or doing something crass like this:

find ~/Development -type d -name "[Cc]lasses" -print0 | xargs -0 qrecall captureprefs exclude contents

Note the exclude contents variant; this tells QRecall to capture the folder (and its metadata), but exclude all of the items contained in that folder.

I hope that helps, for now. Keep an eye on this space for a beta of the next version of QRecall, which may or many not have the features you're looking for.

Mike M wrote:Thanks, working on it now. The repair has been going on for an hour and has "examined" about 1/4 of my data. I didn't realize it would take so long; I need to shut down my computer in a couple hours.

You can stop the repair, but it will have to start over again. The repair takes so long because it has to read and verify all of the data in your archive, then reassemble the various tables and indexes it uses to manipulate it.

This is the second time my archive has needed repair. Any advice about how to minimize these occurrences? It is possible that my hard drive was unplugged without ejecting it--maybe that is what damaged the archive. That is only speculation, however. I don't recall seeing any messages to that effect from OS X.

QRecall should be able to recover from most abnormal interruptions; the notable exceptions are the compact and repair commands. So just unplugging a device won't necessary corrupt your archive, unless the timing is just right, but should still be avoided for obvious reasons.

On the other hand, stuff happens. That's why we take backups and why QRecall is so fastidious about monitoring the health and integrity of the archive data.

Mike,

I would appear that the archive was damaged (no idea how) and that your redundant data files are now out of sync with your archive's primary data file.

This can be a problem for the repair. The redundant data is designed to correct with small amounts of data corruption—a byte here, a 4K block there. It cannot deal with massive amounts of bad data, and when the redundancy files get out of sync it makes it appear as though all of your data is bad.

This can usually be remedied by discarding the redundant data and repairing your archive. I have plans to add this option to the Repair command, but for now you'll need to do it manually:

1) Select the archive in the Finder. Right/Control+click on the icon and choose "Show Package Contents".

2) Locate the repository.data file.

3) Locate any "companion" file; these will start with "repository_" followed by a series of letters/numbers, and end with extensions like .checksum32, .anvin_reed_sol, .reed_sol, and so on. (If there aren't any, then redundancy isn't enabled and you can skip the next step.) Here's an example:

repository.data             <---- keep this one



repository_8k.checksum32    <---- trash all of these

repository_p8w8k16m2.0.anvin_reed_sol

repository_p8w8k16m2.0_8k.checksum32

repository_p8w8k16m2.1.anvin_reed_sol

repository_p8w8k16m2.1_8k.checksum32

4) Select all of the companion files and trash them. DO NOT TRASH the repository.data file.

5) Back in QRecall, repair the archive.

6) Once repaired, turn redundancy back on using Archive > Settings…

Mike,

Send a diagnostic report. (Choose Help > Send Report, in the QRecall application)

It's possible that the process is still running, but since it happened multiple times I suspect the repair process is crashing. The diagnostic report should shed some light on what's going on.

Paul Mann wrote:Will the database structures still be compatible, so users on 10.7 can still use the same db, using 2.0.x, or will that be an upgraded format.

Currently, yes. I haven't added anything to 2.1 that modifies the data structure of the archive in a way that earlier versions of 2.0.x can't continue to use them. Most of the changes to 2.1 are on the client side (interprocess communications, background services, UI, command-line tool, and so on) which doesn't affect the data.

Is the calendar pane the vertical light blue area on the main screen of the app at the moment?

Yes, the new calendar/timeline view replaced the old timeline view on the left. The new view can be shrunk down to a thin linear timeline spanning a year or more, or stretched out (as shown above) to zoom into detail down to the minute.

Paul,

The UI changes are half redesign and half technology refresh. I've raised the macOS requirement for QRecall 2.1 from 10.7 to 10.9. This allows me to take advantage of a lot of UI goodness that was added to OS X 10.9, like stacked views and advanced auto-layout constraints. The result is a much nicer, cleaner, consistent and far more flexible interface.

Here's a sneak peek at the new calendar pane:



And the revamped action document: