Mike,

Thanks for bringing this to our attention. We definitely have not received any recent diagnostic reports from you, so there could be something wrong there too.

Let's start by getting a copy of some of your log files and see if we can't figure out what's going on, or why your diagnostic reports aren't getting through.

I'm sending you a DropBox upload request. Use it to upload your recent log files (in your <home>/Library/Logs/QRecall folder) and any QRecall related .crash files you find in <home>/Library/Logs/DiagnosticReports or /Library/Logs/DiagnosticReports.

Tomas,

Good question!

In QRecall 2.0, the exclusion list has moved to archive settings or you can exclude items individually from all archives. You no longer have to keep the exclusion list for all of your capture actions in synchronization. Any excluded items from 1.x actions should have been automatically migrated to your per-archive settings when you upgraded to 2.0.

To exclude items from one archive, open the archive and choose Archive ➤ Settings…. Add the excluded items to the list.

To exclude an item from all archives, choose the item in the Finder, right/control+click and choose Services ➤ QRecall Capture Preferences…. (If this doesn't appear in the menu, you many need to enable QRecall services. See the "No QRecall Services?" sidebar in the Desktop Integration help section.) In the Capture Preferences window, choose one of the Exclude options.

You can also set per-item preferences using the command-line tool.

The easiest place to start would be for you to send a diagnostic report (in the QRecall app, choose Help ➤ Send Report…), which will include the specifics about those errors.

Byron,

Thanks for your interest in QRecall.

The ability to filter files during a capture is an oft requested feature, and it's already pretty high on the "to do" list. It is, unfortunately, held up for a couple of practical reasons (there are other features higher on the "to do" list) and for technical reasons, but I anticipate this feature will appear in QRecall 2.2.

QRecall 2.0 does have the ability to exclude individual files and folders using capture preferences. These can be set in the Finder (using the Capture Preferences service) or from the command line. Using the command line tool, you can kind-of-sort-of-almost get the effect you're looking for by setting the "exclude" preference en masse.

For example, if you want to exclude all of your *.class files you could use something like this:

find ~/Development -type f -name "*.class" -print0 | xargs -0 qrecall captureprefs exclude

The limitation with the exclude preference is that it is attached to the file. If your compiler deletes a .class file and then creates a new one, the exclude preference gets discarded along with the original file.

For situations like that, a more persistent setting would be to exclude the contents of the folders containing those .class files, either surgically or doing something crass like this:

find ~/Development -type d -name "[Cc]lasses" -print0 | xargs -0 qrecall captureprefs exclude contents

Note the exclude contents variant; this tells QRecall to capture the folder (and its metadata), but exclude all of the items contained in that folder.

I hope that helps, for now. Keep an eye on this space for a beta of the next version of QRecall, which may or many not have the features you're looking for.

Mike M wrote:Thanks, working on it now. The repair has been going on for an hour and has "examined" about 1/4 of my data. I didn't realize it would take so long; I need to shut down my computer in a couple hours.

You can stop the repair, but it will have to start over again. The repair takes so long because it has to read and verify all of the data in your archive, then reassemble the various tables and indexes it uses to manipulate it.

This is the second time my archive has needed repair. Any advice about how to minimize these occurrences? It is possible that my hard drive was unplugged without ejecting it--maybe that is what damaged the archive. That is only speculation, however. I don't recall seeing any messages to that effect from OS X.

QRecall should be able to recover from most abnormal interruptions; the notable exceptions are the compact and repair commands. So just unplugging a device won't necessary corrupt your archive, unless the timing is just right, but should still be avoided for obvious reasons.

On the other hand, stuff happens. That's why we take backups and why QRecall is so fastidious about monitoring the health and integrity of the archive data.

Mike,

I would appear that the archive was damaged (no idea how) and that your redundant data files are now out of sync with your archive's primary data file.

This can be a problem for the repair. The redundant data is designed to correct with small amounts of data corruption—a byte here, a 4K block there. It cannot deal with massive amounts of bad data, and when the redundancy files get out of sync it makes it appear as though all of your data is bad.

This can usually be remedied by discarding the redundant data and repairing your archive. I have plans to add this option to the Repair command, but for now you'll need to do it manually:

1) Select the archive in the Finder. Right/Control+click on the icon and choose "Show Package Contents".

2) Locate the repository.data file.

3) Locate any "companion" file; these will start with "repository_" followed by a series of letters/numbers, and end with extensions like .checksum32, .anvin_reed_sol, .reed_sol, and so on. (If there aren't any, then redundancy isn't enabled and you can skip the next step.) Here's an example:

repository.data             <---- keep this one



repository_8k.checksum32    <---- trash all of these

repository_p8w8k16m2.0.anvin_reed_sol

repository_p8w8k16m2.0_8k.checksum32

repository_p8w8k16m2.1.anvin_reed_sol

repository_p8w8k16m2.1_8k.checksum32

4) Select all of the companion files and trash them. DO NOT TRASH the repository.data file.

5) Back in QRecall, repair the archive.

6) Once repaired, turn redundancy back on using Archive > Settings…

Mike,

Send a diagnostic report. (Choose Help > Send Report, in the QRecall application)

It's possible that the process is still running, but since it happened multiple times I suspect the repair process is crashing. The diagnostic report should shed some light on what's going on.

Paul Mann wrote:Will the database structures still be compatible, so users on 10.7 can still use the same db, using 2.0.x, or will that be an upgraded format.

Currently, yes. I haven't added anything to 2.1 that modifies the data structure of the archive in a way that earlier versions of 2.0.x can't continue to use them. Most of the changes to 2.1 are on the client side (interprocess communications, background services, UI, command-line tool, and so on) which doesn't affect the data.

Is the calendar pane the vertical light blue area on the main screen of the app at the moment?

Yes, the new calendar/timeline view replaced the old timeline view on the left. The new view can be shrunk down to a thin linear timeline spanning a year or more, or stretched out (as shown above) to zoom into detail down to the minute.

Paul,

The UI changes are half redesign and half technology refresh. I've raised the macOS requirement for QRecall 2.1 from 10.7 to 10.9. This allows me to take advantage of a lot of UI goodness that was added to OS X 10.9, like stacked views and advanced auto-layout constraints. The result is a much nicer, cleaner, consistent and far more flexible interface.

Here's a sneak peek at the new calendar pane:



And the revamped action document:

Paul,

Things are actually going pretty well, just not on schedule.

The XPC and memory management changes are all done and appear to be working well. There's a bunch of fixes and some small new features have been added.

The work now is entrenched/mired/submerged in the UI changes, which continue to progress nicely but the scope of which has grown to be much larger than I had originally planned. It's like the pulling the proverbial thread; redesigning one features of the UI opens the possibility of redesigning two others, and so on, until I've found myself reimplementing most of the interface. I think you'll like the changes, it just taking up a lot of time and energy.

Once the UI is settled, I'll probably launch a beta to ferret out any problems with the aforementioned changes while simultaneously adding some of the new remote archive features.

Ralph,

You didn't say which archive you were having problems with, but I'm going to assume (from your description) that it was "5th Backup". Here's what I see in the logs:

A whole bunch of successful captures on Feb 28, followed by a merge of a dozen layers.

On March 2, a capture was started at 11:07. At 11:57 a stop (cancel) request was made by the scheduler. (I assume this was a left over stop condition from setting up the incremental capture.) The capture successfully stopped, wrapped up, and committed the items that had been captured up to that point:

2017-03-02 11:07:41.990 -0800 Capture to 5th backup.quanta

2017-03-02 11:57:41.809 -0800 Received cancel request

2017-03-02 11:57:41.890 -0800 Capture stopped by scheduler

2017-03-02 11:57:47.540 -0800 Captured 12154 items, 2.77 GB (53% duplicate)

2017-03-02 11:58:51.121 -0800 Capture incomplete (50:51)

Now if this is the "accidentally interrupted" backup you were referring to, then you should note that this was not an unsuccessful capture—it just wasn't allowed to finish. From QRecall's perspective, the capture was successful; all captured files were successfully added to the archive, and the archive was properly closed.

The archive is then opened again and someone deletes the folder "Dont backup":

2017-03-02 12:04:33.030 -0800 Delete items in 5th backup.quanta

2017-03-02 12:04:38.465 -0800 Deleting Dont backup

2017-03-02 12:04:38.465 -0800 Ralph Strauch(#XXXX):iMacHD:Users:Shared:Dont backup

You then started a verify of the archive, but stopped it before it could finish.

2017-03-02 12:06:11.227 -0800 Verify 5th backup.quanta

2017-03-02 12:10:40.199 -0800 Action canceled

2017-03-02 12:10:40.389 -0800 No problems were found up to the point that the verify was stopped

2017-03-02 12:10:40.389 -0800 Verify incomplete (04:23)

Now here's where things get a little crazy. The archive is opened again and someone deletes the owner "Ralph Strauch"

2017-03-02 12:11:33.487 -0800 Delete items in 5th backup.quanta

2017-03-02 12:11:37.803 -0800 Deleting Ralph Strauch

Since "Ralph Strauch" is the owner, every volume, folder, and file ever captured by that owner (identity key) was deleted from the archive. If "Ralph Strauch" is the only owner that's been captured to this archive, then everything in the archive would have been deleted.

On the other hand, if other owners (identity keys) have captured items to this archive, then those items should still be in the archive.

but when I open the archive there's nothing it -- the window is completely blank

If "Ralph Strauch" was the only owner, then the archive is now, indeed, empty. A blank browser window is exactly what you'd expect to see.

If you expect other owners to be here then first try to use the navigation bar a the bottom of the browser window to navigate to the top-level of the archive, where you will see the other owners.

If the navigation bar is malfunctioning for some reason, you can reset it: (1) close the archive. (2) in the Finder, select the archive and "show package contents" by right+clicking on the archive. (3) inside the archive, discard the view.plist file. (4) reopen the archive in QRecall. That should reset any assumptions the browser window was making about where the view should be in the archive hierarchy.

while Finder shows the file size as 798gb

When you delete items from an archive, using either the delete or merge commands, the records of those items are erased but the space that data occupied remains. Use the compact action to identify, and reclaim, these empty records.

Finally, the archive was repaired several times. I assume this was an attempt to recover the missing items, but I suspect they had already been deleted (see above) and no amount of repairing can recover them as QRecall always securely erases file and directory records.

Ralph,

Sorry to hear you're having drive problems. Hard drive spontaneously, and inappropriately, unmounting can usually be traced to one of five problems:

Software (operating system)

Bus (computer/interface)

Bus (cable)

Hard Drive Controller

Power

Sometimes, very rarely, this can be an operating system or filesystem plugin issue. For example, OS X 10.8, 10.9, and 10.10 all had various problems with external drives using Firewire; it would occasionally eject volumes when the computer went to sleep. But again, these issues are atypical.

Bus problems are the most common. This can usually be traced to a flaky (USB, Firewire, Thunderbolt, eSATA, Ethernet) controller, the cable, or the physical connection. It might be the controller chip itself or the computer's motherboard.

Drive controller problems are also a possibility, although less common.

Finally, power can be the cause. Most external drive use simple transformers (unlike the switched power supply inside your computer) and are much more sensitive to drops in voltage. The likeliness that this is the problem would depend on how "clean" the power in your neighborhood is.

The typical technique for diagnosing these problems is "divide and conquer." In other words, you try to take each of these possible causes and eliminate or substitute them to identify the weak link.

For example, if your drives are connected via Thunderbolt, try switching to USB. Many external drives these days support multiple interfaces, so switching interface can help identify a bus problem. If you find the drive does not unmount when connected via USB, then you can go back to Thunderbolt and try a different cable.

If it's a motherboard issue, or if you don't have an alternative bus to try, the next step is to connect the drive to a different computer system. For example, you might set aside a weekend and relocate the drive from the desktop to the laptop and perform the captures in the other direction, intensively, for a day or overnight.

(If your external drive is unmounting for both local processes and shared computers via Ethernet, then the problem is probably local. Logically, a drive that unmounts locally will be unmounted for the remote user too.)

If it's a physical hard drive or HD controller issue, then swapping it out for another drive is the simplest test, which is sounds like you already did.

Finally, there's power, which can be difficult to diagnose without specialized equipment. If you have a UPS you can borrow, you might try that.

Additional notes:

QRecall does not control volume mounting or unmounting. The most any application (like QRecall) can do is to make mount and unmount requests to the operating system. It's up the operating system to mount and unmount the volume, and it should never unmount a volume that still has open files. So if a volume that's currently in use is getting unmounted, there is a (physical) problem somewhere.

QRecall does write temporary files during capture and those files won't get deleted until the next action or repair. So you may see an archive that has 90GB of data in it, but open it to find almost nothing. The next action (including verify or even just opening the archive) will invoke an auto-repair that will delete/overwrite these temporary files and recover that disk space. (You can open the archive package in the finder and watch this happen.)

But just to be clear, the space used by QRecall during a capture would never be the cause of a volume being unmounted. It would result in a "disk full" error, and the capture stopping gracefully.

As a workaround, you might consider setting up an incremental capture: create a capture action that runs every hour, but stops after 50 minutes. Every hour the capture will start, picking up from where it left off last time, and capture for 50 minutes. If it hasn't finished, it will then wrap up and commit whatever has been successfully captured so far. The next capture starts where the last one left off, but if it is interrupted only the latest capture will be lost when the archive is auto-repaired. You'll probably end up with a lot of unfinished layers (which can later be merged), but at least you'll have successfully captured your files.

Ralph Strauch wrote:My iMac has started to refuse to do captures, giving me a "Capture requires an identity key" error message.

You are most likely encountering a bug in macOS that occasionally prevents one process (the capture) from accessing the preferences of another app (the QRecall client app). Usually a system restart will fix this.

I've tried re-entering my key but when I do I get a warning that I'm entering a new key and that will create a new owner for all subsequent captures.

You're probably entering a different identity key without realizing it. Here's how to find out.

First, get the serial number of the key you have installed now. Open QRecall > Preferences... and go the Identity Key pane. While holding down the Option key, click the Enter Key... button. When the key entry dialog appears, the "key" field will show the serial number of the currently installed key. Make a note of it before canceling the dialog.

Now log into your www.qaccount.com account using this Magic Account URL. This link will cause your account page to display the serial number of each purchased identity key. Compare those numbers with serial number found in the Identity Key pane.

Following up,

All of your repairs that I've looked at fail because the volume containing the archive spontaneously unmounts during the repair process. Here's an example of a repair that started at 09:49:

2017-01-21 09:49:40.052 -0800 Repair 5th backup.quanta

2017-01-21 09:49:40.052 -0800 archive: /Volumes/volume2/5th backup.quanta

Four hours later the volumes mysteriously, and spontaneously, unmounted. (Note that this doesn't appear to be associated with a sleep event.)

2017-01-21 13:34:42.401 -0800 unmounted volume /Volumes/volume2

2017-01-21 13:34:42.402 -0800 unmounted volume /Volumes/volume1

Not surprisingly, the repair starts encountering problems?about a million of them:

2017-01-21 13:34:41.999 -0800 problem getting volume statfs

2017-01-21 13:34:41.999 -0800 IO exception

2017-01-21 13:34:41.999 -0800 POSIXErr: 2

2017-01-21 13:34:41.999 -0800 Path: /Volumes/volume2/5th backup.quanta/repository.data

2017-01-21 13:34:41.999 -0800 ErrDescription: No such file or directory

This is most often caused by some intermittent hardware problem (failing drive controller, brief power interruption, flakey USB connection, and so on).

QRecall can actually be helpful in diagnosing this. Because the QRecall scheduler watches for volume mount and unmount events, it logs those when they occur. Filter out everything except the scheduler messages in the log, and then look for unmount events that shouldn't be happening. If your a command-line geek, this will also do the trick:

fgrep ' [3.' ~/Library/Logs/QRecall/QRecall.log* | fgrep mount

Ralph,

I think it's the drive, but I need some more information to connect the dots.

The log included in your reports contains thousands of errors like this one:

2017-01-26 04:34:30.152 -0800 IO exception

2017-01-26 04:34:30.152 -0800 POSIXErr: 2

2017-01-26 04:34:30.152 -0800 Path: /Volumes/BUD2/2nd backup.quanta/repository.data

2017-01-26 04:34:30.152 -0800 ErrDescription: No such file or directory

This would indicate that the volume spontaneously unmounted while the repair was in progress. The repair is designed to tolerate I/O errors, and will keep plowing away, trying to read whatever data it can, so this just goes on, and on, and on.

The build in Send Report function only includes the latest log records, so to confirm my suspicions I'd need to more of the log. If you can, compress the files in ~/Library/Logs/QRecall and send them to me. They're likely to be very large, so I'll email you a dropbox upload request seperately.