Trying the 2.0 beta, and I find the new exclusion rules to be less useful than their 1.x counterparts.

- First, despite rebooting, copying QRecall.app around, etc. I can't get LaunchServices to see the new "QRecall Capture Preferences" service. I do have "Capture to QRecall Archive", etc.

- While I understand the value of exclusion filesystem attributes for items that may move, I'm actually much more concerned about never capturing items at particular paths. These are folders that may be deleted and recreated, and thus attributes will be lost. The good news here is that per-archive exclusions may work, except...

- The loss of exclusions per capture mean that one cannot set independent schedules for an item that is a subitem of a larger capture. For example, I backup my boot drive frequently, but exclude a subdirectory of VM images. However, previously the VM images could be captured on a separate event by a capture with a different exclusion set. Note that the "Ignore changes" option for capture preferences is not a good replacement, as it forces me to only ignore on a schedule whose timeline is opaque, I can't force a capture on an event (say, when VMWare quits).

Finally a couple of UI issues:

- The exclusions scroll view in Archive settings is small, and cannot be resized. For exclusions inside ~/Library or long lists it's a frustrating interface to verify that all the items you want are present. This was not an issue with the prior interface in the capture action.

- If one does choose to use capture preferences as extended attributes, there's no simple UI to actually see what's excluded. Again this is necessary to verify you haven't missed something that should be excluded.


Some of these issues can be addressed with UI fixes, but its not clear to me that the new exclusion model actually supports the same set of features as the prior design.

Thanks,

Alex

Alex wrote:- First, despite rebooting, copying QRecall.app around, etc. I can't get LaunchServices to see the new "QRecall Capture Preferences" service. I do have "Capture to QRecall Archive", etc.

You might have to explicitly enable these new services in Finder > Services > Services Preferences…. If they're not listed there, you might have a "stuck" copy of the old services in ~/Library/Services/. Delete the QRecallService.service package from this directory and launch the QRecall app again. QRecall will reinstall the new service and notify OS X that it is available. Afterwards, it should appear in your Services Preferences.

- While I understand the value of exclusion filesystem attributes for items that may move, I'm actually much more concerned about never capturing items at particular paths. These are folders that may be deleted and recreated, and thus attributes will be lost. The good news here is that per-archive exclusions may work, except...

- The loss of exclusions per capture mean that one cannot set independent schedules for an item that is a subitem of a larger capture. For example, I backup my boot drive frequently, but exclude a subdirectory of VM images. However, previously the VM images could be captured on a separate event by a capture with a different exclusion set.

This is exactly the "mistake" that the new exclusion arrangement is designed to prevent you from making. Here's the problem with the way it works in QRecall 1.2:

You capture your home folder every hour, but exclude you VM folder.

Every evening, you capture your home folder but do not exclude your VM folder.

The problem with this arrangement is that your VM folder alternately appears, and then disappears, from your archive. After the first capture your VM folder would not be captured. After the second, it would. If your repeat the first capture again it's gone again. If your hard drive crashes and you restore from the latest capture, your VM folder would not be restored—because, as far as QRecall is concerned, it didn't exist when the third layer was captured.

The second problem this creates is that your VM folder get repeated captured in its entirely, rather than intelligently capturing just the changes. When the second capture runs, the VM folder doesn't exist in the archive history, so all of the files are captured as if they were brand new. After the third capture (that excludes the VM folder), the VM folder is erased in that layer. The next capture of the VM folder starts over from scratch because7mdash;again, from QRecall's perspective—all of the files are new.

What you really want to do is use the Ignore Changes feature. Set your VM folder to ignore changes for 20 hours or so. During the day, your startup volume or home folder can be quickly captured without trying to recapture your VM folder.

You can then create a second capture action to capture just your VM folder at the end of the day, or trigger that to run when your VM application quits.

Tip: I'd also suggest setting the "Deep Scan" option for the VM folder capture action, as VM packages (particularly Parallels) are notorious for tricking the filesystem change history into not seeing all of the changes.

Note that the "Ignore changes" option for capture preferences is not a good replacement, as it forces me to only ignore on a schedule whose timeline is opaque, I can't force a capture on an event (say, when VMWare quits).

Good point.

That's exactly why there's an exception to the rule: an ignore preference set on a folder is ignored when that folder is explicitly a target of a capture action. In other words, the per-item ignore preference only applies to items contained in an item being captured, never to the items listed in the capture action.

Finally a couple of UI issues:

- The exclusions scroll view in Archive settings is small, and cannot be resized. For exclusions inside ~/Library or long lists it's a frustrating interface to verify that all the items you want are present. This was not an issue with the prior interface in the capture action.

I've added this to the to-do list for QRecall 2.1, which will be a major UI facelift.

- If one does choose to use capture preferences as extended attributes, there's no simple UI to actually see what's excluded. Again this is necessary to verify you haven't missed something that should be excluded.

You can do this with the command-line tool. For example, this command will list all of the items contained in your home folder that have capture preferences set:

qrecall captureprefs list --recursive --skipnormal ~

Ooops,

I wrote too soon. I just tested this out, to make sure it worked, and apparently it doesn't.

If you capture a folder, and that folder is set to ignore changes, the changes may, or may not, be ignored (it's a bug and it depends on where the folder is in the hierarchy).

I will definitely make sure this is working before the next version is released.

Fixed.

And I'm so glad you asked.

QRecall will reinstall the new service and notify OS X that it is available. Afterwards, it should appear in your Services Preferences.

Thanks, that worked...

This is exactly the "mistake" that the new exclusion arrangement is designed to prevent you from making. Here's the problem with the way it works in QRecall 1.2:

<snip>

I understand your explanation, but I think its pretty surprising. Why would exclusion translate to a deletion in the archive? I would understand if exclusion translated to removal of all previous versions (CrashPlan works this way, with appropriate warnings). Presumably deletion is a recorded event, and clearly you can carry files unchanged forward through the layers for ignore changes. Can exclusions not be "ignore changes indefinitely"?

Also, could you clarify "captured as if they are brand new"? For unchanged files, the blocks are the same, so aside from the metadata of the files reappearing in the next layer, I presume the impact is small.

That's exactly why there's an exception to the rule: an ignore preference set on a folder is ignored when that folder is explicitly a target of a capture action.

If you capture a folder, and that folder is set to ignore changes, the changes may, or may not, be ignored (it's a bug and it depends on where the folder is in the hierarchy).

I'll try it in the next beta, thanks. You'll definitely want to mention this in docs, I would have assumed it worked according to the rules you mention as a bug.

Alex

Alex wrote:I understand your explanation, but I think its pretty surprising. Why would exclusion translate to a deletion in the archive?

It doesn't get deleted (in all layers), as it would if you selected the item and used the delete command.

What I mean is that it gets recorded as a deleted file in the new layer. Here's an example. You have a folder (A) with two files (X and Y). You capture folder A:

Layer 1: Folder A has two files, X and Y

You now exclude item Y and capture the same folder again.

Layer 2: Folder A has one file, X

As far as QRecall is concerned, file Y was deleted. It existed when layer 1 was captured, and didn't exist when layer 2 was captured.

More importunely, if you recall from layer 2, there will be no file Y. (Replace file Y with your VM folder, and you get the idea.)

If you later include file Y again, file Y get re-captured again. Now this won't add any significant amount of new data to the archive—thanks to data de-duplication—but it defeats QRecall's ability to intelligently capture only the changes. QRecall can't look at that file (or the contents of an entire folder) and determine that nothing has changed and simply skip recapturing it. This adds a significant amount of overhead and also prevents you from determining what changed by examining the individual layers, because it looks like everything changed, every time it's recaptured.

You'll definitely want to mention this in docs, I would have assumed it worked according to the rules you mention as a bug.

Yes, there's a lot that still needs to be documented.

Poll:

Thinking about this issue a little bit, it wouldn't be difficult to make the recapture of items that have the "Ignore" capture preference set into an option, the same way that the new "Deep Scan" option ignores filesystem change history when looking for modified folders.

So instead of having a special exception for items marked as "ignore", applied only to the items specifically being captured, what about a new capture action option ("Capture Ignored Items") that explicitly ignores the ignore preferences, and recaptures all changed items?

Alternatively, I could just bake that feature in to the existing "Deep Scan" option, so a deep scan would also imply capturing ignored items.

Any opinions?

At least in my case, exclusions are important to honor consistently, and not just attempts to optimize archive space. As such I'd be nervous about too much magic in the overrides and would advise against attaching "ignore the exclusions" to anything not clearly labelled as such. "Deep scan", for example, would be a bad choice.

Of the options presented, I actually think the original plan of requiring adding the top level exclusion to the capture and only ignoring that one exclusion is the best. Its the least surprising (when documented) in the sense that it has the fewest side effects. Its explicit and can be as fine grained as the user desires.

Thanks for the feedback.

I tend to agree with you on all counts, but I wanted to throw out the idea to see if an alternative made more sense for some people.

I'll leave the logic the way it is now.

James Bucanek wrote:Tip: I'd also suggest setting the "Deep Scan" option for the VM folder capture action, as VM packages (particularly Parallels) are notorious for tricking the filesystem change history into not seeing all of the changes.

James, I just sent you a QRecall report a short while ago that you can probably now ignore. I have an action set to backup my VM folder after Fusion quits. The action was running, but not finding anything to archive. I suspect your tip will solve my problem. I just turned on the Deep Scan option for that action, but QRecall is in the middle of backing up my entire volume, so I can't test this right now. I'll give it a shot tomorrow and let you know if it worked.

Bruce Giles wrote:James, I just sent you a QRecall report a short while ago that you can probably now ignore. I have an action set to backup my VM folder after Fusion quits. The action was running, but not finding anything to archive. I suspect your tip will solve my problem. I just turned on the Deep Scan option for that action, but QRecall is in the middle of backing up my entire volume, so I can't test this right now. I'll give it a shot tomorrow and let you know if it worked.

I moved this to a new thread in the Beta forum: "Virtual Machines folder not captured". See that one for more information.