Ralph Strauch wrote:Related problem: ... From there, when I open the archive and select a file to delete, but when I hit the Delete button I get a window that says "Waiting for Archive," and nothing happens. It's telling me this while the archive is sitting there open in front of me. What's blocking the delete function on the MBP? (report sent)

This has been mentioned in another thread, but it bears repeating.

Short answer: be patient.

Long answer:

QRecall 2.0 uses a new method for arbitrating access to an archive between current actions and multiple users. Inside the archive package is a new set of invisible semaphore files (.lock, .share, and .semaphore). These files allow QRecall to negotiate access to the archive on devices that don't support the shared-read/exclude-write access control required to safely share the files.

One problem is that these files can become "stale" and indicate that the archive is being accessed by another process when it isn't.

The next action to request access to the archive sees the stale lock and you get the "Waiting for archive" message. Behind the scenes, it begins a series of tests to determine if the archive really is being used. These tests take about 10 minutes to determine that no other process is modifying the archive and the lock is stale. Once it's convinced that it's safe to access the archive, it will "break" the lock and acquire the access it needs.

So, just let it "wait". It will eventually determine that it can safely access the archive and the action will proceed.

Ralph Strauch wrote:I back up both a MBP and an iMac to the same archive, and the paths to excluded items in my archives seem to depend on which computer I'm looking at them with rather than on which drive the files are actually on.

That's expected. QRecall 2.0 uses URL Bookmarks to identify and remember items in places like the archive preferences and actions. Bookmarks work really hard to identify an individual file or folder. Thus, when viewed on a different system, the bookmark will resolve to the same item, but that item might have a different filesystem path.

I guess this may work out OK in practice because the files that need to be excluded are the ones on the computer doing the backup and those paths are correct, so the paths to nonexistent files that are really on the other computer just get ignored. It can be confusing, though, to look at the exclusions list and see a bunch of nonexistent files.

It kind of has to work this way, else you run into the inability to exclude a folder on one system and not exclude it on another.

In a nutshell, the best way to exclude items using the archive settings on multiple systems it to open the archive on the first system and add the items to be excluded for that system. Close the archive and repeat with the next system.

Alternatively, you can use the new QRecall Capture Preferences to exclude each item individually.

The startup filesystem needs headroom for things like swap and temporary files. If you completely run out of space on the VM swap volume, you are in serious doo-doo...

But even non-boot filesystems start to hurt as their free space dwindles towards zero. The problem is the volume's allocation map. As the drive fills up, it's harder and harder of the filesystem to find free blocks, and those blocks become increasingly smaller and more fragmented. Eventually, the filesystem slows to a crawl trying to find and manage the free space. You can improve this by defragmenting the drive, but when it's nearly full even that becomes problematic.

Because of this, and other reasons, QRecall avoids letting the archive volume fill up completely. The capture action monitors the free disk space and will stop capturing when the volume exceeds approximately 98% full. (The actual amount is variable, based on the size of the volume.)

Filling up a volume, completely, is a nasty situation in OS X (a.k.a. BSD Unix). Because of this, QRecall reserves a 4% buffer when estimating the amount of free space on your drive.

So the volume with your 1.23 TB archive has 197 GB of free space. Assuming the archive is just about the only thing on the volume, that works out to about 3% free space. As far as QRecall is concerned, your drive is already full.

To add 1:8 redundancy to at 1.23 TB archive, QRecall will want to see 154 GB (1/8th of 1.23 TB) + 4% of the volume size (~50 GB), or at least 204 GB of free space before it will proceed.

Volumes "A" & "B" are probably in a different owner.

A single QRecall archive can capture data from multiple owners/systems/identities. These owners are at the top-level of the archive. After your first capture, the browser defaulted to the iMac owner, which has captured volumes "C" & "D".

After capturing volumes "A" & "B", your archive's browser view is still looking at the volumes captured for the iMac. Use the navigation bar at the bottom of the window and navigate to the root (archive) level of the browser tree. There you will see two owners, "iMac" and "Laptop" (or, specifically, the owner names you gave them when you entered your identity keys).

Drill into "Laptop" and you'll see the two new volumes.

If the owner names are too similar, you can rename them in QRecall > Preferences > Identity Key (for that system). The new names will appear in your archive (immediately for you local identity, or after the next capture for the other).

Ralph Strauch wrote:I've been getting this "system clock. . ." message repeatedly for the past three days (since starting to use v2 on this computer).

This is normal. The 2.0 scheduler now monitors the system clock for changes. Whenever it is notified that the system clock has been adjusted, the scheduler reevaluates the scheduled actions. This was to solve a problem of users who change their clock to something way in the future, and then set it back. The result would be actions that wouldn't start again for months, even years, because they were waiting for some future date that was now way in the future.

It turns out that OS X adjusts the system clock all of the time, mostly millisecond changes to keep the clock in sync with NTP. Every time this happens, the scheduler dutifully reevaluates its schedule.

For the most part, these messages can be ignored. It's just the scheduler doing its job. There's a reason these log messages are classified as "minutia." (I might even demote them to "debug" messages, as it turns out they are only rarely interesting.)

Other strange things also seem to be happening. A few files have been "recaptured" for some reason,

Determine why an item was recaptured is a really complicate topic, as there are lots of (sometimes obscure) reasons why an item would be recaptured.

The only definitive way of determining why requires that you first set the QRLogCaptureDecision (Log Capture Decisions) setting to true before the next capture. In 2.0, you can do that in the advanced preferences. When this is set, the capture action will log the reason it captured each item. Warning: this setting can result in a tremendous number of log records. I do not recommend leaving it on.

and yesterday afternoon the app ran a verify action that seemed to start all by itself.

That's a new one. The log should indicate what started that action. I'll take a look at your report.

One capture died after a "broke stale lock file" entry.

We were just discussion that in this thread.

Poll:

Thinking about this issue a little bit, it wouldn't be difficult to make the recapture of items that have the "Ignore" capture preference set into an option, the same way that the new "Deep Scan" option ignores filesystem change history when looking for modified folders.

So instead of having a special exception for items marked as "ignore", applied only to the items specifically being captured, what about a new capture action option ("Capture Ignored Items") that explicitly ignores the ignore preferences, and recaptures all changed items?

Alternatively, I could just bake that feature in to the existing "Deep Scan" option, so a deep scan would also imply capturing ignored items.

Any opinions?

Alex wrote:I understand your explanation, but I think its pretty surprising. Why would exclusion translate to a deletion in the archive?

It doesn't get deleted (in all layers), as it would if you selected the item and used the delete command.

What I mean is that it gets recorded as a deleted file in the new layer. Here's an example. You have a folder (A) with two files (X and Y). You capture folder A:

Layer 1: Folder A has two files, X and Y

You now exclude item Y and capture the same folder again.

Layer 2: Folder A has one file, X

As far as QRecall is concerned, file Y was deleted. It existed when layer 1 was captured, and didn't exist when layer 2 was captured.

More importunely, if you recall from layer 2, there will be no file Y. (Replace file Y with your VM folder, and you get the idea.)

If you later include file Y again, file Y get re-captured again. Now this won't add any significant amount of new data to the archive—thanks to data de-duplication—but it defeats QRecall's ability to intelligently capture only the changes. QRecall can't look at that file (or the contents of an entire folder) and determine that nothing has changed and simply skip recapturing it. This adds a significant amount of overhead and also prevents you from determining what changed by examining the individual layers, because it looks like everything changed, every time it's recaptured.

You'll definitely want to mention this in docs, I would have assumed it worked according to the rules you mention as a bug.

Yes, there's a lot that still needs to be documented.

Ralph Strauch wrote:Is it safe to make another try at adding error correction now, or should I wait for the next release?

As far as I can tell, your issues with adding error correction was in the QRecall app (the monitor), not the action (the doer). It appears that the action continued to run (until the computer was restarted), as I can find no crash logs or other evidence that it died abnormally.

I'd suggest you give it another run. Just remember that it has to read the entirety of the archive data, so it will take about as long as a verify.

Ralph Strauch wrote:I threw away the files you suggested and repaired the archive, and everything seems fine. I'm doing another backup now. The repair log listed about 3 dozen instances of "invalid data," mostly just a few hundred bytes but one chunk of half a megabyte. Should I be concerned about these at all.

What you really want to look for are indications of incomplete folders or files that were a result of those stretches of "invalid data". These will also be in the log. If you don't see any messages like that, then the stretches of invalid data were probably detritus from an incomplete merge or deleted items (as a result of an earlier merge) that hadn't yet been erased.

Qrecall continues to be a great app, and one of it's finest features is the quality and responsiveness of the support that comes with it. Thanks again, James, for all you provide.

You're very welcome.

Fixed.

And I'm so glad you asked.

Ooops,

I wrote too soon. I just tested this out, to make sure it worked, and apparently it doesn't.

If you capture a folder, and that folder is set to ignore changes, the changes may, or may not, be ignored (it's a bug and it depends on where the folder is in the hierarchy).

I will definitely make sure this is working before the next version is released.

Alex wrote:- First, despite rebooting, copying QRecall.app around, etc. I can't get LaunchServices to see the new "QRecall Capture Preferences" service. I do have "Capture to QRecall Archive", etc.

You might have to explicitly enable these new services in Finder > Services > Services Preferences…. If they're not listed there, you might have a "stuck" copy of the old services in ~/Library/Services/. Delete the QRecallService.service package from this directory and launch the QRecall app again. QRecall will reinstall the new service and notify OS X that it is available. Afterwards, it should appear in your Services Preferences.

- While I understand the value of exclusion filesystem attributes for items that may move, I'm actually much more concerned about never capturing items at particular paths. These are folders that may be deleted and recreated, and thus attributes will be lost. The good news here is that per-archive exclusions may work, except...

- The loss of exclusions per capture mean that one cannot set independent schedules for an item that is a subitem of a larger capture. For example, I backup my boot drive frequently, but exclude a subdirectory of VM images. However, previously the VM images could be captured on a separate event by a capture with a different exclusion set.

This is exactly the "mistake" that the new exclusion arrangement is designed to prevent you from making. Here's the problem with the way it works in QRecall 1.2:

You capture your home folder every hour, but exclude you VM folder.

Every evening, you capture your home folder but do not exclude your VM folder.

The problem with this arrangement is that your VM folder alternately appears, and then disappears, from your archive. After the first capture your VM folder would not be captured. After the second, it would. If your repeat the first capture again it's gone again. If your hard drive crashes and you restore from the latest capture, your VM folder would not be restored—because, as far as QRecall is concerned, it didn't exist when the third layer was captured.

The second problem this creates is that your VM folder get repeated captured in its entirely, rather than intelligently capturing just the changes. When the second capture runs, the VM folder doesn't exist in the archive history, so all of the files are captured as if they were brand new. After the third capture (that excludes the VM folder), the VM folder is erased in that layer. The next capture of the VM folder starts over from scratch because7mdash;again, from QRecall's perspective—all of the files are new.

What you really want to do is use the Ignore Changes feature. Set your VM folder to ignore changes for 20 hours or so. During the day, your startup volume or home folder can be quickly captured without trying to recapture your VM folder.

You can then create a second capture action to capture just your VM folder at the end of the day, or trigger that to run when your VM application quits.

Tip: I'd also suggest setting the "Deep Scan" option for the VM folder capture action, as VM packages (particularly Parallels) are notorious for tricking the filesystem change history into not seeing all of the changes.

Note that the "Ignore changes" option for capture preferences is not a good replacement, as it forces me to only ignore on a schedule whose timeline is opaque, I can't force a capture on an event (say, when VMWare quits).

Good point.

That's exactly why there's an exception to the rule: an ignore preference set on a folder is ignored when that folder is explicitly a target of a capture action. In other words, the per-item ignore preference only applies to items contained in an item being captured, never to the items listed in the capture action.

Finally a couple of UI issues:

- The exclusions scroll view in Archive settings is small, and cannot be resized. For exclusions inside ~/Library or long lists it's a frustrating interface to verify that all the items you want are present. This was not an issue with the prior interface in the capture action.

I've added this to the to-do list for QRecall 2.1, which will be a major UI facelift.

- If one does choose to use capture preferences as extended attributes, there's no simple UI to actually see what's excluded. Again this is necessary to verify you haven't missed something that should be excluded.

You can do this with the command-line tool. For example, this command will list all of the items contained in your home folder that have capture preferences set:

qrecall captureprefs list --recursive --skipnormal ~

Charles Stembridge wrote:How "beta" is 2.0? I'm hesitant to trust backups to a beta version.

It's actually very close to a release version. There are still some odd issues getting the scheduler run when installed, a few minor UI problems, and missing documentation. But there are currently no known issues that impact data integrity or reliability.

Charles Stembridge wrote:Any idea as to what I did wrong there?

I honestly don't know. Mail is really scattered all over the place now. Mail accounts are defined in the system's Internet Accounts database, preferences are in several locations, etc.

I helped a friend replace the SSD drive in your iMac and we could not get Mail running again after upgrading to El Capitan. I eventually relocated the Mail folder to the desktop, deleted and recreated all of her mail accounts (that restored her IMAP mailboxes), and then imported the local mailboxes back into Mail from the copies on the desktop. That seemed to work pretty well.

If you have captured copies of your mailboxes, you could try recalling your ~/Library/Mail folder to your desktop and try recovering mailboxes manually by reimporting.