Bruce,

The problem is that OS X isn't starting the scheduler. You have the scheduler set to run in the background (the "Start and run actions while logged out" scheduler option is checked). Currently, OS X is not starting the scheduler the way it should when you start up. When the monitor can't connect with the scheduler, most of the commands related to the scheduler will be disabled, and the special menu item that displays the next scheduled action is broken.

When you ran QRecall, it saw that the scheduler wasn't running, (redundantly) reinstalled it, and told OS X to immediately start the service. So now the scheduler is running, and the monitor menus all work.

Short-term solution is to uncheck the "Start and run actions when logged out" and the scheduler will get installed as a user agent, which works reliably (knock wood).

I've spent the better part of this week running this problem to ground, and I think I'm really close to a (partial) solution. Details in the release notes of 2.0.0b21...

Charles Watts-Jones wrote:Running QRecall b20, I tried to run a back-up Action this morning after my attempts to run one to a NAS when logged out, failed. The Action refused to run because it was seeing an 'invalid identity key'. When I looked at Preferences > Identity Key, the Key Status reported 'Valid permanent key'.

That's very odd. The only thing I can think of is that OS X was having problems accessing one of your QRecall preference files (which is where your identity key is stored). This can happen occasionally in OS X 10.7 and later, but it's usually a temporary situation.

I tried re-entering my key only to be warned that doing so would change ownership of the archive. Odd?

That's a new warning in QRecall 2.0. If you change your identity key, it's warning you that items captured with the new key will belong to a different owner inside the archive—so don't be surprised if you don't see the new items inside the existing owner.

Ralph Strauch wrote:I'm seeing log entries for both recaptured files and capture issues attributed to "file length changed during capture" for various log files. Is this normal? Should I just exclude these logs from the capture?

This is a new message in QRecall 2.0. It lets you know that a file was being written to at the same time QRecall was trying to capture it. For things like log files, it's not an issue. Log files are always appended to, so what QRecall captures is always valid, even if it didn't get everything.

If it's something more complicated, like a database or document file, it may be a concern when restoring it. It might mean QRecall did not capture a stable copy of the file, and you should careful evaluate such files after restoring them. You might need to restore an earlier, or later, version in order to obtain a usable document.

I also see that my qrecall logs for September and October are both under 2MB, while the November log is already over 2.6GB. Has the amount of stuff being logged shot way up for some reason, should it be set back to a lessor level?

I can't say for sure without looking at the log file, but the beta writes a lot more in the log than the release version.

Bruce Giles wrote:I have two separate archive files: one called "iMac 24.quanta" which is my main backup file. The second one, called "Virtual Machines.quanta" is used to backup my Virtual Machines folder. Usually, that's for VMware Fusion, but on occasion, I have VirtualBox files in there too. The settings for the "iMac 24" archive has all the checkboxes in the Exclude section checked, but nothing listed in the box where custom locations can be added. The settings for the "Virtual Machines" folder are exactly the same in that respect.

Now, the "Virtual Machines" folder on my hard drive has QRecall capture preferences -- the "Do not capture" button is the only one selected.

From this description, I can tell you that your "Virtual Machines" folder will never be captured.

The crucial setting you set in both of your archives is the "Exclude: Individually Excluded Items" option. This setting honors the "Do Not Capture" preference you set on individual filesystem items. With the "Do Not Capture" preference set on your "Virtual Machines" folder, that folder won't ever be captured.

You have a couple of choices. Probably the simplest is to uncheck the "Exclude: Individually Excluded Items" option of your Virtual Machines.quanta archive. The iMac 24 archive will exclude everything you've individually set to exclude, while Virtual Machines will ignore that capture preference and capture them all.

It seems pretty clear from the log that it's not capturing the folder because it's excluded (by the capture preference). But I thought b20 (which is what I'm running) had a bug fix that was supposed to allow that when I was specifically trying to capture a folder that was ordinarily excluded. If so, that's not working. Or did I misunderstand?

That exception only applies to the "Ignore Changes" preference. Excluded items are always excluded, and should be logged as such.

I suspect the better solution, given my archive setup, is to delete the capture preference, and add an exclusion for the Virtual Machines folder to my "iMac 24" archive only. Then my regular backups should skip the folder, but my Virtual Machines capture action will get it, because it's capturing to a different archive which doesn't have an exclusion for the folder in the archive settings. Do you agree?

Dealer's choice, but I'm inclined to agree. My philosophy on this is:

Add items to the exclusion list of the archive when those exclusions are for the benefit of that archive, and that archive alone. For example, an archive that is focused on a particular subset of files (say, working projects) and you want to hone that focus by excluding extraneous items.

Add items to the exclusion list of the archive when the filesystem the items are on doesn't support extended attributes or there's a chance that the item could be deleted and recreated (which might discard the capture preferences attached to the original item).

Check the "Individually Excluded Items" option and use the capture preferences to conveniently exclude items from multiple archives. It's easy to change what items are excluded (no need to open the archive) and it applies to all archives.

If you have one or more "hot" archives that capture only important files on a regular basis (2 minutes after documents changed, for example) and a second "comprehensive" archive that captures everything (say, once a day), then use capture preferences to exclude the less interesting files from the "hot" archives and uncheck the "Exclude: Individually Excluded Items" from the "comprehensive" archive. The quick captures during the day will exclude items marked "Do Not Capture" and the daily capture will capture them anyway.

In this last scenario, you may find that there are items you never want to capture, ever. That's what the "Exclude from all archives" capture preference option is for. Even if an archive is set to capture "Do Not Capture" items, this option overrides that setting and excludes the item anyway.

Or, to paraphrase Uncle Ben from the Amazing Spiderman: "With great flexibility comes great confusion."

Charles Watts-Jones wrote:Operating Systems have moved on since this thread started and perhaps the conditions have changed, so I'm resuscitating the question.

Not much has changed since this feature was first introduced. When QRecall starts an action, and the archive for that action isn't online (it isn't reachable, in the language of the filesystem), it requests that OS X make it available. OS X then looks to see if the archive was originally on a volume that's physically connected to the system and can be mounted, or if it was on a network volume that the OS can reconnect to. Encrypted and networked volumes might require supplying a password from the keychain.

If any of these things are not true (the drive isn't connected, the server isn't online, the devices driver doesn't understand filesystem mount requests, or the keychain can't be accessed), then the volume won't be mounted and the action can't run.

I'm trying to use ControlPlace (a freebie available at http://www.controlplaneapp.com/) to handle the mounting of a NAS drive. It uses Rules to trigger Actions. The Rule that I've tried is to name the application which will trigger the drive mount. So far I've tried QRecallMonitor and then QRecallScheduler. Neither appears to work; ... I'd appreciate knowing what application should be named.

I doubt this rule will be useful. QRecall actions are performed by an executable. This is binary executable file, but it is not a Cocoa application bundle, which is what is generally referred to as an "application" in OS X. The OS X workspace manager posts notifications when a Cocoa application (like Mail) launches or terminates, and this is probably what ContolPlace is listening for. I highly doubt it's capable of determining when an executable is spawned, but if it is the executable is named QRecallHelper.

I say 'appears' because I'm getting a 'NetAuthSysAgent wants to use the "login" keychain' message when I login.

No clue.

Can ControlPlace be set up to mount your volume when you execute a shell script? If so, you might look into Growl 2.0.

Growl is a notification manager that long preceded—and was probably the inspiration for—the built-in notification manager added in OS X 10.8. Growl 2.0 (available from the Mac App Store) adds the ability to perform various actions when it receives specific notifications. This includes turning the notification in an email or SMS message, or even turning it into an iOS push notification via Prowl (available in the iOS App Store). That last one is the feature I use the most. I have all of my servers push their "Action Failed" notifications to my iPhone.

QRecall sends four different notifications to Growl (and OS X's built-in notification center): Action Started, Action Complete, Action Failed, and Archive Needs Repair.

You can set up Growl to run an ActionScript when it receives any of these notifications. The script can be passed the notification's name (Action Started, etc.) which it could use to determine if the volume should be mounted or unmounted.

You'll still want to the keep the "Hold While No Archive" condition, to avoid the race condition between the action (that will immediately try to open the archive) and the script (that's simultaneously trying to mount the volume that archive is on).

Thanks for the feedback.

I tend to agree with you on all counts, but I wanted to throw out the idea to see if an alternative made more sense for some people.

I'll leave the logic the way it is now.

Ralph Strauch wrote:Related problem: ... From there, when I open the archive and select a file to delete, but when I hit the Delete button I get a window that says "Waiting for Archive," and nothing happens. It's telling me this while the archive is sitting there open in front of me. What's blocking the delete function on the MBP? (report sent)

This has been mentioned in another thread, but it bears repeating.

Short answer: be patient.

Long answer:

QRecall 2.0 uses a new method for arbitrating access to an archive between current actions and multiple users. Inside the archive package is a new set of invisible semaphore files (.lock, .share, and .semaphore). These files allow QRecall to negotiate access to the archive on devices that don't support the shared-read/exclude-write access control required to safely share the files.

One problem is that these files can become "stale" and indicate that the archive is being accessed by another process when it isn't.

The next action to request access to the archive sees the stale lock and you get the "Waiting for archive" message. Behind the scenes, it begins a series of tests to determine if the archive really is being used. These tests take about 10 minutes to determine that no other process is modifying the archive and the lock is stale. Once it's convinced that it's safe to access the archive, it will "break" the lock and acquire the access it needs.

So, just let it "wait". It will eventually determine that it can safely access the archive and the action will proceed.

Ralph Strauch wrote:I back up both a MBP and an iMac to the same archive, and the paths to excluded items in my archives seem to depend on which computer I'm looking at them with rather than on which drive the files are actually on.

That's expected. QRecall 2.0 uses URL Bookmarks to identify and remember items in places like the archive preferences and actions. Bookmarks work really hard to identify an individual file or folder. Thus, when viewed on a different system, the bookmark will resolve to the same item, but that item might have a different filesystem path.

I guess this may work out OK in practice because the files that need to be excluded are the ones on the computer doing the backup and those paths are correct, so the paths to nonexistent files that are really on the other computer just get ignored. It can be confusing, though, to look at the exclusions list and see a bunch of nonexistent files.

It kind of has to work this way, else you run into the inability to exclude a folder on one system and not exclude it on another.

In a nutshell, the best way to exclude items using the archive settings on multiple systems it to open the archive on the first system and add the items to be excluded for that system. Close the archive and repeat with the next system.

Alternatively, you can use the new QRecall Capture Preferences to exclude each item individually.

The startup filesystem needs headroom for things like swap and temporary files. If you completely run out of space on the VM swap volume, you are in serious doo-doo...

But even non-boot filesystems start to hurt as their free space dwindles towards zero. The problem is the volume's allocation map. As the drive fills up, it's harder and harder of the filesystem to find free blocks, and those blocks become increasingly smaller and more fragmented. Eventually, the filesystem slows to a crawl trying to find and manage the free space. You can improve this by defragmenting the drive, but when it's nearly full even that becomes problematic.

Because of this, and other reasons, QRecall avoids letting the archive volume fill up completely. The capture action monitors the free disk space and will stop capturing when the volume exceeds approximately 98% full. (The actual amount is variable, based on the size of the volume.)

Filling up a volume, completely, is a nasty situation in OS X (a.k.a. BSD Unix). Because of this, QRecall reserves a 4% buffer when estimating the amount of free space on your drive.

So the volume with your 1.23 TB archive has 197 GB of free space. Assuming the archive is just about the only thing on the volume, that works out to about 3% free space. As far as QRecall is concerned, your drive is already full.

To add 1:8 redundancy to at 1.23 TB archive, QRecall will want to see 154 GB (1/8th of 1.23 TB) + 4% of the volume size (~50 GB), or at least 204 GB of free space before it will proceed.

Volumes "A" & "B" are probably in a different owner.

A single QRecall archive can capture data from multiple owners/systems/identities. These owners are at the top-level of the archive. After your first capture, the browser defaulted to the iMac owner, which has captured volumes "C" & "D".

After capturing volumes "A" & "B", your archive's browser view is still looking at the volumes captured for the iMac. Use the navigation bar at the bottom of the window and navigate to the root (archive) level of the browser tree. There you will see two owners, "iMac" and "Laptop" (or, specifically, the owner names you gave them when you entered your identity keys).

Drill into "Laptop" and you'll see the two new volumes.

If the owner names are too similar, you can rename them in QRecall > Preferences > Identity Key (for that system). The new names will appear in your archive (immediately for you local identity, or after the next capture for the other).

Ralph Strauch wrote:I've been getting this "system clock. . ." message repeatedly for the past three days (since starting to use v2 on this computer).

This is normal. The 2.0 scheduler now monitors the system clock for changes. Whenever it is notified that the system clock has been adjusted, the scheduler reevaluates the scheduled actions. This was to solve a problem of users who change their clock to something way in the future, and then set it back. The result would be actions that wouldn't start again for months, even years, because they were waiting for some future date that was now way in the future.

It turns out that OS X adjusts the system clock all of the time, mostly millisecond changes to keep the clock in sync with NTP. Every time this happens, the scheduler dutifully reevaluates its schedule.

For the most part, these messages can be ignored. It's just the scheduler doing its job. There's a reason these log messages are classified as "minutia." (I might even demote them to "debug" messages, as it turns out they are only rarely interesting.)

Other strange things also seem to be happening. A few files have been "recaptured" for some reason,

Determine why an item was recaptured is a really complicate topic, as there are lots of (sometimes obscure) reasons why an item would be recaptured.

The only definitive way of determining why requires that you first set the QRLogCaptureDecision (Log Capture Decisions) setting to true before the next capture. In 2.0, you can do that in the advanced preferences. When this is set, the capture action will log the reason it captured each item. Warning: this setting can result in a tremendous number of log records. I do not recommend leaving it on.

and yesterday afternoon the app ran a verify action that seemed to start all by itself.

That's a new one. The log should indicate what started that action. I'll take a look at your report.

One capture died after a "broke stale lock file" entry.

We were just discussion that in this thread.

Poll:

Thinking about this issue a little bit, it wouldn't be difficult to make the recapture of items that have the "Ignore" capture preference set into an option, the same way that the new "Deep Scan" option ignores filesystem change history when looking for modified folders.

So instead of having a special exception for items marked as "ignore", applied only to the items specifically being captured, what about a new capture action option ("Capture Ignored Items") that explicitly ignores the ignore preferences, and recaptures all changed items?

Alternatively, I could just bake that feature in to the existing "Deep Scan" option, so a deep scan would also imply capturing ignored items.

Any opinions?

Alex wrote:I understand your explanation, but I think its pretty surprising. Why would exclusion translate to a deletion in the archive?

It doesn't get deleted (in all layers), as it would if you selected the item and used the delete command.

What I mean is that it gets recorded as a deleted file in the new layer. Here's an example. You have a folder (A) with two files (X and Y). You capture folder A:

Layer 1: Folder A has two files, X and Y

You now exclude item Y and capture the same folder again.

Layer 2: Folder A has one file, X

As far as QRecall is concerned, file Y was deleted. It existed when layer 1 was captured, and didn't exist when layer 2 was captured.

More importunely, if you recall from layer 2, there will be no file Y. (Replace file Y with your VM folder, and you get the idea.)

If you later include file Y again, file Y get re-captured again. Now this won't add any significant amount of new data to the archive—thanks to data de-duplication—but it defeats QRecall's ability to intelligently capture only the changes. QRecall can't look at that file (or the contents of an entire folder) and determine that nothing has changed and simply skip recapturing it. This adds a significant amount of overhead and also prevents you from determining what changed by examining the individual layers, because it looks like everything changed, every time it's recaptured.

You'll definitely want to mention this in docs, I would have assumed it worked according to the rules you mention as a bug.

Yes, there's a lot that still needs to be documented.

Ralph Strauch wrote:Is it safe to make another try at adding error correction now, or should I wait for the next release?

As far as I can tell, your issues with adding error correction was in the QRecall app (the monitor), not the action (the doer). It appears that the action continued to run (until the computer was restarted), as I can find no crash logs or other evidence that it died abnormally.

I'd suggest you give it another run. Just remember that it has to read the entirety of the archive data, so it will take about as long as a verify.