AZ 2011 wrote:I am filevaulted.

Ah, that's the problem.

QRecall and FileVault don't play well together. It seems as though every security measure Apple adds to OS X in general, and FileVault in particular, breaks something in QRecall. Don't get me wrong; I'm actually big fan of OS X's security model, it just sometimes makes life hard for us legitimate developers.

My best suggestion is to uninstall QRecall in your current account, create a second admin account (if you don't have one already), and install QRecall there. Set it up so that actions run when logged out, then schedule the capture actions to run when you'll be logged into your FileVault account.

I had QRecall working with FileVault for awhile, but recent changes have broken that again. I have a to-do item to revisit FileVault compatibility, but I've put it off until I've added encryption to QRecall archives—users that are encrypting their home folder generally don't want an unencrypted copy setting on an external drive. As a workaround, I do have a few users who store their archives on encrypted disk images or hardware encrypted hard drives.

Johannes wrote:Thanks again for the quick response (this kind of support is really outstanding!).

I aim to please.

Time Machine was able to detect it. Doesn't Time Machine use FSEvents too?

Time Machine might detect it. Time Machine also uses FSEvents, but how it uses varies. If the process subscribes to the "live" feed of change events, the change event appears in the feed. But if a process requests the "historic" log of events, the change event is missing.

Time Machines uses a combination of both live and historic events. QRecall only uses historic events. I've performed controlled experiments and shown than when Time Machine (and every other backup utility that uses FSEvents) gets its change information from the historic record, it misses those changes too.

So I am in (b) too it seems.

That's good information to know. Can you also tell me what application was used to change the contents of the package?

Couldn't QRecall simply check the modification time in addition to FSEvents?

Yes, it's called a "deep scan".

QRecall can either read the metadata for every file and folder or it can trust FSEvents. There's really no middle ground.

Changing this setting to 0.0 brought home the changed file ...
It would be nice to have an option for a "deep scan" in the Action window.

I would prefer to address this some other way. I'm not a fan of adding user interface complexity just to work around a bug in the OS.

So I would have my longtime and off-site Archives run a deep scan always, while the hourly thing could do without.

Set QRAuditFileSystemHistoryDays to 0.95. That will force a deep scan once a day.

Or maybe I should go for always deep scanning. Missing changes is a serious issue for backups.

At least one of my users has chosen this route.

Johannes wrote:To me it looks like QRecall does not detect the change at all.

OS X didn't detect the change.

To speed up captures, QRecall uses OS X's File System Events service (or just FSEvents). This file-system level facility is supposed to keep track of every directory that contains a change (new file, a change in data, etc.) that occurs anywhere on the volume. Applications can receive this information as a real-time stream of changes, or can query the volume for a list of historical changes.

QRecall uses the later. When it starts, it requests a list of every folder that has been changed or contains a file that changed, since the previous capture. This appears in the log as "Collected XXX folder changes." It then restricts its scan to just those folders that contain modifications. When you see a log message like this:

CaptureProcessor(Tscho:Privat:Korrespondenz:GEZ.mellel) content assumed stable: inherit package 676866 (10 KB)

it means that OS X has assured QRecall that no changes have occurred in that folder, or any folders it contains. QRecall accepts this, assumes all of the previously captured information is up-to-date (i.e. it "inherits" the previously captured folder) and moves on.

However, it turns out that this is not always the case. I have two other users who were able to document where files were changed, but never reported by FSEvents. The two things they have in common are (a) they were running Windows in a virtual machine, and (b) the changed file was inside a package. I've reported this to Apple, but Apple has not yet released a fix or provided a workaround.

There are also other, well documented, situations where FSEvents also fails to report changes. QRecall deals with this by periodically ignoring the FSEvent information and preforming a "deep scan" where it examines every file and folder for possible changes?obviously, a much more time consuming process. This appears in the log as "Filesystem change history ignored." The default period is approximately once a week, but you can change this using the QRAuditFileSystemHistoryDays advanced QRecall setting.

Johannes,

This might be garbage metadata from OS 9 that OS X isn't handling. One key difference in the "before and after" shot is the 00 10 vs 00 00. These 16 bits contain the Finder flags, and the difference is that the first one has the kNameLocked attribute set.

This is a deprecated file attribute that I don't think is even supported anymore. It might be that OS X is inconsistently reporting the status of this field, so QRecall thinks it keeps changing. QRecall doesn't interpret any of the legacy FileInfo or FolderInfo structures; it simply compares them byte-for-byte and recaptures the item's metadata by making an exact copy.

This issue has come up with other metadata. The so-called "extended" finder information is now so full of deprecated/unsupported fields that QRecall now captures and compares only the two remaining supported values (extended finder flags and the put-away folder ID) from that structure.

You might try this as an experiment. Pick one of these files and restore it using QRecall. Capture the enclosing folder, and then capture it again. If QRecall doesn't capture it the second time, then I think this might solve the problem. If it works, it's because QRecall would recreate a copy of the file through the OS X file API. If there are any deprecated file attributes that OS X is now ignoring, it would discard them when creating the new file and should return 00's when asked for them back.

As for the other folders, it could be a similar issue if these are really old (OS 9 era) folders. Folders don't have much useful information in their finder info structures, so discarding it isn't a big loss. Just create a new folder, move the the contents of the old folder into the new one, and trash the old folder.

AZ 2011 wrote:Does something need to be setuid or have some other special flag in order to use administrator permissions?

Yes. Immediately after preauthorizing QRecall to run with administrative privileges, a SUID QRecallHelper should be installed in your Application Support folder, like this:

marchhare:~ james$ ls -l ~/Library/Application\ Support/QRecall/

lrwxr-xr-x  1 james  501       65 Dec 11 17:04 QRecallBundledHelper -> /Applications/QRecall.app/Contents/Resources/QRecallHelper

-r-sr-xr-x  1 root   501  3725104 Dec 11 17:04 QRecallHelper

lrwxr-xr-x  1 james  501       70 Dec 11 17:04 QRecallMonitor.app -> /Applications/QRecall.app/Contents/Resources/QRecallMonitor.app

I reviewed the code and your log file, and I found two really strange anomalies.

First, even though your QRecall application is install in /Applications, QRecall thinks that /Applications and /Users/you/Library/Application Support are on different volumes. Is there any chance that this is true?

The serious problem appears to be that your system is not allowing QRecallHelper to run SUID as root. The log shows that the QRecallHelper is copied to the Application Support folder and set to SUID. It then immediately runs the newly installed tool to check it out. It doesn't run as root, so the tool is immediately deleted.

That's why I asked you if your Application Support folder was on a different volume. Mac OS X, as a security measure, now ignores the SUID attribute of executables on external volumes.

QRecall 1.2.0(24) alpha is available to download, for anyone who wants to give it a spin.

This version addresses

Correct classification of Growl notifications

A crash when using the QRLogCaptureDecisions advanced setting

Memory efficiency while reading the names index.

A rare situation where the checksum of a new record was incorrectly calculated

I received your log file, and something is definitely off. When you authorize QRecall, the QRecallHelper process temporarily gets the correct administrative privileges, runs, and installs the pre-authorized copy of itself. However, the next time the system runs it, the helper doesn't have administrative (root) privileges. So it either reports an error or tries again.

Is your ~/Library folder on a different volume than the one you boot from?

Also, I'd be very interested to see the results of issuing the following two commands in the Terminal:

ls -ld ~/Library/Application\ Support

ls -l ~/Library/Application\ Support/QRecall

Johannes wrote:

James Bucanek wrote:The "exclude caches" filter works by getting the home-path property of each user account, and using that path to locate the ./Library/Caches folder for each user. If the home-path property returns an invalid path, then none of the user-specific filters (caches, trash, ...) will work.

That's the reason for the problem. I have two users on my machine with their user folder on disks that are not attached most of the time. Perhaps when revising the filter section next time you can teach your filter to ignore those unavailable paths (no need to exclude something thats not available )

They already do. I checked the code again, and the filters are built on a per-account basis. Filters that end up pointing to non-existent items are ignored, but all of the others should work. I'd be really curious to know what the home path property of your user account is.

Johannes wrote:I don't know who's the culprit Growl or QRecall.

QRecall.

QRecall was including the wrong internal identifier in some of the Growl notifications, so disabling the "started" notification was also disabling other notifications too.

Version 1.2.0(23) alpha, which I'm about to build and post to the forum, should also address this problem too.

Johannes wrote:Nice feature - but not crashing the QRecallBundledHelper

Ouch, definitely not.

Thanks for the crash report. I'm testing a fix for this now. I'll post a link to download an alpha release later today.

AZ 2011 wrote:Found the key combination: shift + command + option.

Ooops, my bad.

Help report has been sent, you should have it now.

It didn't arrive, so something is really wrong.

Uninstall, reinstall, test... same error in the log.

Try to sending me your log file manually. In your <home folder>/Library/Logs/QRecall folder, find the QRecall.log file. Select it in the finder and compress it (File > Compress "QRecall.log"). Send the compressed file to james@qrecall.com.

Johannes wrote:I wonder: what could be the reason for this behaviour?

There's a setting just for this.

In the Advanced QRecall Settings post, you'll find the QRLogCaptureDecisions settings. Set it to true before your next capture and QRecall will log the reason (well, at least the first reason) it decided to capture each item. Do remember to turn it off once you've conducted your investigation.

Johannes wrote:

James Bucanek wrote:Look in your system console (either around the time that you last logged in or QRecall was installed for the first time). Look for a message that says "Could not load Growl.framework". If you find this, then that's the problem.

Nothing like that in Console. Strange thing.

Are there any Growl related message at all? If so, please forward them to me.

If there's no message like that, then QRecall found the Growl.framework and successfully loaded it. The next thing to check is that the QRecallMonitor background application is running when you start an action. Start up Activity Monitor and search for QRecall. You should see a QRecallMonitor process running. If not, then that's the problem. I've looked at your logs and it appears that the monitor is getting started whenever you log in, so it should be running when your actions start.

One explanation might be that you were logged out. If you had QRecall actions scheduled to run and you're logged out at that time, either QRecallMonitor or Growl could miss that event. This can even happen even if the scheduler isn't set to run actions while you're logged out; when you log back in the scheduler, the monitor, and Growl are all starting up at the same time. If the scheduler gets started first (and it likely will), all of the start action events will happen before Growl is ready for them.

And for the record, the Growl.framework package is a resource of the QRecallMonitor application, not the QRecall application. QRecallMonitor is the process that runs all of the time you're logged in and provides services like the activity window, disk mount events, and Growl notifications.

Adam Horne wrote:Would you mind going through your process for installing your slim OS on a USB drive?

I know how to install the OS on the drive itself, but what do you do after that?

As little, or as much, as you want. The goal is to create a small footprint for the recovery OS. Depending on how big your emergency startup volume is, this might mean stripping every last possible byte to make it fit, or just trying not to waste too many gigabytes.

The biggest gain (loss?) is to start with the OS X install options. During installation, make sure to uncheck any optional features (additional languages, printer drivers, etc.).

You said you erase everything, but terminal and disk utility.

Optional. But there are scores of applications that you'll never use on your recovery drive, so that's just space you can't use for backups.

Do you literally just go through the application folder and delete everything?

Pretty much.

Do you delete more?

I suppose you could, but the optional install packages and the extraneous applications are the low hanging fruit. Trying to find more to shed gets progressively harder, with little to gain.

Do you install all the updates Apple puts out for the OS?

I don't bother. The only really important thing is that the major version of the operating system must be the same as the one you're trying to restore. So if you're running 10.6.4 you'll need to boot some version of 10.6.x in order to restore it. The same is true for 10.5 and 10.4. Remember, the key to an emergency startup volume is that it's reliable, not that it's up-to-date.

Best of luck.

Johannes wrote:One reason might be that my user folder is a dedicated Volume and therefore not within the /user folder of the System volume.

That could explain it. The log contains a number of messages like this:

+[SpecificItemFilter filterWithCPath:onVolumeID:shouldExist:] could not find '/Volumes/Elke'

This indicates that one of the filters couldn't find a path that it expects to exist. The "exclude caches" filter works by getting the home-path property of each user account, and using that path to locate the ./Library/Caches folder for each user. If the home-path property returns an invalid path, then none of the user-specific filters (caches, trash, ...) will work.

I have not used a utility to alter the system font (but my eyes are quite happy that the font is not smaller). A simple solution would be to allow the user to adjust the column width.

and

Bruce Giles wrote:I do not see the problem Johannes reported about the log window on my system either. However, from Johannes' name, as well as the name he gave to the screen shot, I'm guessing he's using a non-US system. Could that account for a slight difference in the font size?

Bruce might be on to something, although I'd be surprised if the system font metrics were different. Most of the system fonts in OS X are full Unicode fonts that work for most of the languages which OS X supports.
None the less, this is as good a theory as any. I'll have to try some different localizations and see what happens. Thanks for the suggestions.

I am using the same version, the notifications are enabled and I did a restart. But I noticed that the QRecall package does not contain Growl.framework in the framework folder (but the Growl preferences lists QRecall).

That might explain it. Look in your system console (either around the time that you last logged in or QRecall was installed for the first time). Look for a message that says "Could not load Growl.framework". If you find this, then that's the problem.