QRecall

James Bucanek wrote:This appears in the log as "Filesystem change history ignored."

Johannes wrote:
Here are the file records; ...

Thanks, Johannes! As soon as I looked at those records, I knew exactly what was wrong.

Please download and install QRecall 1.2.0(25) alpha, which fixes this bug. The problem was a mismatch between the way extended finder information was captured and how it gets restored. I'm embarrassed to say that this was caused when I merged two branches of the QRecall project to create 1.2.0a24. Some recent optimizations in metadata handling got into a24, while others didn't.

The good news is that the errent code was isolated to the restore logic. All of the information in the archive is correct, and if you restore the same items using a25 they should be OK.

Now I know what the Dump-Command is for. Is this alpha only?

This command (and a few others) only appear in the alpha and beta releases, and are intended for debugging problems just like this.

What is the diference between Archive>Dump and File>Dump??

One of them works.

Could this be a left over from Big/Little Endian issue between Intel and PowerPC machines? When I created this file I was on a PowerPC. Now I am on Intel. Just a thought ?

It's a good thought. There were a few file system structures in 10.4 and 10.5 that didn't get swapped property during the early migration to Intel, but to my knowledge all of these have been fixed. For cross-platform compatibility, all structures are stored big-endien in the filesystem. So anything originally written using a PowerPC machine should be correct on the disk. The only possible problem would be Intel software that fails to swap the bytes when it reads the data.

Good catch!

Johannes wrote:

James Bucanek wrote:
Set QRAuditFileSystemHistoryDays to 0.95. That will force a deep scan once a day

On what base will QRecall do the maths? Once a day for each Archive? For each Action? For each machine?

It's based on how long it's been since a deep scan was performed for the items being captured.

Will I get a deep scan for Action 2 and 3 every time with the above setting?

Yes.

QRecall keeps track of how long it has trusted FSEvents to determine changes for each (top-level) item captured in the archive. During the next capture, it compares the scope of what's being captured to what's in the archive and determines the minimum common overlap between the two. It then compares the "FSEvent trust" dates and finds the largest interval.

Let's say QRAuditFileSystemHistoryDays was set to 1.0. Now let's say you captured '/' two days ago and '/Users/you/Document' two hours ago, both using a deep scan.

If you were to capture 'Documents' again, the difference in trust dates would be 2 hours, so no deep scan is required. If you captured '/Users/you' the difference would be 2 days, because the oldest capture that encompasses '/Users/you' in the archive is '/', which is now two days old.

Now instead of capturing '/Users/you', let's say you captured '/Users/you/Documents/Projects' an hour later, QRecall would not perform a deep scan because 'Projects' is encompassed by '/Users/you/Documents', and the latest capture of 'Documents' trusted FSEvents for 2 hours, so the aggregate trust time is now 3 hours.

So as you can see, it's kind of complicated, but it also can't easily be tricked into trusting FSEvents for longer then it should.

Johannes wrote:ls -la@ in recall folder and in original folder (no difference):

Well, the file got restored correctly. That's good news.

xattr -l in recall folder and in original folder (looks like the same difference like with some other old files):

00000000  50 49 43 54 47 4B 4F 4E 01 00 00 46 FF 51 00 00  |PICTGKON...F.Q..|

00000010  00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00  |................|

00000020



00000000  50 49 43 54 47 4B 4F 4E 80 00 00 46 00 00 00 00  |PICTGKON...F....|

00000010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|

00000020

Similar, but not exactly. In this case one file has Finder flags of 0100 and the original has 8000. The value 0100 indicates kHasNoINITs (an obsolete OS 9 attribute), while 8000 mean kIsAlias. (Are these backwards by any chance?)

It would be interesting to find out what finder information QRecall read, and captured, when it got the file in the first place. And since you're running an alpha version of QRecall, you can do that.

:

Open the archive with this file and choose Archive > Dump. In the data section, check "File Package" "File Package: Details" and "Scan by Package". Uncheck every other option in the dialog. Click Dump (it may take awhile, during which QRecall will be frozen). Open the dump file with a text editor and find the file records for the 'bergkristal1.pict' file.

Johannes wrote:Any idea why they became Aliases when restoring (recalling) them?

I have no clue. I've never seen that happen.

What do the files look like if you list then in the terminal (ls -l)? And if you have GetFileInfo installed, it would be interesting to see what their file attributes are, and how they differ from the attributes of the original.

Also, try duplicating them in the Finder and see if the Finder still thinks they're alias files.

Johannes wrote:Is there a terminal command to clear their extended finder info?

I though there was, but I can't find one. I know you can delete ACL with chmod, but I don't know of a utility that will let you arbitrarily delete extended attributes. I guess that's why I probably end up writing a program to do it.

AZ 2011 wrote:I am filevaulted.

Ah, that's the problem.

QRecall and FileVault don't play well together. It seems as though every security measure Apple adds to OS X in general, and FileVault in particular, breaks something in QRecall.

Don't get me wrong; I'm actually big fan of OS X's security model, it just sometimes makes life hard for us legitimate developers.

My best suggestion is to uninstall QRecall in your current account, create a second admin account (if you don't have one already), and install QRecall there. Set it up so that actions run when logged out, then schedule the capture actions to run when you'll be logged into your FileVault account.

I had QRecall working with FileVault for awhile, but recent changes have broken that again. I have a to-do item to revisit FileVault compatibility, but I've put it off until I've added encryption to QRecall archives—users that are encrypting their home folder generally don't want an unencrypted copy setting on an external drive. As a workaround, I do have a few users who store their archives on encrypted disk images or hardware encrypted hard drives.

Johannes wrote:Thanks again for the quick response (this kind of support is really outstanding!).

I aim to please.

Time Machine was able to detect it. Doesn't Time Machine use FSEvents too?

Time Machine might detect it. Time Machine also uses FSEvents, but how it uses varies. If the process subscribes to the "live" feed of change events, the change event appears in the feed. But if a process requests the "historic" log of events, the change event is missing.

Time Machines uses a combination of both live and historic events. QRecall only uses historic events. I've performed controlled experiments and shown than when Time Machine (and every other backup utility that uses FSEvents) gets its change information from the historic record, it misses those changes too.

So I am in (b) too it seems.

That's good information to know. Can you also tell me what application was used to change the contents of the package?

Couldn't QRecall simply check the modification time in addition to FSEvents?

Yes, it's called a "deep scan".

QRecall can either read the metadata for every file and folder or it can trust FSEvents. There's really no middle ground.

Changing this setting to 0.0 brought home the changed file ...
It would be nice to have an option for a "deep scan" in the Action window.

I would prefer to address this some other way. I'm not a fan of adding user interface complexity just to work around a bug in the OS.

So I would have my longtime and off-site Archives run a deep scan always, while the hourly thing could do without.

Set QRAuditFileSystemHistoryDays to 0.95. That will force a deep scan once a day.

Or maybe I should go for always deep scanning. Missing changes is a serious issue for backups.

At least one of my users has chosen this route.

Johannes wrote:To me it looks like QRecall does not detect the change at all.

OS X didn't detect the change.

To speed up captures, QRecall uses OS X's File System Events service (or just FSEvents). This file-system level facility is supposed to keep track of every directory that contains a change (new file, a change in data, etc.) that occurs anywhere on the volume. Applications can receive this information as a real-time stream of changes, or can query the volume for a list of historical changes.

QRecall uses the later. When it starts, it requests a list of every folder that has been changed or contains a file that changed, since the previous capture. This appears in the log as "Collected XXX folder changes." It then restricts its scan to just those folders that contain modifications. When you see a log message like this:

CaptureProcessor(Tscho:Privat:Korrespondenz:GEZ.mellel) content assumed stable: inherit package 676866 (10 KB)

it means that OS X has assured QRecall that no changes have occurred in that folder, or any folders it contains. QRecall accepts this, assumes all of the previously captured information is up-to-date (i.e. it "inherits" the previously captured folder) and moves on.

However, it turns out that this is not always the case. I have two other users who were able to document where files were changed, but never reported by FSEvents. The two things they have in common are (a) they were running Windows in a virtual machine, and (b) the changed file was inside a package. I've reported this to Apple, but Apple has not yet released a fix or provided a workaround.

There are also other, well documented, situations where FSEvents also fails to report changes. QRecall deals with this by periodically ignoring the FSEvent information and preforming a "deep scan" where it examines every file and folder for possible changes?obviously, a much more time consuming process. This appears in the log as "Filesystem change history ignored." The default period is approximately once a week, but you can change this using the QRAuditFileSystemHistoryDays advanced QRecall setting.

Johannes,

This might be garbage metadata from OS 9 that OS X isn't handling. One key difference in the "before and after" shot is the 00 10 vs 00 00. These 16 bits contain the Finder flags, and the difference is that the first one has the kNameLocked attribute set.

This is a deprecated file attribute that I don't think is even supported anymore. It might be that OS X is inconsistently reporting the status of this field, so QRecall thinks it keeps changing. QRecall doesn't interpret any of the legacy FileInfo or FolderInfo structures; it simply compares them byte-for-byte and recaptures the item's metadata by making an exact copy.

This issue has come up with other metadata. The so-called "extended" finder information is now so full of deprecated/unsupported fields that QRecall now captures and compares only the two remaining supported values (extended finder flags and the put-away folder ID) from that structure.

You might try this as an experiment. Pick one of these files and restore it using QRecall. Capture the enclosing folder, and then capture it again. If QRecall doesn't capture it the second time, then I think this might solve the problem. If it works, it's because QRecall would recreate a copy of the file through the OS X file API. If there are any deprecated file attributes that OS X is now ignoring, it would discard them when creating the new file and should return 00's when asked for them back.

As for the other folders, it could be a similar issue if these are really old (OS 9 era) folders. Folders don't have much useful information in their finder info structures, so discarding it isn't a big loss. Just create a new folder, move the the contents of the old folder into the new one, and trash the old folder.

AZ 2011 wrote:Does something need to be setuid or have some other special flag in order to use administrator permissions?

Yes. Immediately after preauthorizing QRecall to run with administrative privileges, a SUID QRecallHelper should be installed in your Application Support folder, like this:

marchhare:~ james$ ls -l ~/Library/Application\ Support/QRecall/

lrwxr-xr-x  1 james  501       65 Dec 11 17:04 QRecallBundledHelper -> /Applications/QRecall.app/Contents/Resources/QRecallHelper

-r-sr-xr-x  1 root   501  3725104 Dec 11 17:04 QRecallHelper

lrwxr-xr-x  1 james  501       70 Dec 11 17:04 QRecallMonitor.app -> /Applications/QRecall.app/Contents/Resources/QRecallMonitor.app

I reviewed the code and your log file, and I found two really strange anomalies.

First, even though your QRecall application is install in /Applications, QRecall thinks that /Applications and /Users/you/Library/Application Support are on different volumes. Is there any chance that this is true?

The serious problem appears to be that your system is not allowing QRecallHelper to run SUID as root. The log shows that the QRecallHelper is copied to the Application Support folder and set to SUID. It then immediately runs the newly installed tool to check it out. It doesn't run as root, so the tool is immediately deleted.

That's why I asked you if your Application Support folder was on a different volume. Mac OS X, as a security measure, now ignores the SUID attribute of executables on external volumes.

QRecall 1.2.0(24) alpha is available to download, for anyone who wants to give it a spin.

This version addresses

Correct classification of Growl notifications

A crash when using the QRLogCaptureDecisions advanced setting

Memory efficiency while reading the names index.

A rare situation where the checksum of a new record was incorrectly calculated

I received your log file, and something is definitely off. When you authorize QRecall, the QRecallHelper process temporarily gets the correct administrative privileges, runs, and installs the pre-authorized copy of itself. However, the next time the system runs it, the helper doesn't have administrative (root) privileges. So it either reports an error or tries again.

Is your ~/Library folder on a different volume than the one you boot from?

Also, I'd be very interested to see the results of issuing the following two commands in the Terminal:

ls -ld ~/Library/Application\ Support

ls -l ~/Library/Application\ Support/QRecall

Johannes wrote:

James Bucanek wrote:The "exclude caches" filter works by getting the home-path property of each user account, and using that path to locate the ./Library/Caches folder for each user. If the home-path property returns an invalid path, then none of the user-specific filters (caches, trash, ...) will work.

That's the reason for the problem. I have two users on my machine with their user folder on disks that are not attached most of the time. Perhaps when revising the filter section next time you can teach your filter to ignore those unavailable paths (no need to exclude something thats not available )

They already do. I checked the code again, and the filters are built on a per-account basis. Filters that end up pointing to non-existent items are ignored, but all of the others should work. I'd be really curious to know what the home path property of your user account is.

Johannes wrote:I don't know who's the culprit Growl or QRecall.

QRecall.

QRecall was including the wrong internal identifier in some of the Growl notifications, so disabling the "started" notification was also disabling other notifications too.

Version 1.2.0(23) alpha, which I'm about to build and post to the forum, should also address this problem too.