Gary K. Griffey wrote:So...it would appear that the 10.6 File System events did not advise QRecall to backup the Vm package..even though it most certainly had been updated. From your previous notes...I do not see any reason the file system events on the laptop's drive should have been deleted or otherwise compromised. What am I doing wrong?

You're not doing anything wrong. If QRecall is not scanning the directory containing your VM image, then it's because file system events is not recording changes for it. I have a couple of thoughts...

First, check your console for anything related to FSEvent or fseventd. In the past, people have reported errors being logged by the fseventd daemon. This may indicated that file system events are not being recorded property.

How do you run your Windows system? Do you reboot your computer into Windows (a la Bootcamp) and the restart again in OS X, or do you run something like Parallels so that OS X and Windows are running concurrently? It occurred to me that in the former situation, OS X isn't actually running, so changes to VM files (even if they're recorded in the file system) won't be recorded by the file system events service.

If you're running the later configuration, you can see if file system events records changes for your VM package using a free utility like FSEventer (insert Google search here). Your VM might be going behind OS X's back (so to speak) and reading and writing blocks at the driver level. This would also circumvent the file system events service.

A tool like FSEventer still won't show you exactly what QRecall sees because it only displays file system events as they occur, whereas QRecall requests historic events (i.e. please Mr. OS, send me all change events from point X in the past). So where FSeventer logs events as they occur, QRecall is interested in the permanent record of events stored on the drive.

You mentioned that these file system events are located in a hidden file on the drive itself...

That's they way it's supposed to work. But if FSEventer shows change events for your VM folder, you reboot, and then QRecall doesn't see them, then the FSEvents service might not be recording them property.

obviously, you are using an API or equivalent to read these values...it there any way that I could actually peer into this hidden file to see if the VM package file is being flagged as changed?

I'm not aware of any freeware tool for requesting historic FSEvent data. I wrote one of my own for testing. If you can't figure this out, I'll see if I can dig it up and polish it off.

I just wonder what other user/system related files are being skipped in my incremental backups...makes you question the value of any backup system using these events..like Time Machine, etc.

That's a good question. QRecall, Time Machine, Apple Events, and most other disk backup and synchronization software all rely on FSEvents. If it's not reporting changes consistently, then none of these solutions will function property.

James,

Thanks for your thoughts...

Just for clarification...I am using VMWare Fusion...and not Bootcamp...so I would certainly think that FSEvents should catching changes to the VM package file...

I will look at the system logs and see if I see any errors...and also take a look at FSEventer...

Thanks again...

Gary

Gary K. Griffey wrote:and also take a look at FSEventer

For the record, FSEventer is donationware (not entirely free), and you can download it from here: http://fernlightning.com/doku.php?id=software:fseventer:start

I'm playing with it now, and it's pretty cool! Please donate if you find it useful.

James,

Just a quick update...I see no errors for FSEvents daemon on the laptop...

One interesting observation...the VM has been running on the laptop since about 07:20 this morning (about 4 hours)...and...although the VM package file (aka...folder) has a Date Modified of 09/09/2010 07:21....inside the package...the virtual machine's disk file (.VMDK) has a date modified of the current time...which is what you would expect...as the disk is being updated constantly...

I just wonder if QRecall is actually looking "inside" the VM package when it does its timestamp compare on an incremental..because it would seem that if it simply used the package's timestamp....it would bypass backing it up...

GKG

Gary K. Griffey wrote:I just wonder if QRecall is actually looking "inside" the VM package when it does its timestamp compare on an incremental..

Absolutely. A package is just a directory (folder) that acts as a single item in the Finder. The modification date of a directory does not change when the contents of a file it contains changes. So the mod date of the directory is not used as a criteria for deciding if the files within that directory have changed.

QRecall will record any changes to the modification date of the directory, and then proceed to examine every file inside the directory for changes?unless, file system events ensures QRecall that nothing in that folder has changed, in which case QRecall will skip it.

You need to determine if file system events are being recorded for your VM package. Start FSEventer, start up your VM, shut it down, and then go see if a change event for your VM package was recorded. If not, then that's your problem.

Ok...I will give it a try....

James,


I am using the FSEventer utility program that you mentioned...and certainly seeing the entire virtual machine package and the VM disk file get a "Content Modified" event fired...whenever the VM is started or shutdown...and, therefore, I would think that it would always be included in an incremental backup if it was started/stopped after the previous backup....so, I am still not sure what is happening.

One other question...if I open a Finder "Smart Folder" window...and mask on "Last Modified Date"...the VM package folders are correctly flagged as being updated...now I realize that a Smart Folder uses Spotlight...but isn't Spotlight "fed" by the same FSEvent daemon as QRecall? And, therefore, shouldn't the results be the same, i.e., if it shows up in a Smart Folder "Last Modified" mask shouldn't it be captured with QRecall?

Thanks,

GKG

Gary K. Griffey wrote:but isn't Spotlight "fed" by the same FSEvent daemon as QRecall? And, therefore, shouldn't the results be the same, i.e., if it shows up in a Smart Folder "Last Modified" mask shouldn't it be captured with QRecall?

Yes and no. As I mentioned earlier, FSEventer, the Finder, Spotlight, and others subscribe to the "live" feed of file system change events. You change something; a message is immediately sent to those processes.

QRecall, Time Machines, and similar application request historic file system change events. These are collected by a background process (fseventsd) and periodically written to the hidden folder .fseventsd at the root level of each volume. At some later date, a process can request the history of change events from some arbitrary point in time, and they receive a "replay" of those events.

You already mentioned that you after restarting, QRecall didn't capture your VM package. The possibilities at this point seem to be narrowed down to (a) your historic FSEvent event records aren't being saved properly on your volume or (b) QRecall is misinterpreting those events.

At this point, I think the most expedient diagnostic approach would be to prepare a special version of QRecall that logs exactly what it gets from the FSEvents API. That way, you can try restarting and running QRecall and see what folders FSEvents has recorded. Fixing this might be as simple as deleting the hidden .fseventsd folder and letting the operating system rebuild it, but I want to rule out QRecall as the culprit first.

James,

Thanks for the update...I wonder what the time delay is between fseventsd receiving an event and actually performing the write to the hidden file...

I can also certainly try your suggestion of simply deleting the fseventsd hidden file on the laptop's volume and allowing it to be re-created from scratch...I don't think that would hurt anything...

If you do have time to work-up a special version that would log exactly what QRecall "sees" in that hidden file it would be great...I would just love to get to the bottom of this...

Thanks again...

GKG

Gary K. Griffey wrote:I wonder what the time delay is between fseventsd receiving an event and actually performing the write to the hidden file...

File system events are coalesced approximately every 30 seconds. The coalesced version is what gets saved as history. When these actually get written to the drive will largely depend on buffering, etc., but I'm sure the process flushes them from time-to-time. You can always fire up iosnoop if you really want to know...

I can also certainly try your suggestion of simply deleting the fseventsd hidden file on the laptop's volume and allowing it to be re-created from scratch...I don't think that would hurt anything...

Ideally, the the most definitive course of action would be to leave it as it is until you can demonstrate the problem. If the problem is that file system events are not being saved correctly, and you can prove that, you can then delete the .fseventsd folder and see if that fixes the problem. If you "fix" the problem before you isolate it, you'll be left wondering what was really wrong.

If you do have time to work-up a special version that would log exactly what QRecall "sees" in that hidden file it would be great...I would just love to get to the bottom of this...

I'll get to that later today.

Understood...

I will not make any changes or delete the fseventsd hidden file...

Thanks...

Gary (and anyone else who wants to play),

Download QRecall 1.2(a)9. Install it.

From the Terminal, you'll need to set both the QRLogCaptureDecisions and QRLogFSEvents (undocumented) settings to true:

defaults write com.qrecall.client QRLogCaptureDecisions -boolean true

defaults write com.qrecall.client QRLogFSEvents -boolean true

With both of these settings on, QRecall will log (a) all of the historic file system event paths reported by the operating system, (b) the final tree of changed directories (this is QRecall's internal picture of what has changed on the volume based on the FS events it received), and (c) which folders in the capture were skipped because file system events ensured QRecall that nothing within those folders changed.

Once that's all done, I suggest that you:

Perform a capture of the volume containing your VM image.

Start and stop your VM.

Perform the same capture again.

Start and stop your VM.

Restart your computer.

Perform the same capture again.

Send a diagnostic report (Help > Send Report...)

That should pretty definitively determine whether QRecall is either not getting the correct change events or if it's misinterpreting them.

Thanks James...

I will install it and follow the steps you have outlined...

Thanks...

GKG

James,

Ok...followed your list and here is what happened:

1) Capture performed at 06:04...virtual machine was captured as expected.
2) Opened VM...closed VM...to update VM disk file, etc.
3) Shutdown/Restarted laptop
3) Ran second capture at 06:15...this attempt did NOT capture VM...even though the VM package directory has an update time of 06:13...
4) Your new logging clearly indicates that the ~/Documents/Virtual Machine directory was not flagged as updated for the second capture attempt..and, therefore, QRecall did not back it up.

I also sent the report as well. So...it appears that OSX and FSEvenetsd are at fault here...your thoughts?

Also, any suggestions or thoughts as to why FSEventsd is not working? It seems that if this was a common issue there would be a lot of "long faces" out there come restore time.

Thanks again for the special version.

GKG

James,

I have just repeated the same steps on another macBook Pro laptop that I have...using a VM, etc...same exact result....i.e., on the second Capture...after the VM Startup/Shutdown and the system shutdown/restart...the virtual machine directory is not flagged as being updated even though Finder clearly indicates that it has been updated. So...QRecall bypasses the VM...

So...I do not believe that this is an issue on a specific laptop...like the hidden FSEevents file is corrupted, etc...this seems to be easy to recreate on any system...

Very scary...for QRecall, Time Machine,etc...

I just don't see how Finder could "know" about the directory update but evidently not FSEventsd...could the VMWare Fusion application somehow be suppressing this from FSEventsd? Sounds odd...

Gary