Currently, each action is saved as an individual .qraction file in <your home folder>/Library/Preferences/QRecall/Actions.

Backup that folder and you've saved all of your actions.

You can restore actions by placing .qraction files in this folder, but you have to get the scheduler rescan the folder before they will appear in QRecall. The simplest way is to restart the computer or the QRecallScheduler process. If you don't want to do that, open QRecall and do anything that adds or removes an action, such as duplicating an existing action and then deleting it. Whenever actions are added or removed, the scheduler re-reads all of the action documents it finds there.

Cody,

I think everything you're interested in will be in 1.3, which will focus heavily on filtering and automation.

Cody Frisch wrote:I think it would be nice to see the "actions" list available through a menubar item, easily suspend, stop, reschedule, and pause.

That's an excellent suggestion. I'm adding to the wish list now.

Also I'd love to see "grouping" in the actions list. Mostly for organization purposes since it shouldn't matter beyond that really. Just group by archive, though obviously once can sort now, it would be nice to have everything for one archive then sorted by capture, merge, compact, etc.

I've had similar requests to this in the past, but I'm loath to add complexity unless it also adds functionality.

As for grouping actions, most installations of QRecall have, at most, six to eight actions. I'd consider some kind of grouping if it solved some specific problem, but right now my thinking is "how lost can you get with eight items?"

Would something like the ability to sub-sort on other columns be sufficient? That is, sort first by archive and then by action (or next run time, or schedule, ...).

I'd also love to see "end actions". When one action completes it automatically calls another action. Rather than relying on the scheduler having a different time.

Similarly, if you could describe what you're trying to accomplish that the current interface can't do, I'll be more than eager to investigate a new interface or feature.

Paul,

Thanks for the suggestion and the feed back. I'm not entirely sure that what you're asking for is technically feasible, but I'll add it to the wish list.

I realize that there need to be other means of accessing the data in an archive, and this is a priority for future versions.

Gary K. Griffey wrote:It seems that the directory that contains the VM package files is not being updated when a virtual machine inside of it has been updated...

I might have explained this earlier, if not ...

Modifying a file in a folder (directory) does not change the modification data of the directory. The modification date of a directory indicates that the contents of that directory "file" has changed. (In UNIX, a directory is conceptually a file containing metadata about other files.) If you create, delete, or rename an item in a directory this changes the logical contents of the directory "file", which in turn updates it's modification date.

Merely changing the contents of a file within that directory does not change the directory, and thus does not change the directory's modification date. It only changes the modification date of the file that changed.

There are a variety of reasons it works this way, some abstract (UNIX's "everything's a file" philosophy) and some practical (the less that has to change the faster changes can be made).

Now, I'm sure you're wondering why sometimes when you save a file the modification date of the enclosing folder changes. The reason is that many applications use a technique called a "safe save" to write documents. The contents of the document are written to some temporary, often invisible, location on the drive. When the document has been completely saved, the original document is deleted and the new document is moved into its place. This avoids the possibility of destroying the only good copy you had of the document if the save operation failed to complete successfully. The acts of deleting and renaming the document files are a directory changes, which naturally update the modification date of the enclosing folder.

Really huge files, like VM images and QRecall archives, can't be updated in this manor; it's just not practical. Updates to those kinds of files only change the data within the file and don't touch the enclosing folder. Now you may notice that the last modified date of a QRecall archive changes when it's updated, and that's because QRecall goes out of its way to update (touch) the enclosing package folder when its done. This is so that both you and Spotlight see that the contents of the archive have changed. However, software is not obligated to do this and a folder's modification date can remain unchanged for months, even though its contents are changing constantly.

If you're curious, you can see this in the Finder too. Create a folder and save a small text file (e.g. TestFile.txt) inside that folder. In the Terminal, use a command like "echo 'hi' >> TestFile.txt" to append some new data to that file. The modification date of the file will change, but the modification date of the folder will not.

Gary K. Griffey wrote:I guess as far as running incremental captures with QRAuditFileSystemHistoryDays = 0.0 is concerned...my concern is that there could be other folders/files that are also being inadvertently "missed" by FSEventsd...it just seems odd to me that a virtual machine folder is the only one that could experience this issue...although I have no proof, of course, that other folders are not being backed-up...

As I've mentioned, virtual machines do not interact with the file system the way other software does. It's been my experience that file system events is reliable, which is why I was initially very reluctant to implicate it as the source of your problems. If it weren't I'd have a lot of complaints from other QRecall users, and thousands of Time Machine users would be tearing up Apple's support forums; neither of which have happened.

Certainly, the overhead for performing the full deep system scan each time can be huge...no doubt...possibly, if you believe this to be isolated to virtual machine folders only...

It's not horrendous, but it will add 10-20 minutes to each capture.

I will take your advice and split this into 2 backup actions...one for the entire volume that specifically excludes the single VM directory...and another that only includes the VM directory...since you stated that the named target directory itself is always "deep scanned"...and only sub-directories employ the FSEventsd logic.

Don't exclude the VM directory from the full capture. Excluding an item treats the item as if it didn't exist, which will create a layer where your VM folder doesn't exist at all. Keep your current action that captures the entire volume, then create a second action that captures just the VM folder. Schedule the new action to run immediately after the first.

And please keep me posted on any future developments...

Gary K. Griffey wrote:I then cycled the VM...rebooted....and ran a Capture. The Capture used the "deep traversal"...even though QRAuditFileSystemHistoryDays was set to the default value....I guess because .fsevenetsd was recreated?

That's correct. When QRecall requests historic file system event information for a date earlier than what's in the database, it receives a message that basically says "there's no history for that period, you're on your own," so QRecall has to do it the hard way.

I then, however, cycled the VM again...restarted the computer...and ran another capture...this one missed the VM...so it still does not work correctly. I am attaching the .fseventsd-suspect folder as a single zip file.

Thanks. I'll also ask that you send another diagnostic report (Help > Send Report) so I can report the actual events it did receive and what was expected.

I hope this provides the info you need to supply Apple with a bug report.

I'll report this to Apple, and I thank you for your patience and perseverance in isolating this problem. I would also suggest that you bring this to the attention of VMWare, or I'll be happy to do so on your behalf. I strongly suspect that this has something to do with VMWare. As I've mentioned, software that hosts a virtual operating system do not interact with OS X the way normal software does, which may result in side-effects like the one you're seeing.

I guess for now...I will have to run all my captures using QRAuditFileSystemHistoryDays = 0.0...just to be sure everything is backed-up.

Have you tried some of the other workarounds (shutdown script, separate capture)? I'd be really surprised if this was a fundamental flaw in file system events. I've been working intimately with the FSEvents API since Leopard and this is the first time I've ever seen it miss or forget a change event (at least outside the reasons that are known to trip it up).

Gary K. Griffey wrote:Ok...followed your list

Almost. I also wanted you to perform a capture, open your VM, close it, and immediately capture the drive again without restarting; and then do the same again with a restart in between stopping the VM and capturing. (And, of course, send a diagnostic report when you're done.)

You stated that when running FSEventer you saw change event being logged for your VM image package. I wanted to see if you could catch OS X reporting a change event for something that happened very recently, but then not report something that should have been saved between restarts.

By the way, what is the path to your VM file—the one that should be in the events history?

So...it appears that OSX and FSEvenetsd are at fault here...your thoughts?

It would appear so, which is disappointing. I can fix QRecall, I can't fix OS X...

It seems that if this was a common issue there would be a lot of "long faces" out there come restore time.

This is the first instance that I've ever seen where file system events didn't record a change event.

I just don't see how Finder could "know" about the directory update but evidently not FSEventsd...

Like I said, there are two sources of file system change events: The live event stream that is broadcast to procsses as events occur and the historic events recorded on the volume. The Finder subscribes to the live stream. QRecall, Time Machine, and so on request historic change events. (Actually, that's not entirely true; Both Time Machine and QRecall request historic events, and then switch to the live stream during the backup so they can catch additional changes as they happen.)

But the point is, just because file system events detects and broadcast a change event doesn't (apparently) guarantee that it's also getting saved and replayed in the future.

Also, any suggestions or thoughts as to why FSEventsd is not working?

I'd like to stick with the theory that the fseventsd process is receiving change events, but failing to record them permanently for some reason. So after running the experiment that I mentioned above, try this experiment:

In the Terminal, issue this commands:
$ sudo mv /.fseventsd /fseventsd-suspect
(you'll be prompted for your password)
$ sudo chown -R $UID /fseventsd-suspect

Restart your computer.

Now perform the "capture, run VM, restart, capture" sequence again and see if it works any differently.

If that works, then there may have been something weird with the records stored in your .fseventsd folder. Please archive the fseventsd-suspect folder you'll find at the root of your hard drive and e-mail that to me. I'd like to include it my bug report to Apple.

could the VMWare Fusion application somehow be suppressing this from FSEventsd?

There's no official API for doing this. An application can flush (erase) all of the history on a volume and there's a means for disabling file system events on a volume, but there's no API for selectively deleting or suppressing change events for an individual folder.

Apple does provide an API for excluding individual items from a Time Machine backup, but that shouldn't interfere with file system events. If it is, then that's a bug.

But as I mentioned very early on, it might have something to do with the nature of the VMWare system. Virtual machines interface with the operating system at a very low level. It's entirely possible that VMWare is making changes to the file system in such a way that the operating system isn't seeing it, or doesn't believe that it's significant enough to record for posterity. It's also possible that VMWare doesn't like Time Machine trying to copy its VM image and has figured out how to "hack" the file system events service into not recording specific changes.

Out of curiosity, does the VMWare system provide any means of automatically running a script, ideally when the VM shutsdown? It might be possible to work around this problem by adding a script that touches the VM image whenever you shut down.

Another thing you might try is create a second capture action the captures just the VM Image package. Assuming the package doesn't have subfolders, QRecall always captures the items that are in its capture set. It only consults the file system events to determine if subfolders need to be captured. In essence, this would ignore file system events for the one VM image folder.

Gary (and anyone else who wants to play),

Download QRecall 1.2(a)9. Install it.

From the Terminal, you'll need to set both the QRLogCaptureDecisions and QRLogFSEvents (undocumented) settings to true:

defaults write com.qrecall.client QRLogCaptureDecisions -boolean true

defaults write com.qrecall.client QRLogFSEvents -boolean true

With both of these settings on, QRecall will log (a) all of the historic file system event paths reported by the operating system, (b) the final tree of changed directories (this is QRecall's internal picture of what has changed on the volume based on the FS events it received), and (c) which folders in the capture were skipped because file system events ensured QRecall that nothing within those folders changed.

Once that's all done, I suggest that you:

Perform a capture of the volume containing your VM image.

Start and stop your VM.

Perform the same capture again.

Start and stop your VM.

Restart your computer.

Perform the same capture again.

Send a diagnostic report (Help > Send Report...)

That should pretty definitively determine whether QRecall is either not getting the correct change events or if it's misinterpreting them.

Gary K. Griffey wrote:I wonder what the time delay is between fseventsd receiving an event and actually performing the write to the hidden file...

File system events are coalesced approximately every 30 seconds. The coalesced version is what gets saved as history. When these actually get written to the drive will largely depend on buffering, etc., but I'm sure the process flushes them from time-to-time. You can always fire up iosnoop if you really want to know...

I can also certainly try your suggestion of simply deleting the fseventsd hidden file on the laptop's volume and allowing it to be re-created from scratch...I don't think that would hurt anything...

Ideally, the the most definitive course of action would be to leave it as it is until you can demonstrate the problem. If the problem is that file system events are not being saved correctly, and you can prove that, you can then delete the .fseventsd folder and see if that fixes the problem. If you "fix" the problem before you isolate it, you'll be left wondering what was really wrong.

If you do have time to work-up a special version that would log exactly what QRecall "sees" in that hidden file it would be great...I would just love to get to the bottom of this...

I'll get to that later today.

Gary K. Griffey wrote:but isn't Spotlight "fed" by the same FSEvent daemon as QRecall? And, therefore, shouldn't the results be the same, i.e., if it shows up in a Smart Folder "Last Modified" mask shouldn't it be captured with QRecall?

Yes and no. As I mentioned earlier, FSEventer, the Finder, Spotlight, and others subscribe to the "live" feed of file system change events. You change something; a message is immediately sent to those processes.

QRecall, Time Machines, and similar application request historic file system change events. These are collected by a background process (fseventsd) and periodically written to the hidden folder .fseventsd at the root level of each volume. At some later date, a process can request the history of change events from some arbitrary point in time, and they receive a "replay" of those events.

You already mentioned that you after restarting, QRecall didn't capture your VM package. The possibilities at this point seem to be narrowed down to (a) your historic FSEvent event records aren't being saved properly on your volume or (b) QRecall is misinterpreting those events.

At this point, I think the most expedient diagnostic approach would be to prepare a special version of QRecall that logs exactly what it gets from the FSEvents API. That way, you can try restarting and running QRecall and see what folders FSEvents has recorded. Fixing this might be as simple as deleting the hidden .fseventsd folder and letting the operating system rebuild it, but I want to rule out QRecall as the culprit first.

Gary K. Griffey wrote:I just wonder if QRecall is actually looking "inside" the VM package when it does its timestamp compare on an incremental..

Absolutely. A package is just a directory (folder) that acts as a single item in the Finder. The modification date of a directory does not change when the contents of a file it contains changes. So the mod date of the directory is not used as a criteria for deciding if the files within that directory have changed.

QRecall will record any changes to the modification date of the directory, and then proceed to examine every file inside the directory for changes?unless, file system events ensures QRecall that nothing in that folder has changed, in which case QRecall will skip it.

You need to determine if file system events are being recorded for your VM package. Start FSEventer, start up your VM, shut it down, and then go see if a change event for your VM package was recorded. If not, then that's your problem.

Gary K. Griffey wrote:and also take a look at FSEventer

For the record, FSEventer is donationware (not entirely free), and you can download it from here: http://fernlightning.com/doku.php?id=software:fseventer:start

I'm playing with it now, and it's pretty cool! Please donate if you find it useful.

Gary K. Griffey wrote:So...it would appear that the 10.6 File System events did not advise QRecall to backup the Vm package..even though it most certainly had been updated. From your previous notes...I do not see any reason the file system events on the laptop's drive should have been deleted or otherwise compromised. What am I doing wrong?

You're not doing anything wrong. If QRecall is not scanning the directory containing your VM image, then it's because file system events is not recording changes for it. I have a couple of thoughts...

First, check your console for anything related to FSEvent or fseventd. In the past, people have reported errors being logged by the fseventd daemon. This may indicated that file system events are not being recorded property.

How do you run your Windows system? Do you reboot your computer into Windows (a la Bootcamp) and the restart again in OS X, or do you run something like Parallels so that OS X and Windows are running concurrently? It occurred to me that in the former situation, OS X isn't actually running, so changes to VM files (even if they're recorded in the file system) won't be recorded by the file system events service.

If you're running the later configuration, you can see if file system events records changes for your VM package using a free utility like FSEventer (insert Google search here). Your VM might be going behind OS X's back (so to speak) and reading and writing blocks at the driver level. This would also circumvent the file system events service.

A tool like FSEventer still won't show you exactly what QRecall sees because it only displays file system events as they occur, whereas QRecall requests historic events (i.e. please Mr. OS, send me all change events from point X in the past). So where FSeventer logs events as they occur, QRecall is interested in the permanent record of events stored on the drive.

You mentioned that these file system events are located in a hidden file on the drive itself...

That's they way it's supposed to work. But if FSEventer shows change events for your VM folder, you reboot, and then QRecall doesn't see them, then the FSEvents service might not be recording them property.

obviously, you are using an API or equivalent to read these values...it there any way that I could actually peer into this hidden file to see if the VM package file is being flagged as changed?

I'm not aware of any freeware tool for requesting historic FSEvent data. I wrote one of my own for testing. If you can't figure this out, I'll see if I can dig it up and polish it off.

I just wonder what other user/system related files are being skipped in my incremental backups...makes you question the value of any backup system using these events..like Time Machine, etc.

That's a good question. QRecall, Time Machine, Apple Events, and most other disk backup and synchronization software all rely on FSEvents. If it's not reporting changes consistently, then none of these solutions will function property.

Glenn Henshaw wrote:The backup target (/Volumes/MBP/) is a drive mounted on an Airport Express.

When you attach it directly, first run Disk Utility and verify/repair the volume. If there's any problems found, repair the QRecall archive.

Also, restart the Airport Express before reattaching the drive.