Charles,

I'm not sure what's wrong, but your system is not behaving correctly and I don't know why.

When you install QRecall to run actions while you're logged out, it installs the QRecallScheduler as a system-wide user agent in /Library/LaunchAgents/com.qrecall.scheduler.plist (OS X 10.5 only). The scheduler is configured to run in the "background" so it should get launched whenever you log in or start up, and should then run forever.

But that's not what's happening. Something is killing your scheduler when you log out:

2009-08-08 15:15:07.096 +0200 #debug# user logout

2009-08-08 15:15:11.517 +0200 #debug# received SIGTERM <-- something killed the scheduler process

2009-08-08 15:15:12.045 +0200 #debug# stopping Scheduler-1.1.2-Jul 21 2009

This shouldn't be happening. I tested this on three other 10.5.8 systems here and none of them stop the schedule when the user logs out.

Because the schedule is configured to run as a Background daemon, the system doesn't automatically restart it again when you log back in (as it does the QRecallMonitor).

At this point I'm a little baffled. The OS should not be killing user agents that are configured to run in the background (at least not until you shutdown).

My only guess is that something else is sending kill signals to your user processes when you log out. Once the scheduler is stopped, it's never started again—until you launch the QRecall application, it notices that the scheduler is dead, and starts it manually.

My only suggestion is that you track down what might be killing the scheduler or change the option to run the scheduler only while you're logged in. This would install the scheduler as a user agent, just like the QRecallMonitor, which seems to be working.

Charles,

Thanks for sending the diagnostic report. The problem is that your scheduler isn't, in fact, running. Or, more precisely, it's not running when it's supposed to, which is all the time.

Your log shows that your scheduler is getting stopped when you log out, but not restarted when you log in. When you launch QRecall, it sees that the scheduler isn't running and starts it. That's what's causing your actions to start.

Looking deeper into the log, it appears that QRecall is trying to install the schedule as a daemon (so that it runs while you're logged out), but your configuration assumes that runs only while you're logged in. So I suspect QRecall's configuration and/or installation are confused:

Try this:

Open your QRecall preferences, go to the Authorization tab, change the option that says "Start and run actions while logged out", wait a few seconds, and change it again. This will cause QRecall to uninstall and reinstall the scheduler (twice). I suspect that you want this option off as Mac OS X doesn't mount volumes while you're logged out, so trying to run actions while logged out doesn't do any good.

Now, log out and back in (or restart) and send me another diagnostic report. I'll be able to tell if the scheduler gets restarted when you log back in.

Thanks,

James

Charles,

It might be a event bug, the scheduler wasn't starting correctly, or the action just isn't showing up in the activity window.

The easiest way to begin diagnosing this would be to send a diagnostic report (Help > Send Report...). When you send the report, please note the approximate date/time that the the volume was mounted and the action didn't run.

Christian Roth wrote:I thought the compact action was atomic and once force-cancelled would either leave the archive in inconsistent state (requiring a re-index or repair) or have not achieved anything.

Stopping a compact won't reduce its overall size, but in the interim space in the archive is being consolidated by packing data blocks together. Think of sliding beads on a string; you can't cut the string and make it shorter until you move all the beads to one end, but you can move some beads, stop, and come back later to move more beads, until you're done.

Christian Roth wrote:I once did a compact action on that large archive and it took (I think) about 30 hours.

See my note about stopping and restarting compact actions. Compacting should never get in the way of being employed.

Christian Roth wrote:When I combine this with the possible reduction in size of the archive, I am now willing to give that strategy a try.

If you're going to go for it, I'll make a couple of points:

Turning on compression only compresses new data until you perform a compact.

An automatic compact won't recompress anything until the archive has at least 4% of empty space, but you can force that by starting a compact manually from the QRecall application.

Recompressing a terabyte of data over a slow network connection might take weeks. Fortunately, compacts are incremental and you stop them and restart them later. I'd suggest setting up a compact action the runs every night and set it to stop after running for 4 or 5 hours. Also set the QRCaptureFreeSpaceSweep option, at least temporarily. Eventually the compaction will work its way through your entire archive and it will be completely compressed.

In setting up the above, you might also consider leaving the capture level compression off and setting the compact compression high. Compression will only be performed during the compact action, improving the performance of new-data captures.

Turning off compression later will never uncompress any of the compressed data in the archive. So you can't undo this if it doesn't work out.

OK, one more suggestion.

This is a bit radical, but you might consider it. I do something similar with my system here (for completely different reasons, but that's another story).

Set up two archives on your NAS: A compressed archive for long term backups, and an uncompressed one for short term backups.

Capture to the short term archive every day, or even multiple times a day. Set up a rolling merge to keep only the last 14 days worth of layers.

Capture to the long term archive once a week.

If you're keeping archive data for a long period of time, the sheer bulk of your existing archive is probably the biggest drain on your performance. By keeping a short term archive that never grows beyond 100GB and a dozen layers (or so), you'll see significant improvements in incremental captures.

Just an idea...

Christian,

Having said that about the performance of compression, you might consider changing your QRCaptureFreeSpaceSweep setting. This will cause the archive to grow more quickly when capturing, but does improve the performance of the capture. You'll want to schedule an occasional compact action to recover the free space now being ignored by the captures.

Christian,

I can imagine that there might be a situation where the access to the archive volume was so slow (maybe 802.11b?) that compression would improve performance, but the difference between processing power and I/O speed would have be to extreme.

In practice, adding compression invariably reduces performance. I did a few tests this morning just to confirm (since I haven't benchmarked compression in awhile). I set up a capture from a brand-new eight-core Xeon Mac Pro networked to a MacMini (PPC) with an external Firewire drive. The transfer rate between the Mac Pro and the Mini tops out at about 10 MB/sec. Not terribly slow, but certainly not fast.

I captured a reasonable number (9,000) of fairly compressible files (mostly word processing documents and uncompressed TIFF images used for publication). The difference between using an uncompressed and a highly compressed archive were:

Uncompressed archive: 1.7GB
Compressed archive: 687MB (about a 60% reduction in size)

Initial capture: 04:35 (uncompressed), 06:52 (compressed) - 50% slower
Recapture (100% duplicate): 03:32 (uncompressed), 02:10 (compressed) - 80% faster

So in the case where all the data was being recaptured, a really fast computer was able to take advantage of the slightly smaller data to outperform the uncompressed archive. But in almost all other situations, the amount of work required to compress and uncompress the data far outweighs the slight advantage of a smaller data transfer. If you average these figures (assume that most captures are a combination of new and duplicate data), the performance is about the same. And if you work with data the resists compression (video, audio, most images), then there will be no advantage at all.

In conclusion, you're not likely to see any increase in performance by increasing your compression level, and you could see significant decreases. The bottom line is that compression saves space, not time.

Gabriele Caniglia wrote:- how do I exclude files from capture whose name is/contains/starts_with 'something'?
Say I want to exclude ".DS_Store" files or "Thumbs.db"?

Filtering items by name or other criteria is an oft requested feature that hasn't been added yet—but it's on the to-do list.

- how do I add invisibile files and folders to the filters list? I have workarounded this by opening a Disk Broswer window in BBEdit, then dragging and dropping the invisible items onto the filters section. Is there a better way to accomplish this?

That's an excellent solution. I'll add a request to see invisible files in the picker dialog to the wish list.

- how can I schedule multiple captures at specific times of the day? Say I want to capture at 9:30am and 4pm from Mon to Fri. I can only see an "Interval" option or a "Daily" option...

Christian Roth answered this one already. You can define as many actions as you want for a single archive, each with their own schedule and criteria. For example, I capture my development folder every 20 minutes, my home folder every 2 hours, and my entire startup drive once a day.

Chris Caouette wrote:any eta for an update?

A few more hours.

I've been struggling with a persistent performance problem with the package and names index. They perform just fine on small archives, but their processing overhead grew exponentially relative to the size of an archive.

For months I tried to find a feasible solution, but eventually gave up on the code. In late June I rewrote the entire package and names indexing code from scratch. I've been testing it for the past couple of months, fixing bugs, and refining performance.

The fruits of that labor have been running here for about a three weeks now. If the rest of the testing goes well, I'll push out a release towards the end of this week.

This has been a huge roadblock to adding new features, which I hope to get back to very soon.

Charles Watts-Jones wrote:I seem to recall that QRecall should alert the user to errors through a QRecall Activity Monitor message which is displayed until dismissed by the user. This does not seem to be so on my machine and I wonder if it's a problem with my installation or something more general.

In general, QRecall tries to be as annoying as possible when an error occurs.

However, there are some limitations which might be affecting you.

The warning indicator appears in the QRecall Activity window when an action fails to finish properly. If the activity monitor process isn't running, most likely because you're logged out at the time, then there's nothing to catch and display the warning. Also, if you use spaces the warning may be appearing in another space.

Is there a way to ensure that QRecall advises errors automatically? If not, might I add it to the wish list?

QRecall also works with Growl, which has many notification options—but again only while you're logged in.

I'll add a wish list item to be notified of problems that occur while you are logged out.

Steven Arnold wrote:To followup on this issue, I have now had an archive on a regular disk -- not a DMG, encrypted or otherwise -- since June 18th. That's about three weeks. Normally the problem would have come up by now, so I would conclude that QRecall has a problem with large archives on DMGs. Note that the problem could be an artifact of encryption, or it could just be an issue with DMGs.

Thanks for following up on this.

If you need some help writing a script to do this, I can probably knock something out in Ruby pretty fast to create the original files to be backed up, and then add a few files at random locations.

Thanks for the offer, but I've got plenty of QRecall test cases already set up. I'll put this on my list of things to test and see if I can't get it to happen here.

Steve Mayer wrote:Where are these stored so that I don't have to set these up again?

Actions are stored in the Actions folder inside ~/Library/Preferences/QRecall.

If you just copy those action files back into the Actions folder, the scheduler won't "see" the change until the next time it restarts or you make a change to the schedule. The easiest way to accomplish the latter is to create any new action (it doesn't have to do anything), save it, and then delete it. This will cause the scheduler to rescan all of the action files and update its schedule.

Bruce Giles wrote:The virtual machine file is currently about 60 GB in size, and is actually a package. The virtual disk itself is broken up into numerous chunks, each no larger than 2 GB in size. (That's one of the options for how Fusion segments and stores its virtual disk.)

This is a popular trend in large disk images, primarily so that file-oriented backup and synchronization applications don't have to copy the entire image every time.

So, I'm wondering how QRecall deals with this.

QRecall deals with this like any other set of files. It recaptures any of segment files that have changed since the last capture, analyzing each one for changed data.

Even though QRecall is backing up only the parts of the virtual disk files that change, it still takes significant time to back them up, and they're continuing to change during the backup. If I ever have to use QRecall to restore the virtual machine file from a backup, is it (the VM file) even likely to work? Or do I end up with a virtual disk inside the VM file in which the various chunks are no longer consistent with each other, because they were backed up at different times during the archive session?

The fundamental problem here is that the files are changing during the capture. The fact that there are multiple files is immaterial; the problem is essentially the same whether there were one file or a hundred.

It doesn't matter what backup/copy/synchronization utility you use, the inescapable problem is that the contents of the file are an incomplete representation of the current state of the application. This is also true for database files, dynamically modified document files, and so on. But it's particularly true for disk images, since the directory structure of the disk will more than likely be inconsistent with what's buffered in memory.

The same problem exists when you capture the OS X operating system. It's possible because most of the files that are constantly being modified aren't that significant. These include log, cache, scratch and temporary files along with memory mapped files. Failing to capture all of the data that should be in those files doesn't prevent your system from restarting.

I suspect that the same would be true for the VMWare disk image. Most of the files (the OS, applications, documents, ...) would be stable and are dependably captured. The one transient data structure that's most likely to give you problems is the directory structure itself. After recalling an older version of the VMWare disk image, I would immediately repair the volume before proceeding. If you have the disk space, I think it would be a good experiment to try this a couple of times using various captured images.

Would I be better off to not back up the VM file until the virtual machine is not running?

That's the only real solution—and that's independent of what backup solution you're using.