Frederic Thomas wrote:Pretty much 22GB of useless junk.

Far from it. According to the logs, QRecall found approximately 2 MB of damaged data. The other 99.992% of 12.4GB is just fine.

Following a repair, damaged and suspect files will be marked as such. If you follow the repair by a new capture, all damaged files will be recaptured. You can then merge the new layer with the damaged one and restore the archive to pristine condition.

What's odd is how scattered the damage is. Can I ask you a few questions about this archive?

Is this archive new or have you been using it for some time? If it's old, approximately what version of QRecall was used to create it?

What were the actions (if any) leading up to the one that failed?

Where is the archive stored? Are there other files/archives on this volume and are they OK?

When you repaired the archive, did you tell QRecall to recover incomplete files?

It shouldn't have any problems. A virtual machine is just another file as far as QRecall is concerned.

But that's not your problem. It appears that your archive has become corrupted. QRecall found an invalid checksum in the archive's data file.

First, run disk utility on your volume to make sure you don't have any cross-linked files or directory problems. Then try repairing the archive and see what happens.

After some experimentation, I find that I can't reproduce this problem. On every system I have to test on, QRecall's normal disk and network activity is enough to prevent the system from sleeping.

Never the less, I've added code that will "tickle" the power manager at regular intervals while an action is running. This should prevent a system from going to sleep (it should not prevent it from dimming or sleeping the display).

This change will appear in the next version. If you're dying to try it out, let me know and I'll make an interim release.

That's exactly what happened.

Here's the interesting log entries:

This is where your computer woke up.
2007-08-19 01:53:48.572 +0200 #debug# -[QuantumScheduler powerDidChangeNotification:] PDPowerManagementPoweredOn

Then immediately afterwards
2007-08-19 01:53:59.038 +0200 Failure Could not capture (filename)
2007-08-19 01:53:59.039 +0200 Details archive I/O error
2007-08-19 01:53:59.039 +0200 Details Data exception
2007-08-19 01:53:59.039 +0200 Details Cause: <IO> failed to write envelope content { Class=DataFile, Length=32776, API=FSWriteFork, Pos=5591783004, File=/Volumes/Leopard/krapul.quanta/repository.data, OSErr=-50 }
OS Error -50 is an I/O error. QRecall can no longer write to the archive. Since this is a network volume, I surmise that either the connection was interrupted during the sleep, or the server went to sleep and/or dropped the connection.

And finally:
2007-08-19 01:53:59.049 +0200 Capture failed
2007-08-19 01:53:59.049 +0200 A storage or disk error occurred while writing to the archive data. The archive is probably damaged.

Stephen,

Thanks for the great feedback and suggestions. However, some of these ideas are at odds with QRecall's basic design and philosophy.

QRecall was designed to capture everything that's important, quickly and efficiently, and allow you either recover earlier versions or restore any item to the exact condition it was in when captured.

The filtering mechanism was designed solely to avoid capturing superfluous or redundant files (cache, spotlight indexes, backup copies, etc.). It was not designed to micro-manage an archive, arbitrarily capturing this file but not that one.

Excluding important items from a capture creates a confusing middle ground that I'm not comfortable with. In your iTunes example, podcasts are excluded but you clearly don't want them erased. My question is "Which is it? Are the files important or not?" It would make sense to exclude them if you could easily re-download the lost podcasts. It would not make sense if you could not.

Taken to it's logical end, not capturing items and then not erasing those items during a restore would result in a mish-mash of interleaved versions. Simple examples would be backup files that are newer than the working version. Replacing some files but not others in an iTunes library will definitely cause iTunes to become confused. Or let's say you decided to reduce the size of your archive by excluding all foreign language resources from application bundles. Restoring an application package without deleting those excluded items would result in a Frankenstein mixture of old and new resource files. This could cause application instability or worse. If you did this to the operating system, the results would be disastrous.

Being be bit of a propeller-head myself, I'm all for giving sharp knives to those who want them. I am considering adding an "Overlay" command. This would be a restore that doesn't delete any existing files that do not appear in the archive; it would only recall deleted files and replace newer files with the version from the archive.

Combined with arbitrary rules for excluding items, you could (at your own risk) overlay a set of captured items on top of an existing set of files. (Note that you can accomplish this right now by recalling the items to a separate location, then merging the two directories with a little command-line work.)

I think we should always know exactly what is being backed up, and will be restored, and both these issues affect that.

I agree—which is exactly why I don't like this feature.

Frederic Thomas wrote:Using the new QRecall on a new network archive, I have tried twice to back up a 80+ GB MacBook Pro but each time the machine goes to sleep, and then, when waking up, all hell breaks loose for QRecall.

QRecall will happily survive through sleep cycles, whether actions are running or not.

I'd be interested in seeing the log file entries that occur when the machine is woken up again, but I suspect I know what's going on...

Incomplete backup, damaged archive (reindexing fails), ...

My guess is that the computer is losing it's connection with the volume that contains the archive. Normally (at least on my systems), putting a computer to sleep does not disconnect mounted volumes. (On the other hand, putting the system that is hosting the volume to sleep will certainly cause them to disconnect ungracefully.)

Regardless of what's causing it, there is no way for QRecall to recover from this situation any more than it could if you unplugged a firewire drive in the middle of a capture.

The archive can't be reindexed because the data is incomplete. (Reindex simply rebuilds the index files, it won't fix incomplete data). You could Repair the archive. Repair will recover as much data as possibly by creating a recovery layer that contains the recovered and incomplete items.

I still have "power management features" on my to-do list. One of these would be to prevent the computer from going to sleep while an action was running.

Deborah McMillion-Nering wrote:SHould the reindex be this slow?

No. There's an issue with the Reindex command in 1.0.0(46). One of those situations where trying to make things faster actually slows it down.

There are two competing processes (one reading the data and the other updating the index) that cause your hard drive to thrash, dramatically slowing down the reindex process.

You can avoid this problem by first opening the archive package (select the archive icon and choose Show Package Contents from the Right/Control+Click menu). Delete the negative.index file. Now reindex the archive.

A fix for this problem will appear in the next release.

Version 1.0.0(46) will now bring the activity window to the front (assuming it's open) when you choose Window > Bring All to Front in the QRecall application.

Version 1.0.0(46) now ignore users with invalid home directory paths.

This should eliminate the repeated errors you are seeing when constructing capture filters for that user.

Version 1.0.0(46) will now auto-mount volumes using an alias.

See the release notes for details.

Once you upgrade to 1.0.0(46) you will need to update any existing actions to use an alias record by opening the action, re-choosing the same archive, and then saving the action.

Steven M. Alper wrote:You mean like Apple's Address Book's alarm: Repeat in 1 minute; 5 mines, 30 minutes, 1 hour, 1 day, 1 week... etc.? That would be fine, although I think it should be a separate button on the activity d.b. so that you don't have to go through another screen if you really do want to completely cancel at this point.

I was considering two interfaces: If the action was canceled, an additional "Reschedule..." button would appear in the confirmation dialog. Click it and the "Run At..." dialog would appear.

Alternatively, I'm considering adding these kinds of "power user" features to some contextual pop-up menus, rather than cluttering up the core interface. If you right/command+clicked on a running action, "Cancel and Reschedule..." would be one of the options.

Of course, the other way around, if the user left the pause "hanging indefinitely" they'd always have the activity d.b. floating around to remind them.

Assuming they haven't closed it.

Steven M. Alper wrote:1. Could QR play more fairly with the other children? Share the CPUs in a less monopolistic fashion?

That's the job of the operating system. QRecall just does its thing. It's up to the kernel to balance the load amongst multiple processes.

That said, there are number of ways in which QRecall can be less of a burden on system resources. Significant ones have already been done, which you'll see in the next release. Others I'm still experimenting with.

2. Could QR's activity monitor gain a "Pause" button, obviating the need to remember to launch QR and manually start a canceled action?

A "pause" button is interesting, but I'm worried the process could be left hanging indefinitely. What would you think about a "reschedule" button that would appear if you cancel an action?

Steven M. Alper wrote:Although no other app was able to handle the situation, is there perhaps a way to design QR to exit more gracefully in such a situation?

Sadly, no. QRecall (like every other application) is at the mercy of the operating system. If an application makes a system call and that call fails to return, there's nothing—short of killing the process—that can be done about it.

You took the only course of action available.

On my MacBook Pro I do notice that the hard drive is significantly slower than in my desktop machines. A few applications that fly on my G5 seem to crawl on my MBP, even though my MPB has more (CPU) horsepower and just as much RAM. When I put my ear to the MBP, I can hear the drive just trashing about.

I haven't really noticed any slowdown with QRecall because I run my captures at night.

Most performance degradation is due to swapping. The amount of RAM you have can have much more impact on performance than CPU or even drive speed.