QRecall 1.2 beta program

Dawn to Dusk Software is pleased to announce the beginning of the QRecall 1.2 beta test program.

To get started, go to the QRecall Download page.

If you already have a permanent or trial identity key, you can continue to use that. If don't have a permanent key, or your trial key has expired, you can obtain a free Beta Identity Key that will be valid for the entire beta test period.

The theme of QRecall 1.2 is "performance and interface." The performance part involves a lot of under-the-hood changes and the bulk of that work is complete. Later beta versions will begin to incorporate a new interface, but I wanted to get the mission-critical code changes done first so they can get as much testing time as possible.

As always, feedback is welcome and encouraged.

Having installed on a late 2007 iMac (iMac7,1) running 10.6.3 with 4 GB RAM, I turned off scheduled wake-up in Energy Saver and asked QRecall to wake and then sleep machine on completing its tasks. Worked flawlessly - great! I didn't however noticed a faster Verify (1:07:12 for a 237.6 GB archive) than I got with QRecall 1.1.4. Don't know if I should have expected this(?).

Charles Watts-Jones wrote:Having installed on a late 2007 iMac (iMac7,1) running 10.6.3 with 4 GB RAM, I turned off scheduled wake-up in Energy Saver and asked QRecall to wake and then sleep machine on completing its tasks. Worked flawlessly - great!

Excellent news!

I didn't however noticed a faster Verify (1:07:12 for a 237.6 GB archive) than I got with QRecall 1.1.4. Don't know if I should have expected this(?).

Verify was highly optimized a couple of versions back, and will be I/O bound for most users. In other words, QRecall can verify the archive as fast as it can be read from the drive. So until you get a faster hard drive/interface, verify won't get any faster.

The principle improvements in QRecall 1.2 are more efficient use of the quanta and packages indexes, expanded RAM usage (thanks to 64-bit addressing), and multi-processor load balancing. These changes improve capture performance the most, and impact compact and merge actions to a lessor degree.

James Bucanek wrote:The principle improvements in QRecall 1.2 are more efficient use of the quanta and packages indexes, expanded RAM usage (thanks to 64-bit addressing), and multi-processor load balancing. These changes improve capture performance the most, and impact compact and merge actions to a lessor degree.

Had a quick check in the log. Version 1.1.4 captured 1.12 GB (75% duplicate) in 12:53 while the beta managed 1.15 GB (73% duplicate) in 09:56. Looking good.

Are v1.2 archives compatible with earlier versions, or does the index changes preclude that? What I'm really asking is whether or not I could install 1.2 on my MBP and leave 1.1.4 on my wife's G4 iMac, both of which back up to the same archive?

Ralph

Ralph Strauch wrote:Are v1.2 archives compatible with earlier versions

No. Writing to an archive using version 1.2 will mean that earlier versions will not be able to use the archive.

For what it's worth, the core data format has not changed. If you use QRecall 1.2b1 to modify an archive and later want to revert back to using 1.1.x, reindexing the archive with 1.1 will make it usable again.

I've run a first capture to an archive lying on a network volume at an AEBS. Before 1.2 the capture took something like 16 hours. Now it ran through in just about 3 hours.

-- Current Stable --
Captured 8668 items, 19,8 GB (29% duplicate)
Capture finished (16:03:19)

-- Current Beta --
Captured 11914 items, 24,5 GB (24% duplicate)
Capture finished (3:04:10)

A more than 5-fold speedup!!!
Is this actually expected?

I did actually use the machine while the capture ran and I didn't see much slow downs or something like that. And by the way: Yes this was a full capture with ignoring of the filesystem change history.

So I guess this beta is quite promising! Good Work so far!

ciao,
Jochen

Jochen Schmidt wrote:A more than 5-fold speedup!!!
Is this actually expected?

Performance improvements for individuals is really hard to estimate. It all depends on your circumstances and configuration.

For most users, I expect mostest performance improvements. But in specific combinations of OS, archive size, RAM size, CPU type, and I/O configuration, the speed increase can be dramatic. I've created test cases here where the beta runs 40x faster than the previous version, capturing a test folder in 40 minutes where version 1.1.4 took over a day.

James,

I just wanted to provide my feedback with respect to performance using the new Beta 1.2 of QRecall.

I have a nightly scheduled QRecall backup that was consistently running in excess of 8 hours to an external Firewire drive...that same backup...using the new Beta 1.2...is now running in less than 55 minutes...a huge 8X performance increase.

This is really outstanding...your product has always worked very well for me...it has saved my "backside" on numerous occasions...but with this new update...it is truly an amazing utility.

As always...I have recommended your application to all of my mac clients...thanks again for all of your dedication!!

GKG

Gary,

Thanks for the feedback. It's always good to see real-world numbers—especially when those numbers show improvement.

Hi James,

During the past week...I had a chance to test some additional captures using the beta 1.2 version. I seem to have hit on an issue that I am hoping you can shed some light on.

Last Wednesday...I restarted one of my MacBook Pro's in target disk mode...and performed a QRecall capture of the entire laptop's hard drive via Firewire to another MacBook Pro. The initial capture went smoothly....no issues.

I continued using the first laptop normally since last Wednesday...including a VMWare Fusion virtual machine. This morning...I performed the exact same procedure...I restarted the laptop in target disk mode...attached it to my second laptop using a FireWire cable...and attempted a QRecall recapture of the entire drive...the recapture finished in about 10 seconds...unfortunately, QRecall did not recognize the fact that the VMWare virtual machine had changed over the past 3 days...and so it did not choose to recapture it. The modify timestamp on the VMWare virtual machine package is today's date...and thus...I would think that QRecall should have certainly included it in its recapture operation.

Any ideas?

Thanks...

(This should probably should be a new topic, but JForum won't let me split an existing thread....)

Gary K. Griffey wrote:This morning...I performed the exact same procedure...I restarted the laptop in target disk mode...attached it to my second laptop using a FireWire cable...and attempted a QRecall recapture of the entire drive...the recapture finished in about 10 seconds...

I suspect that you've been bitten by the file system events (a.k.a. FSEvents) service. I'll quote from the Advanced QRecall Settings page:

Leopard's folder change detection is not foolproof. There are a number of obscure situations where the file system will not accurately report the changes on a volume.

One situation where it isn't foolproof is (drumroll) moving a drive between different systems—especially if those systems aren't running the exact same version of the OS.

What happens is that the volume's file system change log gets reset, and when QRecall queries the volume for changes it comes back with nothing (or very little), and QRecall skips capturing items that have, in fact, changed since they were last captured. You can verify this by looking in your QRecall log. Open the QRecall log window and slide the details control all the way to the right. In the log messages for the capture you'll find something like this:

Locating changes since Tuesday, August 3, 2010 7:30 PM

Collected 117,752 folder changes

If the number of changed folders was zero or very small, then the operating system has lost the history of changes for that volume.

QRecall knows that the file system change log isn't reliable and will periodically ignore it. Unfortunately, the default period is about a week:

To guard against this, QRecall only trusts the operating system for a limited amount of time. After that (approximately 7 days) the capture will ignore the system and perform a deep, exhaustive, scan of the entire directory structure looking for changes. Once the deep scan is complete, QRecall will again trust the operating system's change detection for another 7 days.

You can work around this problem be reducing the amount of time QRecall trusts a volume, or disabling the feature altogether, by setting the QRAuditFileSystemHistoryDays advanced settings. Put your MBPro into target disk mode, plug it into your other system, open a Terminal window, and issue the command:

defaults write com.qrecall.client QRAuditFileSystemHistoryDays -float 0.0

This will completely disable QRecall's use of the file system change events log. It will, instead, check every file and folder for changes on every capture. Note that this can significantly increase the amount of work QRecall does on each capture.

Capture your MBPro. QRecall should find and capture all changes on the volume. When you're done, restore the default be deleting your custom setting. Again in Terminal, issue the command:

defaults delete com.qrecall.client QRAuditFileSystemHistoryDays

In the future, you have the option of repeating these steps each time you capture your MBPro or you could leave QRecall's "trust" period set to something much smaller than its default value of 6.9 days. For example, setting it to 0.9 would mean that a second capture that's more than 22 hours after the previous capture would trigger QRecall to perform an exhaustive scan for changes.

James...as always...thanks for the quick reply....

I proceeded with the steps that you outlined...and sure enough...the files were captured correctly with the subsequent attempt.

This does, however, raise some rather disturbing questions in my mind...most notably...is this happening with other QRecall captures that I rely upon...and am I simply not aware of it? Both of the MacBooks in this case are running 10.6.4...with no outstanding updates...so it would appear that OS version incompatibility is not the issue at least in my case.

I understand that this is an Apple issue...not a QRecall issue...but it is really scary to think about how many folks could be missing deltas in their QRecall archives.

Thanks again for your time...your product rocks!

Gary K. Griffey wrote:This does, however, raise some rather disturbing questions in my mind...most notably...is this happening with other QRecall captures that I rely upon...and am I simply not aware of it?

It's entirely possible. If you are regularly moving source volumes from one system to another, then any software that relies on file system events to detect changes on those volumes should be treated with suspicion. This issue will also impact users who maintain multiple instances of the operating system (often for testing) and who regularly reboot their system from alternate volumes.

These are, however, uncommon scenarios so it isn't a problem for most users. Most of the volumes/devices uses to store documents—the kind of volumes that you would capture to an archive using QRecall—do not get passed around from one system to another. For volumes that are occasionally shuffled between systems, the 7 day setting of QRAuditFileSystemHistoryDays insures that they will get captured correctly sometime in the next few days.

The volumes that are regularly shared between systems are more likely be used to backup to (rather than from). The volume containing the archive is immune to this issue.

The best I can recommend is to be aware of the limitations of OS X's file system events log; if it's a problem for QRecall, use the QRAuditFileSystemHistoryDays setting to limit its impact or ignore it altogether.

James,

Thanks for your valuable insight...I agree...when volumes are unmounted, moved, mounted to another system, etc., there can be issues with file system events and their accuracy.

I know that I have also encountered issues in the past with Time Machine losing track of various file/folder updates if certain system volume actions were performed. This would appear to fall in the same category.

As long as this is known...and QRecall is configurable using the simple line commands that you provided...I think my concerns are no where near as "dire" as I thought. I have never encountered a problem recalling a file, folder or even an entire volume with QRecall if the captures were made on the same source machine, i.e., without using TDM, etc.

Possibly, in a future release...QRecall could provide the ability to override the use of the file system events via preferences...but now that I have the terminal commands...I am happy.

Thanks again...

GKG