That's a good question.

The QRecall Activity window is independent of the QRecall application. So, technically, it doesn't belong to the QRecall application and isn't affected by its window order. Hiding or quitting the QRecall application won't hide or close the QRecall Activity window either.

Having said that, I'll look into the possibility of hooking the "Bring All to Front" command and bring the QRecall Activity window to the front at the same time.

You can always bring the activity monitor to the front by using the Window > Activity Monitor > QRecall Activity (Command+0) command.

All good suggestions. I personally don't like most manuals in PDF format unless they are intended to be read like a book; the Photoshop manual comes to mind. But other people feel differently. I probably won't do something like this in the near future because that means maintaining two different versions of the documentation, which would be counterproductive at this point.

The next time I see David I'll ask him.

Steven M. Alper wrote:Are Merges and Rolling Merges exclusively for manipulation of the archive itself? There's no capturing going on?

That's correct. A merge action only combines layers that have already been captured. Captures and merges are complements of each other: A capture adds a new layer, a merge removes old layers.

So in other words, I would likely need three actions per archive?:
1. A recurring capture
2. A recurring merge or rolling merge of the resulting archive
3. A recurring compact of the archive

And if you're paranoid (like I am), you'll want to schedule a recurring verify.

Note that I'm contemplating adding an assistant to QRecall that would help you set these up the first time.

[P.S. - I hate the way (at least your installed version of) JForum ignores the BB Code on first submission (actually displays the code) until you edit and resubmit.]

So do I. I'm working to see if I can upgrade to a version that has some of these problems fixed. I find I have better luck when I uncheck the "Disable HTML in this message" option.

Steven M. Alper wrote:I keep re-reading the Rolling Merge Action page, but seem to fall short in understanding exactly how the "Followed by:" merging works and what effects my choices will have.

Rolling Merge is really a simple idea, but it has been a struggle to explain it clearly. Let me explain the logic that QRecall uses and see if that makes it any clearer. Then I'll try to answer your other questions.

Each "tier" in the rolling merge defines a set of time periods, counting backwards from midnight today. Let's say you defined a rolling merge that has no recent layers, 5 day layers, 2 week layers, and 1 month layer. When that rolling merge is run on Friday May 4th, 2007, it would find the following time periods:



- 0 recent layers: nothing
- 5 day layers: One period each for April 29th, 30th, May 1st, 2nd, and 3rd.
- 2 week layers: One period each for the week of April 22nd and the week of April 15th.
- 1 month layer: All of the days in March, plus a second period for April 1st through the 14th; this represents the fractional month between the end of the last month layer and the beginning of the first week layer.
- Remaining layers: Everything before March 1st.

All of the layers in each time period is then merged into a single layer. If this archive only had one layer per day, then the first 5 day layers would do nothing as those days already have only one layer.
All of the layers between April 22nd-28th would merged into a single layer. (week layer)
Layers between April 15th-21st would be merged together. (week layer)
Layers between April 1st and the 14th would be merged. (partial month layer)
Layers between March 1st and the 31st would be merged together. (month layer)
Everything between the first layer and February 28th would be merged. (remaining layer)

If I said Keep most recent 30 days and left it alone, what happens to the 1st day once I've reached 31?

Assuming that you are created one layer each day, the 1st and 2nd layers would be merged together, leaving you with the 30 most recent layers.

Said more precisely, any layers captured in the past 30 days would be left alone. Any layers captured more than 30 days ago would be merged into a single layer.

If I say keep most recent 30 days, followed by 30 day layers, is that not the same as simply keeping the most recent 60 days?

If you create one layer a day, they are equivalent.

As a counter example, I have QRecall capture the source to the project files every 20 minutes while I work. At the end of the day there might be 20 new layers in the archive. After working on QRecall another 29 days I might have 600 new layers. (Yikes, that's a lot of layers!)

If the rolling merge was set to keep the most recent 30 days followed by 30 day layers, it would leave all 600 layers as they are because they were all captured in the past 30 days.
If the rolling merge was set to 0 recent days and 60 day layers, the 20-odd layers created each day would be merged into a single layer, reducing those 600 layers into 30.

If I say keep most recent 30 days, followed by 3 week layers, then:
- The 1st day gets bumped into the first week layer
- Once the first week layer has 7 "bumped" days, then, what, the first of those 7 days gets bumped to the second week layer? Or the whole first week layer gets bumped to the second week and the first week layer now contains only the next bumped day?

A complete week layer is a single layer. What happens the next day is that there is a gap -- a fractional time period -- between the most recent week and the end of the 30 day. Over the next 6 days all of the layers in that gap are merged into a single layer. At the end of the week, that single layer becomes the most recent week. At the same time, the oldest week falls off the end (rolls) into the first monthly period, or the factional portion of the month, and is absorbed there. The next day the process starts all over again.

If I set up like this, what happens?

Most recent 30 days
3 week layers
2 month layers
1 year layer

Whatever layers you capture in the past 30 days are left alone. Layers in the recent period are never merged.

All of the layers between the 30 day period and the next three full weeks will be merged into 3 or 4 layers. One for each full week, and one for any fractional week between the last week and the beginning of the 30 day period.

Moving backwards from the earliest week, QRecall finds the next two full months and merges all of those layers into 2 or 3 layers (again, two full months plus a partial month if there is one).

It then moves backward and finds all the layers captured during the previous calendar year and merges those into one layer. A second layer might be created if the earliest month layer wasn't January.

All of the layers created before Jan 1st of last year are then merged into a single layer.

If I don't do anything beyond "Most recent 30 days," the bumped days get merged with the original?

Yes. The layers in the "keep" period would be left alone and anything beyond that will fall into the single "remaining" layer.

Sorry, it seems clear until I try to think it through. Perhaps the help page could use some specific examples to help understand what the real effect of choices in the action will be?

It does seem simple -- until you start to worry about the details. Personally, I rather people not worry about the details and simply focus on goal: What resolution and duration of incremental backups do you need?

If, two months from now, it is important to be able recover both the version of a document from Thursday May 3rd and Friday May 4th, then you'd probably want daily layers going back 90 or 120 days.

On the other hand, if you only care about getting back files in the past few days as well as that vacation video you deleted last November to clear up some disk space, then keeping the past 7 days plus 12 month layers is probably more in line with your needs.

I keep struggling to find a simple way of explaining this. I'm considering adding some more detailed examples to the help file. My reluctance is that I try to keep the help short and to the point, and explanations like this get rather tedious.

The missing file messages shouldn't be any cause for alarm, unless you think the files are important. If you are using your computer while a capture is in progress this is inevitable. Files are created and deleted constantly, and QRecall may prepare to capture a file then find that it's been delete before it can read it. These events usually aren't interesting, but I feel obligated to record them in the log none the less.

The QRecall application wasn't designed to run as root. (In general, the GUI isn't designed to run as root and Apple frowns on it.) You may run into some odd behavior.

Steven M. Alper wrote:Reading through the help files, I found the following:

Thanks Steve, that's a great help. These corrections will appear in the next release.

Steve,

No problem -- I'm just glad thinks are working now.

Beta identity keys are permanent. The beta application itself will expire, but your identity key won't.

The only thing that should cause you to lose your identity key is if you lost the preference file/folder where your identity key is stored. The self-uninstall procedure (Command+Option+Shift+Q) should not delete your key, but performing the manual uninstall instructions found in the advanced topic of the help will. If you performed the manual uninstall in an attempt to diagnose your earlier problem, that would definitely erase your key. Identity keys are also user specific, so running QRecall from another user account or from a different boot volume will require that you reenter a key. Note that if you are running QRecall from multiple accounts on the same machine you can enter the same identity key for all of those users, or use a different identity keys to keep the data captured by each user separate.

And I agree that the identity key entry screen is confusing. Apple added "default text" to text entry fields recently and I used that to display a sample identity key. However, Apple's method of displaying the default text (i.e. the value of the field if you don't enter anything) is to draw it dimmed. I've had two individuals think that there was already an identity key entered, and two more now that thought the field was disabled.

I'll probably replace this with a prompt or simply leave the field blank in the next version.

Thanks for the feedback.

Here's a tip:

If you are recapturing large disk images, you'll get much better performance if you turn off (lowest setting) shifted-quanta analysis in the archive's settings. This will greatly improve recapture performance.

Disk images are an array of immovable blocks. Data doesn't "shift" in a disk image the way it can in a file. So searching for shifted data wastes a great deal of resources.

I'm not sure exactly what a VM file is, but if VM files are like large multi-media projects or disk images, then they are exactly the kind of incremental backup problem that QRecall was designed to deal with.

When the file is recaptured, QRecall will read each block (quanta) of data and compare it with those that have already been captured. Only new data blocks are added to the archive.

In the situation of a disk image, take a 600MB disk image file. If you mount that image and add a single 1MB file, only a little over 1MB worth of data has changed. When QRecall recaptures the file, it will discover 599MB of duplicate data and only adds 1MB of new data to the archive.

There's a similar situation with large multimedia files like Photoshop or audio files. Open up a 300MB Photoshop file and add an adjustment layer. The original 300MB of data is still in the file, although it might have shifted to a slightly new location. That's where QRecall's shifted-quanta analysis gets to work. It still finds the 300MB of existing data and only adds the new data to the archive. Ditto for changing the MP3 tags on a large audio file. The audio hasn't changed, only the tiny bit of tag data.

Now this still requires QRecall to re-analyze all of that data, which will be time consuming. So this is definitely something that you'll want to schedule to run at night or when you're not using your system.

(And I'm glad you like the answers.)

john hampson wrote:I rescanned my 15GB archive, and it found/added/changed 588 items from 92,457 in around 3minutes. That was pretty impressive! Also the system was still pretty responsive while it was happening!

Good! That's the way it is supposed to work.

Incremental recaptures are designed to be very fast, as this is the action that will occur the most often. And I have some optimizations to make it even faster in the future.

I can't legally comment on Leopard features because I'm an Apple developer and have signed a non-disclosure agreement.

However, from the information that Apple has made public about Time Machine I can make two comparisons.

First, QRecall is vastly more efficient than Time Machine (and most other backup applications in general). Time Machine simply copies files, while QRecall analyses the data in files and eliminates any duplicate data from the backup archive. So your backup storage is guaranteed to be larger than the original set of files with Time Machine, while with QRecall your backup archive plus incrementals will more than likely be smaller. This means that you can use a significantly smaller backup drive or keep incremental data longer, or both.

QRecall is significantly more configurable than Time Machine.

Personally, I was both delighted and dismayed when Apple announced Time Machine. Delighted, because Apple recognized many of the same problems with existing backup software that QRecall is trying to overcome. I think it will also encourage more people to implement a disk-to-disk backup plans.

At the same time it's disheartening to see Apple integrate many of the features that we've been developing over the past two year then build it into the OS and give it away for free.

But QRecall still has a distinct set of features that I think will provide some competition to Time Machine.

john hampson wrote:I seemed to have 600MB of inactive memory, so the memory wasn't an issue.

This is a common misconception. Inactive memory is not unused memory. It's just memory that hasn't been accessed recently (read "few seconds").

The speed would go up to 400MB/min for smaller files, but seemed to slow considerably for larger files. I would have expected the opposite.

I've been a longtime user of Chronosync - it seems to "play better" with other applications.

This is because of the fundamental difference between QRecall and other backup applications. QRecall analyses every block of data in a file to determine if any of that data has already been captured (in that file or any other file). It also searches for duplicate data that has shifted to a different location in the file.

By contrast, applications like Chronosync, SuperDuper, Retrospect, et. al. merely look for files that have changed and copy those files. The larger the file, the less work they have to do.

For QRecall, the larger the file the more work is has to do.

This is an unfortunate consequence of the amount of work that QRecall is doing. The QRecallHelper is that task that's doing the lion's share of the work. It uses a lot of RAM and can saturate the I/O at the same time.

If you are trying to use other applications at the same time, the problem is probably virtual memory swapping. While QRecall is working it pushes segments of code and data from other applications out of memory and into the swap files. When those applications need that code or data they have to swap it back in. This is made more difficult because the disk drive is already busy copying file data to the archive, so those swap requests have to wait.

You can verify this by watching the page in/out numbers in the Activity monitor. Sluggish behavior will be accompanied by ever rising page in/out numbers.

You can reduce the amount of work that QRecall does by lowering the Shifted Quanta detection setting for the archive. (Setting at it lowest setting essentially turns it off). But this won't reduce the burden on I/O. In fact, it will tend to increase I/O.

I have plans to improve the performance of QRecall in several places. This requires restructuring some of the code to take better advantage of multiple processor systems and reduce the amount of memory it uses. Both of these should improve how QRecall plays with others.

Unfortunately, these changes require significant modifications to the core capture routines -- changes which must be made very carefully. So I don't have plans to make these performance improvements in the immediate future. But if performance becomes a persistent problem for people, I can re-prioritize that work.

I live for pedantic.

john hampson wrote:should say "capture is performed".

Thanks. That correction will show up in the next release.

Oops!

Yes, if version 1.0.0b37 is your first version of QRecall, you are going to have this problem. The logic for creating the support folders changed recently and was never tested on any brand new users.

I've fixed the problem and have released 1.0.0b38. Choose QRecall > Check for Updates... to get the corrected version.