Greetings beta testers,

I'm trying to track down a bug in QRecall and I'd like your help. Here's the scenario that I'm looking for:

(1) You do something with QRecall 1.2 beta (capture, compact, merge, ...)
(2) You verify the archive and get an error that looks like this:


If you encounter this, please send a diagnostic report (Help > Send Report...) and then reindex your archive.

What's happening is that something, at some time, is deleting one or two data records from the archive but leaving the corresponding index entry for those records in the quanta hash index. In this example, the hash is supposed to have 48,064,742 entries (one for each data record), but instead has 48,064,744. This means that there are two extra entries in the index.

This is harmless problem from the standpoint of data integrity, since no archive data is missing or lost. But it is an inconsistency that I'm keen to nail down and eliminate.

The reason that I'm asking for help is that I've yet to duplicate the situation under which the problem occurs. Once it's happened, the information about what these phantom index entries referred to is already lost. I need to be able to analyze an archive, make the problem happen, and analyze the archive again. To do that, I need to figure out the sequence of events that causes it to occur. I'm hoping that if anyone else is encountering this problem that a review of their log files will reveal a pattern.

James Bucanek wrote:So to efficiently perform multiple actions simultaneously, you should (a) have lots of RAM and CPU cores, ...

To comment on my own post ...

You have 8GB of RAM and 2 CPUs with hyperthreading. This puts your system in the midrange of performance.

You can probably run two QRecall actions simultaneously with good to excellent performance, as long as those actions aren't thrashing a single drive.

QRecall can perform any number of actions simultaneously, with some logical and practical limits:

The logical limit is that a QRecall archive can only be modified by one action at a time. For example, if you schedule multiple actions to act on the same archive, they will be performed one at a time.

QRecall can capture the same source item to more than one archive simultaneously.

By extension, it can capture multiple source items to multiple archives simultaneously.

Normally, QRecall (at least the current Beta) will try to avoid doing this because it often isn't very efficient. There's a hard disk phenomenon known as "thrashing" that can significantly reduce the efficiency of data I/O. A single process reading or writing to single physical drive is usually extremely efficient, reading or writing data at pretty close to the drive's maximum performance. However, if you start two processes reading or writing data to a single physical drive, the two compete with each other. They cause the drive to constantly jump back and forth between the two files—the drive spends more time seeking than it does reading/writing data. For example, verifying two archives on a volume might take an hour (30 mins each) if done sequentially. Try to verify them simultaneously, and it could 3 or 4 hours, to say nothing about the wear and tear on the drive. Reading/write to two volumes (partitions) of the same physical drive or RAID set is particularly bad.

Another factor is memory and CPU resource. QRecall uses a lot of memory and is highly optimized to use multiple CPU cores. Unless you have tons of RAM and lots of CPU cores, QRecall will quickly saturate your system. Too many heavy processes causes memory "thrashing", which can also degrade the performance of your system.

So to efficiently perform multiple actions simultaneously, you should (a) have lots of RAM and CPU cores, (b) be reading from volumes that are on physically separate hard drives, and (c) be writing to archives that are on physically separate hard drives.

The current beta release of QRecall has a number of settings to help you avoid thrashing and to run QRecall actions one at a time, because it's generally more efficient. These settings can be found in the Scheduler tab of the QRecall preferences. See the beta release notes for a complete description. Turn these limits down or off as you see fit.

Rex Carlson wrote:Hello.

Hello,

I guess you didn't get my direct reply via e-mail.

My old intenal HD died; I just installed a new one. Also installed OSX 10.5 with original discs.
Now what? Exactly how do I get everything from the archive back into my computer just like it was. If I just grab the cube from my external drive to the new internal drive, I'll just end up with the same cube on my laptop. I need everything back the way it was, not a cube just sitting there that I have to access every time via QR.

What you want to do is restore your entire hard drive from the QRecall archive. This assumes that you set up QRecall to capture the entire volume, and not just pieces (restoring pieces of a drive won't usually result in a bootable volume).

Here's what you need to do:

1. Install 10.5 on your new laptop. You've already done that, so that's cool.
2. Install QRecall. You don't need an identity key to recall/restore.
3. Open your archive. Make sure the Owners and Volumes drawer is visible (see the View menu).
4. Select the captured volume from your old hard drive.
5. While holding down the option key, select Archive > Restore To...
6. Choose your new hard drive from the list.
7. Authorize QRecall to use administrative privileges when restoring the volume (this is a must to restore the OS).

QRecall will now overwrite your entire hard drive with the contents of your old drive, exactly as it was when you last captured it. Since QRecall is overwriting your operating system, applications, and just about everything else, I suggest that you do not attempt to run any applications or do anything with the system until the restore is finished. When it does finish, immediately restart your computer.

Are there unlimited free QR upgrades?

Yes. All versions of QRecall are compatible with any permanent identity key that you've been issued. That's why they're called permanent keys.

I have no plans to alter this arrangement for the immediate future. If some future version of QRecall won't work with older identity keys, you'll be given plenty of warning. I recommend configuring QRecall to automatically check for updates, and updating QRecall whenever new versions become available.

Nicholas Sloan wrote:My question is, could I have continued capturing to the new copy of the quanta with the same settings.; i.e. would QR have equated the new Work and Archive with the old Work and Archive?

Probably not. QRecall uses Mac OS aliases to keep track of an action's archive and capture folders. Aliases are good about recognizing when you move or rename an item, but copying something to a new volume will usually treat it as something completely unrelated to the original.

And if not, could I have changed the settings to point QR in the same direction, but continued capturing to the same quanta in such a way that it would simply have continued recording changes?

Yes. If you ever relocate or change an item (archive or capture item) in such a way that the action can't find it anymore, simply edit the action to restore the association: Open the action and re-choose the archive, or remove and re-add the capture item(s). The next time the action runs it will use the updated archive or capture item and pick up where it left off.

Where the archive is, or where you move it to, won't affect how QRecall works. Source items, however, that move to a new volume will cause QRecall to recapture all of the items ("Current work" in your case) in that folder. That's because they reside on a different volume than the original and QRecall will treat them as a completely separate set of files. The result will be a new volume in your archive with a fresh copy of every item in the folder.

But since all of the items in the new folder are copies of the old folder, QRecall will see that all of the file data is duplicated and won't add any new data to the archive. You'll simply have a new layer of duplicate items.

Also, would it in theory be possible to merge the old quanta with the new one?

That ability is on my short list of new features.

Finally, how's progress with the faster, smarter, all-singing-and-dancing version 2?

It's already faster and getting smarter. I'm finishing up an iPhone app this week and will be back on the all-singing-and-dancing part (i.e. new user interface) this week.

Ryan Sandridge wrote:Just to be clear, I wasn't suggesting using WebDAV (unless I'm confused about what WebDAV is). Amazon S3 uses Representational State Transfer via the HTTP protocol (REST). Perhaps the distinction is irrelevant for the purposes of considering adding cloud backup to QRecall, I don't know.

The distinction is largely irrelevant. I only bring up WebDAV because it's been discussed at length on the forum already. All of these protocols are layered on top of HTTP, which just makes them a bad fit for working with QRecall—without some extensive modifications.

Anyway, it is something to kick around.

I'm definitely kicking it around. I'd very much like to provide an off-site or remote backup solution compatible with standard remote file services like WebDAV or S3. I just haven't worked out the details.

Welcome, Ryan

Encryption is on the short list of features for the next major release.

I don't have plans to turn any part of QRecall into open source. Open source is a wonderful thing, but it doesn't completely protect you against obsolescence. My response to your concerns is this: Test QRecall and satisfy yourself that it can reliably recall your files. Should QRecall stop working with some future version of OS X, you can always recall your files and migrate them to another backup solution.

What to do about WebDAV based services like S3 is another matter. On one hand, WebDAV backups are great because they're simple and easy to access. On the other hand, they are typically expensive (in terms of cost, time, limited size, and network bandwidth) and would really benefit from QRecall's compression and efficiency.

However, the way QRecall works and the way WebDAV works are like oil and water. What I'd like to offer is a way of efficiently "cascading" an archive from a local disk to a WebDAV volume, so that your principle backup would be local and QRecall would periodically migrate incremental updates to the WebDAV volume. I'm not even close to working out the details, but that's the direction I'd like to go in.

In the nearer term, I do plan to create a command-line version of QRecall. This would be more flexible for those with scripting knowledge, and would also be less prone to breaking when new versions of the operating system come out.

I know this doesn't address all of your wishes, but I hope it clarifies the direction QRecall is headed.

What brought this issue to the fore for me was finding my G5 awake in the mornings when it should have gone to sleep as normal after a SuperDuper backup scheduled for 3am. This seemed to be becoming a habit that only started around the time I updated QR to 1.2b, and I found that if I paused QR actions manually the evening before,it would not happen. There would be no QR progress windows showing, and I can't be certain that QR was implicated, but it looks like a probability.

Nicholas,

I still don't know what's going on here. I did get a chance to review the diagnostic report you sent me, and there's a possible coorrilation between your computer waking up around 3AM and QRecall, but I can't reproduce it here.

You do have a set of QRecall actions scheduled to run starting at 3:10. So if you have a SuperDuper copy scheduled to run around 3:00, QRecall will definitely fire up soon after. You don't have any power management requests associated with these actions, so I don't see why QRecall (by itself) would be causing your computer's monitor to wake up. I've created test cases here, using the same version of Mac OS X, where I allow the monitor to go to sleep followed by a scheduled QRecall action and the monitor never wakes up. There could be some interaction between QRecall and SuperDuper, but I can't imagine what.

Anyway, you could simply reschedule those actions to run at some other time and see what happens. If the QRecall actions continue to wake up your computer, I can prepare a version of QRecall that doesn't try to keep your system from going to sleep and see if that changes its behavior.

Ralph,

Everything you posted is spot-on correct. I couldn't have phrased it better myself.

If you select one of the older volumes and delete your System or Applications folder, it will only discard the files and data from the layers containing that volume.

Ralph Strauch wrote:Another advantage to mounting the archive as a volume would be that it would provide a column display, making it much easier to browse the archive.

Alternate views (column and icon) are the next addition to QRecall. I hope to have these in the beta by late June.

If you did that, would be mounted volume also allow browsing of layers, as the current display does?

No. The finder and OS will see it a standard volume. There are a number of ways of approaching this, and I haven't made any decisions yet. If I can manage it, it would be nice to see if I can "trick" the OS into treating the volume as if it were a Time Machine volume. If that were possible, you could use Time Machine's interface with a QRecall archive.

Otherwise, I'll either have to simulate a Time-Machine-like folder organization, where each layer is a top-level folder in the volume, or something else clever. There's still a lot of details to work out.

BTW, I've just found that backing up my new drive -- with the same name as the old one -- has created a new volume in the owners and volumes list.

Correct. Every "new" volume is treated as a unique entity by QRecall. "New" in this context means different. If you reformat or resize a volume, QRecall treats is a separate volume. You can delete the old volume from your archive (select the volume in the Owner and Volumes drawer and choose Delete Item). This will delete all of the archive information about the old volume. You might want to wait a few months before doing that. I also plan to add a command that will let you merge two volumes, making migration a little smoother.

Nicholas Sloan wrote:Is there any reason to suppose that things might be different on PPC/10.5.8 (which I am running now) from Intel/10.6.2 (which I expect to be running next week)?

I wouldn't expect so, unless there are differences in how these two operating systems behave that I'm not aware of.

I'm out of town this week. I'll review your diagnostic report when I return. In the mean time, please note any differences your see when you move to Snow Leopard, and feel free to submit another diagnostic report after trying it out on your new computer.

Ralph,

That's a brilliant idea!

I had been toying with the idea of writing a driver so that QRecall archives could appear as regular volume on the desktop. I had considered this idea somewhat of a novelty and hadn't really invested much time in it.

Exposing the archive as a volume, however, would allow Apple's migration assistant to use a QRecall backup as the source for a new install (assuming that one had captured the entire volume), which I think is a very significant advantage indeed.

I will definitely push this issue higher on the new feature list. It might require a little patience, though. Something like this is not likely to show up until 1.3 or 1.4.

Nicholas Sloan wrote:Hi, I am using and enjoying 1.2 beta, almost without event. Thank you.

That's always good to hear.

One small suggestion: in the Conditions section of the Action settings dialog, you can set an "Ignore Between" time span, but if you press + to add another condition and try to choose "Ignore Between" again, it is greyed out: so you cannot choose two separate time spans. Is this an arbitrary limitation, or would it be tricky to code to allow for multiple time spans here?

Neither, I just wanted to avoid confusion. It just so happens that I was thinking about this this other day because I have plans to add other conditions; conditions that would also make sense to add to the schedule more than once. I really just wanted to avoid the and/or anxiety that can come with some conditions. Thanks for pointing this out. I'll add it to the to-do list.

What brought this issue to the fore for me was finding my G5 awake in the mornings when it should have gone to sleep as normal after a SuperDuper backup scheduled for 3am. This seemed to be becoming a habit that only started around the time I updated QR to 1.2b, and I found that if I paused QR actions manually the evening before,it would not happen.

There could be a number of factors in play here. First, QRecall 1.2b1 uses a different (more modern) method of telling the operating system not to go to sleep while an action is running. The modern API behaves a little differently than the old one.

If you've added any power management options to your action schedules, this could cause your system to wake up. For example, one side-effect of asking the computer to go to sleep, restart, or shutdown after a series of QRecall actions have finished is that the OS will wake up the monitor to show you the dialog telling you that the computer is going to go to sleep. <Insert old joke about nurses waking up patients to give them their sleeping pills.>

There would be no QR progress windows showing, and I can't be certain that QR was implicated, but it looks like a probability.

The log should tell you what QRecall was doing, and when. You'll need to slide the details control all the way to the right to see insignificant events like power requests.

Jochen Schmidt wrote:A more than 5-fold speedup!!!
Is this actually expected?

Performance improvements for individuals is really hard to estimate. It all depends on your circumstances and configuration.

For most users, I expect mostest performance improvements. But in specific combinations of OS, archive size, RAM size, CPU type, and I/O configuration, the speed increase can be dramatic. I've created test cases here where the beta runs 40x faster than the previous version, capturing a test folder in 40 minutes where version 1.1.4 took over a day.

Ralph Strauch wrote:Are v1.2 archives compatible with earlier versions

No. Writing to an archive using version 1.2 will mean that earlier versions will not be able to use the archive.

For what it's worth, the core data format has not changed. If you use QRecall 1.2b1 to modify an archive and later want to revert back to using 1.1.x, reindexing the archive with 1.1 will make it usable again.