Peter B. wrote:This limit will increase in future versions, right?

It probably will.

I've experimented with archives up to 10TB, but the performance is (currently) abysmal. The limiting factor is the de-duplication logic. The work to determine if a data block is a duplicate increases exponentially with the size of the archive. You don't notice the difference up to about 1-1.5TB, but after that things really slow down.

Performance can be improved by caching more information in RAM, but a cache big enough to get a 10TB archive to capture smoothly is more memory than most systems have.

As Mac ships with more and more RAM, and ever faster I/O systems, the useable size of an archive will increase. Until then, consider splitting your archives. Store your virtual machine images or media projects in one archive, and your system and regular documents in another.

Peter B. wrote:Does this refer to encryption for the QRecall archive (putting the QRecall archive on an encrypted volume or encrypted sparse bundle), or to the data that will be backed up on the QRecall archive (backing up an encrypted volume or encrypted sparse bundle)? I think that it's the former, but I'd like to be sure.

It's the former.

You can encrypt a QRecall archive now by storing it on an encrypted volume or in an encrypted disk image.

Since this isn't always practical, or convenient, there's a plan to add record-level encryption to the archive. (Tip: turning on compression for the archive will also obscure any sensitive data from all but the most determined hackers.)

Bruce Giles wrote:... can you say anything about the two books you're working on?

According to my publisher, I'm supposed to be shouting it from the rooftops.

I recently finished revising David Mark's venerable Learn C on the Mac.

I'm two-thirds done with Learn iOS App Development (no link yet). It should be on the shelves by September.

Ralph,

There's only one set of status values per archive, and it is updated whenever new items are captured—regardless of what is captured or by whom.

What you're asking for is per-owner or per-volume status information, and that's an interesting idea that I'll add to the wish list.

Dr. D,

QRecall obtains the command parameters of the dynamic_pager process. It examines these to determine if the VM backing store has been relocated, say to a dedicated swap drive.

If QRecall cannot find a running dynamic_page process, or it can't make sense of its parameters, it falls back to assuming that the VM backing store is in /var/vm, and logs the message you see.

If you're not running dynamic_pager, you can ignore this message. Just be aware that nothing in /var/vm is going to be captured, which probably doesn't make any difference for you.

Dr. D wrote:One question I could not easily work out with the excellent test scheme is how to perform a full restore of a hard disk.

Finally, an easy question.

To restore a volume, you must first have captured a whole volume (not just some files and folders in that volume).

To recall the contents of a captured volume to the volume it was originally captured from:

Select the volume icon in the archive.

Choose Restore from the Archive menu.

To recall a captured volume to a different volume (i.e. not that one it was originally captured from):

Select the volume icon in the archive.

Hold down the option key and choose Restore Volume To? from the Archive menu.

Select the volume you want to overwrite.

Alternate method:

Drag the volume icon from the archive and drop it into the volume icon you want to overwrite.

QRecall will ask you if you want to recall the contents of the volume into a folder, or replace the entire volume. Choose the later.

In all of these situations, the original contents of the volume will be erased. See Help > QRecall Help > Guide > The Basics > Restore Items > Restoring Volumes.

Caveats: You must use the same (or later) major version of OS X to restore a system (bootable) volume. In other words, to restore a bootable 10.7 volume you should be running 10.7 or later. You cannot restore a 10.7 system volume while running 10.6, for example. The reason is that recent major releases of OS X have introduced new filesystem features (like hard-linkable directories and compressed data forks). Earlier versions of the OS are incapable of recreating these new features, and the volume cannot be restored property.

Performance tip: If your drive is fast, erasing the volume first usually results in better recall performance. If it's on the slow side (say, an external drive connected via USB), or if the drive reads significantly faster than it writes, then letting QRecall individually replace the existing files with the recalled ones is usually faster.

Clemens Oertel wrote:The authors of another backup tool provide a detailed technical description of their backup format, as well as the source code for a restore-only command line utility. While I am not using this tool for different reasons, I find this approach very reassuring.

I'm not philosophically opposed to documenting QRecall's file structure, or even providing code that can read it. The only reason I haven't do so is due to resources and practicality. The QRecall file structure is not trivial. I could probably document it with a few days work, but that really wouldn't help anyone; you'd have to spend another month writing code that could decode it. I could provide much of that code, but again?even if you were a programmer?you'd still have a boatload of work to do to get any useful information out of an archive.

Two, more practical, solutions are already in the works: a command-line version of QRecall and a filesystem driver that will mount a QRecall archive as a volume. I think either of those solutions would give you the kind of "back up" recovery method you're looking for.

Norbert Karls wrote:Make this a paid upgrade, it will totally be worth it (to me, at least).

I'm sorry to disappoint you, but the next version of QRecall will be a free upgrade.

Johannes wrote:Life would be so much easier if QRecall only had one simple scripting command: "run Action #NameOfAction# now".

That's in the works, and I apologize for the delay.

I already had a crude command-line version of QRecall, which I used for testing. My original plan was to polish it up and make it part of the QRecall product. But it soon became evident that a stand-alone command-line tool has a lot of limitations and some security issues. For example, your example of "run Action xxx" wouldn't work with a stand-alone tool; you'd have to reproduce all of the parameters of the action you want to perform on the command line (-capture this/that/other/thing -exclude this -ignore that ...).

So I've scrapped the idea of a self-contained command-line to in favor of a command-line front-end to the existing scheduler and helper architecture. This will allow much more flexibility in what QRecall actions you can perform, won't introduce new security vectors, and opens the possibility of scheduler related commands (i.e. qrecall schedule --holdall 20min).

Unfortunately, this means a complete rewrite of the command-line tool, which will take a little more time. But I think it will be worth it.

QRecall development took a dramatic turn last fall. Like the command-line tool decision, a lot of QRecall's architecture was keeping me from pushing it in the direction I wanted it to go in. There was a long (long!) list of features that couldn't be implemented without breaking the current archive format, so they kept getting pushed to the back.

That dam broke in October. Apple's decision to deprecate much of the filesystem API that QRecall relies on has forced me to rewrite all of the filesystem code using the low-level BSD API. This required significant changes to the archive data record formats. One side effect is that the next version of QRecall will NOT be forwards compatible with previous versions. In other words, once you start using the new version, you can't use that archive with an older version.

But every cloud has its silver lining: This break has allowed me to jettison all kinds of legacy data and code, like aliases and PPC support. Aliases have been replaced by modern URL bookmarks and I can now use the latest compilers. I can finally add support for things like per-item filtering, a more rational method of ignoring changes, support for hard-linked directories, data redundancy, new event schedules, simpler inter-process communications, better logging/reporting, and so much more.

Much of this work has already been done, but a lot still remains. I'm currently writing two books for Apress, which is consuming most of my time, but I assure you that work on QRecall continues.

I have to say that I've been overwhelmed by the incredible support, encouragement, and engagement of the QRecall community. I couldn't ask for better customers.

Norbert Karls wrote:Forgive me the length of this post, I tried to keep it short but failed.

Norbet, the length is not a problem. It's great to get detailed suggestions.

Let me just address your suggestions one at a time:

• Prioritize Capture Actions: As the QRecallHelper's can be quite memory-hungry I had to limit QRecall to three concurrent actions. At times this can lead to a long queue of waiting actions, especially in the mornings after connecting to the backup drive and network location. I would greatly appreciate being able to tell QRecall to reserve one action slot exclusively for Capture Actions. A Compact or Verify Action can easily run for an hour, and there may be a handful of them waiting. Actual data snapshot Captures shouldn't need to wait for the backup application's internal data management.

It's interesting that you would suggest this. I've actually played around with an experimental scheduling feature that I think might provide what you want. The setting would limit the schedule from running more than one action of the same "type" (capture, compact, verify) at a time. So if three verify actions were queued up, only one would run at a time, even if you allowed more than one concurrent action.

Having said that, I would suggest you experiment with fewer concurrent actions, and most definitely limit the number of actions that use the same destination volume to 1. Trying to run more than one capture action that's reading from the same physical drive can cause "trashing". This can slow down the actions (and your system) and end up taking longer than if you ran the actions one at a time. It's worth testing.

• Arbitrary timeouts for event triggered actions would be great: ... Being able to use not just 1?5min but maybe 10?20min timeouts, things would be much more relaxed.

This is already in the works to support new types of event schedules. It should show up in the next (major) version.

• Non-Capture Actions should be able to be triggered by Capture Actions:

This, and a couple of similar features, are on the to-do list, but still needs some design work. Chance of making it into the next major version is about 50%.

• Multiple event triggers should be supported: When I need to mount a .sparsebundle image as well as connect an external backup drive for a Capture Action to work, it would be nice to be able to configure the action accordingly. Right now the action just fails and is re-scheduled which doesn't do any harm, but it would be "more right" to watch both trigger events.

To some degree, they already are. You can schedule an action to start when the archive volume connects, and then add a "hold while no capture items available" condition that will wait until the capture source item appears. (You can also write the reverse: start when source mounts, then wait for archive volume.) If you're looking for something else, write and explain the situation in more detail.

• This isn't actually a scheduling feature but just let me add it here: It would be pretty cool to have one Capture Action use several backup targets instead of configuring several Capture Actions for the same data source.

Sorry, but QRecall just doesn't work that way. It wouldn't result in any efficiency gain.

When people suggest this, I think what they really want is a "waterfall backup" or "cascading archive" feature, which is still on the drawing board.

I do hope I came across friendly and politely yet understandable.
I'm delighted that you posted, and I hope I've addressed some of your issues.

Jonathan Edwards wrote:Can there be an option to keep deleted items permanently? I currently have this set to '999 months'. I sometimes delete files from the source knowing I safely have a captured backup. I don't want to risk permanently deleting those items in the archive through Merge and Compact actions.

Welcome Jonathan,

I do not have plans to add a setting to never remove deleted items. The original idea for the feature was not to keep deleted items forever—for fear that they would eventually consume all available archive space—but as a safety net for those "oops!" moments.

That said, 999 months is as close to "forever" as you're likely to get. It works out to 83¼ years. So my suggestion would be to set a reminder for the year 2095 and either make a copy of your current archive and start a new one, or request this feature again.

Do you have any plans to enable high resolution icon and font support for Retina Macs please?

It's on the to-do list. Hopefully, in the next major release.

Keep up the great work!

We're on it!

Bruce Giles wrote:Anyway, I think you need to do something better with the progress bar in that situation. It's really confusing (or misleading) when it goes all the way to 100% and then just sits there for tens of minutes. If the "locating unused space" step is not something you can quantify in order to show progress with the bar, then how about switching to an indeterminate ("barber pole") type progress bar instead?

It's a bug. It's supposed to display an indeterminate progress indicator.

In the version that introduced the "Keep deleted items" option, a new phase was added to the compact action. After the initial archive analysis, QRecall walks through the recent layers looking for deleted items to expunge from the archive. This is associated with a progress bar that advances as each layer is processed.

What's supposed to happen next is the "Looking for unused space" phase takes over, changes that status message, and switches the progress bar to indeterminate. What really happens is that the progress bar remains at 100%, left over from the "erasing deleted items" phase.

I've fixed the code and it will eventually show up in future version of QRecall.

David,

You're doing it right.

As a test, I started up a Parallels VM, created a file in the VM, shut it down, restored from an earlier version, and started it up again. The guest OS booted up just fine, and the file was gone.

If you want to verify that the recall is happening correctly, open the VM package and check the dates/sizes. In the Finder, select the VM package, right/control+click on it, and choose the Show Package Contents command. Perform the Restore/Recall and check the dates and sizes of the restored items to make sure they are what you expect.

I will note that there are pitfalls associated with trying to capture VM packages while the virtual machine is running/paused. Like databases, the VM software may have information caches in memory that doesn't get captured, so the recalled VM package is incomplete. I suspect, however, that this is not the case here.

If everything looks correct, then my guess is that there's some other issue related to your virtualization software.

Adrian Chapman wrote:This is very interesting, but no matter what I do I still see all layers.

Try switching to icon or time view and drilling down into a folder that contains items that change infrequently.

The layers displayed/highlighted in the layers pane are those that contain any items visible in the browser pane. If you are in column or outline view, this will tend to encompass a lot of items.

Gary K. Griffey wrote:My question is...why is this layer considered "unrelated to the items in the browser"?

Excellent question!

When you are browsing items in an archive, QRecall identifies which layers contain data about those captured items. The other layers it either dims or hides (depending on your view settings).

As an example, let's say you capture your home folder in a new archive, creating a single layer. Now let's say you make some changes in iTunes (modifying items in your Music folder) and capture your home folder again.

When you open the archive and view your home folder, you'll see two layer. If you navigate into your Music folder, you'll see two layer.

If, however, you navigate into your Documents folder, QRecall will only display one layer. The second layer will be dimmed or hidden. That's because the second layer only captured items in your home and Music folders. There are no items from your Documents folder captured in the second layer.

By trimming down the layers pane to just the layers that are significant, it's easier and more productive to browse the layers. You only see the dates where the items you are looking at were captured, and if you shade layers to browse or recall earlier items, you'll be working with just those layers where something changed.