Peter B. wrote:I want to be able to define a maximum size of an archive in bytes (or KB, MB, GB, TB), independent of the space left on the volume.

I'll add that to the to-do list. It wouldn't be a difficult feature to include.

The archive settings sheet should specify that the leftmost position of the three sliders is "off." This would be helpful because users don't always read the manuals before starting to use software.

I've added that to the to-do list too.

I'd like a separate button for the menu which is normally available by right clicking the stop button in the QRecall Activity window. Recently, I accidentally left clicked the stop button for a process in the activity window, when I wanted to have right clicked.

This issue comes up occasionally. There's a wish-list item to split the button into two buttons: a stop button and an action button.

I want options to be able to delay an action more than 5 minutes, so I can perform 12 or more actions in a specified sequence when a volume connects. In my case it's because I'll be using two archives on the same volume, which have six actions each.

That's already in the works, to support new event actions being developed for the next release.

Peter B. wrote:I, too, would like to have source data validation.

The way QRecall stores data, source data validation doesn't make much sense. In fact, QRecall's archive structure is specifically designed so that is does not have to rely on comparing the data backed-up with the original as a means of validation.

All data in a QRecall archive is validated via interlocking layers of checksums and consistency guards. There's no need to compare the data in the archive with the orignal file, because the archive stores enough redundant information to determine if the data has been damaged or altered in any way.

You could compare the data in the archive with the original files, but that would only tell you if the original file had changed or not. That's a feature that's on the to-do list, but it won't improve the security of the data in the archive.

However, I think that not all applications or processes update the modification date of files that are written to.

Applications do not update the creation, modification, attributes, or access time of file objects. That's handled automatically by the filesystem. In fact, an application would have to go to extraordinary measures to modify a file in a way that made it appear that it hadn't. I'm not aware of any apps that do this.

Ralph Strauch wrote:It did not seem to be reading the whole drive this time, so whatever was causing it earlier didn't transfer to the direct mount.

It's also possible that the previous capture decided to resize one of the index files. This can be a very I/O intensive and time consuming process (especially over a wireless link). Index files in the archive are periodically resized/rebuilt for efficiency. This will happen a dozen or so times over the lifetime of an archive.

Ralph Strauch wrote:Is qrecall reading the entire archive? If so, why, and is it normal?

It sounds like it's recapturing all of the items on your drive. Why, I can't say (definitively). If that's what it is doing, then capturing to a drive directly connected to your MacBook is likely to be much faster.

There are a number of reasons why QRecall will rescan all of the items on a drive, or decide to recapture items.

QRecall periodically rescans all of the files on your volume looking for changes. This usually happens about once every 14 days. If a layer is damaged or incomplete, that will also force it rescan all of the items.

If you've reformatted or resized a volume, that may cause it appear as a new volume, which QRecall will dutifully recapture—in its entirety.

The short answer is (I'm hoping) that QRecall is just doing its job, re-capturing items to ensure that they're all backed up.

Dr. D wrote:Concerning older QRecall versions, where can I find those, please?

Sorry, I forgot to mention that.

Go to the Release Notes page. (You can also get to the release notes from the Download page.)

Next to each set of release notes is a download link for that version.

Dr. D wrote:1) "Standard" users every now and then don't execute the daily backup run - QRecall patiently shows "waiting for archive" even when the volume with the archive is mounted and the archive visible in the Finder and no other machine uses the archive. Restarting the machine generally seems to solve the issue but that obviously misses the backup window at night. Is this a result of several machines sharing the archive file? Is there anything one can do to avoid the issue?

QRecall uses the standard OS X sharing modes to detect if another process has a file open for reading or writing. It won't update (capture to) an archive if any other process is currently reading or writing the files in that repository package. You can use the lsof command line tool (on the server) to determine which process has the archive file(s) open.

More than likely, it's going to be the file sharing server process. A network file servers opens file on behalf of its network clients. What sometimes happens is the client momentarily loses its connection with the server. The server then leaves those files open forever, because there's no client to tell it to close the file. (When the client reconnects, it looks to the server like a new client.) All subsequent capture operations are then stymied by the phantom user that's keeping the files open.

Stopping and starting the file sharing process will usually resolve this situation. But that will also kick off all of the network users, which is a hassle.

2) "Special" users sometimes (but not always) have trouble mounting both the network volume and their dedicated sparse image file. Restarting or manual mounting seems to solve the issue but I wonder if there is a better way to do things.

QRecall mounts network volumes using the standard OS X alias resolution functions. Most of the time this works, but sometimes it doesn't. If network communications are a tad dicey (which they might be, given your first question), then it's a crap shoot. I used to have this problem all the time on my local area network, but recently it's been working flawlessly. I can't tell you why.

3) I noticed QRecall having trouble on Mac OS X 10.5.5 (odd crashes) but working well on Mac OS 10.5.8. We are dependent on specific OS versions due to legacy software for certain tasks. Are there any older versions of QRecall available that could operate using 10.5.5 or even 10.4?

Honestly, QRecall barely supports 10.5.8. Very little serious testing has been done on 10.5, and the next version of QRecall will drop support for PPC systems, bringing the minimum OS version requirement to 10.6.

You're welcome to try older versions of QRecall on your 10.5.x machines. If your archives haven't gotten too big, you can probably downgrade all the way to version 1.2.0 without running into compatibility issues. QRecall archive maintain compatibility flags, so it's always safe to try an older version of the application. If the format of an archive is not compatible with the running version of QRecall, QRecall will report that and refuse to perform the operation. There are different compatibility requirements for the actions, so a version might be able to read and recall from an archive, but not capture to it.

If you try to downgrade, you must uninstall QRecall before installing the older version. QRecall will automatically upgrade its components, but it won't downgrade them.

Peter B. wrote:This limit will increase in future versions, right?

It probably will.

I've experimented with archives up to 10TB, but the performance is (currently) abysmal. The limiting factor is the de-duplication logic. The work to determine if a data block is a duplicate increases exponentially with the size of the archive. You don't notice the difference up to about 1-1.5TB, but after that things really slow down.

Performance can be improved by caching more information in RAM, but a cache big enough to get a 10TB archive to capture smoothly is more memory than most systems have.

As Mac ships with more and more RAM, and ever faster I/O systems, the useable size of an archive will increase. Until then, consider splitting your archives. Store your virtual machine images or media projects in one archive, and your system and regular documents in another.

Peter B. wrote:Does this refer to encryption for the QRecall archive (putting the QRecall archive on an encrypted volume or encrypted sparse bundle), or to the data that will be backed up on the QRecall archive (backing up an encrypted volume or encrypted sparse bundle)? I think that it's the former, but I'd like to be sure.

It's the former.

You can encrypt a QRecall archive now by storing it on an encrypted volume or in an encrypted disk image.

Since this isn't always practical, or convenient, there's a plan to add record-level encryption to the archive. (Tip: turning on compression for the archive will also obscure any sensitive data from all but the most determined hackers.)

Bruce Giles wrote:... can you say anything about the two books you're working on?

According to my publisher, I'm supposed to be shouting it from the rooftops.

I recently finished revising David Mark's venerable Learn C on the Mac.

I'm two-thirds done with Learn iOS App Development (no link yet). It should be on the shelves by September.

Ralph,

There's only one set of status values per archive, and it is updated whenever new items are captured—regardless of what is captured or by whom.

What you're asking for is per-owner or per-volume status information, and that's an interesting idea that I'll add to the wish list.

Dr. D,

QRecall obtains the command parameters of the dynamic_pager process. It examines these to determine if the VM backing store has been relocated, say to a dedicated swap drive.

If QRecall cannot find a running dynamic_page process, or it can't make sense of its parameters, it falls back to assuming that the VM backing store is in /var/vm, and logs the message you see.

If you're not running dynamic_pager, you can ignore this message. Just be aware that nothing in /var/vm is going to be captured, which probably doesn't make any difference for you.

Dr. D wrote:One question I could not easily work out with the excellent test scheme is how to perform a full restore of a hard disk.

Finally, an easy question.

To restore a volume, you must first have captured a whole volume (not just some files and folders in that volume).

To recall the contents of a captured volume to the volume it was originally captured from:

Select the volume icon in the archive.

Choose Restore from the Archive menu.

To recall a captured volume to a different volume (i.e. not that one it was originally captured from):

Select the volume icon in the archive.

Hold down the option key and choose Restore Volume To? from the Archive menu.

Select the volume you want to overwrite.

Alternate method:

Drag the volume icon from the archive and drop it into the volume icon you want to overwrite.

QRecall will ask you if you want to recall the contents of the volume into a folder, or replace the entire volume. Choose the later.

In all of these situations, the original contents of the volume will be erased. See Help > QRecall Help > Guide > The Basics > Restore Items > Restoring Volumes.

Caveats: You must use the same (or later) major version of OS X to restore a system (bootable) volume. In other words, to restore a bootable 10.7 volume you should be running 10.7 or later. You cannot restore a 10.7 system volume while running 10.6, for example. The reason is that recent major releases of OS X have introduced new filesystem features (like hard-linkable directories and compressed data forks). Earlier versions of the OS are incapable of recreating these new features, and the volume cannot be restored property.

Performance tip: If your drive is fast, erasing the volume first usually results in better recall performance. If it's on the slow side (say, an external drive connected via USB), or if the drive reads significantly faster than it writes, then letting QRecall individually replace the existing files with the recalled ones is usually faster.

Clemens Oertel wrote:The authors of another backup tool provide a detailed technical description of their backup format, as well as the source code for a restore-only command line utility. While I am not using this tool for different reasons, I find this approach very reassuring.

I'm not philosophically opposed to documenting QRecall's file structure, or even providing code that can read it. The only reason I haven't do so is due to resources and practicality. The QRecall file structure is not trivial. I could probably document it with a few days work, but that really wouldn't help anyone; you'd have to spend another month writing code that could decode it. I could provide much of that code, but again?even if you were a programmer?you'd still have a boatload of work to do to get any useful information out of an archive.

Two, more practical, solutions are already in the works: a command-line version of QRecall and a filesystem driver that will mount a QRecall archive as a volume. I think either of those solutions would give you the kind of "back up" recovery method you're looking for.

Norbert Karls wrote:Make this a paid upgrade, it will totally be worth it (to me, at least).

I'm sorry to disappoint you, but the next version of QRecall will be a free upgrade.

Johannes wrote:Life would be so much easier if QRecall only had one simple scripting command: "run Action #NameOfAction# now".

That's in the works, and I apologize for the delay.

I already had a crude command-line version of QRecall, which I used for testing. My original plan was to polish it up and make it part of the QRecall product. But it soon became evident that a stand-alone command-line tool has a lot of limitations and some security issues. For example, your example of "run Action xxx" wouldn't work with a stand-alone tool; you'd have to reproduce all of the parameters of the action you want to perform on the command line (-capture this/that/other/thing -exclude this -ignore that ...).

So I've scrapped the idea of a self-contained command-line to in favor of a command-line front-end to the existing scheduler and helper architecture. This will allow much more flexibility in what QRecall actions you can perform, won't introduce new security vectors, and opens the possibility of scheduler related commands (i.e. qrecall schedule --holdall 20min).

Unfortunately, this means a complete rewrite of the command-line tool, which will take a little more time. But I think it will be worth it.

QRecall development took a dramatic turn last fall. Like the command-line tool decision, a lot of QRecall's architecture was keeping me from pushing it in the direction I wanted it to go in. There was a long (long!) list of features that couldn't be implemented without breaking the current archive format, so they kept getting pushed to the back.

That dam broke in October. Apple's decision to deprecate much of the filesystem API that QRecall relies on has forced me to rewrite all of the filesystem code using the low-level BSD API. This required significant changes to the archive data record formats. One side effect is that the next version of QRecall will NOT be forwards compatible with previous versions. In other words, once you start using the new version, you can't use that archive with an older version.

But every cloud has its silver lining: This break has allowed me to jettison all kinds of legacy data and code, like aliases and PPC support. Aliases have been replaced by modern URL bookmarks and I can now use the latest compilers. I can finally add support for things like per-item filtering, a more rational method of ignoring changes, support for hard-linked directories, data redundancy, new event schedules, simpler inter-process communications, better logging/reporting, and so much more.

Much of this work has already been done, but a lot still remains. I'm currently writing two books for Apress, which is consuming most of my time, but I assure you that work on QRecall continues.

I have to say that I've been overwhelmed by the incredible support, encouragement, and engagement of the QRecall community. I couldn't ask for better customers.