Hi James,
I'm using QRecall 1.2.2.5 with much greater pleasure and success than I have used Time Machine before, so let me start with a capitalized Thank You. ;)
Forgive me the length of this post, I tried to keep it short but failed.

For background: I've set up my system to hold iPhoto and iTunes libraries inside .sparsebundle images, personal data, virtual machines, client files and their test databases reside in separate encrypted .sparsebundle images, most of which are mounted at login, some of which only when needed. Hence I have created a plethora of Capture etc. Actions to keep everything safely backed up: OS X and the contents of the personal and client files images go to a secondary internal hard drive in short intervals. Everything goes onto an external backup drive once a week. Client files additionally go to a corporate network location once a day. Summarizing, I'm currently using QRecall to back up nine data sources into a total of 21 archives, every new client would mean two more data sources and six more backup targets.

Juggling all these backup chains leaves me with a few ideas for optimizing the scheduling of Actions. I've read a few hints on the upcoming QRecall 1.3 in other posts and I'm looking forward to see some of the features in action, so I left out any points that I already found addressed. I've ordered the points by my personal "pain level" from high(top) to low(bottom).

? Prioritize Capture Actions: As the QRecallHelper's can be quite memory-hungry I had to limit QRecall to three concurrent actions. At times this can lead to a long queue of waiting actions, especially in the mornings after connecting to the backup drive and network location. I would greatly appreciate being able to tell QRecall to reserve one action slot exclusively for Capture Actions. A Compact or Verify Action can easily run for an hour, and there may be a handful of them waiting. Actual data snapshot Captures shouldn't need to wait for the backup application's internal data management.
? Arbitrary timeouts for event triggered actions would be great: The external backup drive is always connected while my machine is in the office. Unfortunately, OS X recognizes it very soon during wakeup and then deals with external displays and all the USB gimmicks before letting me log in. By the time I decide that I need a full reboot today, the first Capture etc. Actions have already started, and more are queued. Being able to use not just 1?5min but maybe 10?20min timeouts, things would be much more relaxed.
? Non-Capture Actions should be able to be triggered by Capture Actions: Being able to schedule time-consuming Actions during nighttime is great for stationary machines. For mobile workstations which only have access to their backup media during office hours it's not quite as helpful, especially with floating working hours. For example, I don't want a Verify run every two days or every Thursday at 15:00, I want one after every sixth Capture run. I want a Compact run only after every three Merge runs that actually did merge something.
? Multiple event triggers should be supported: When I need to mount a .sparsebundle image as well as connect an external backup drive for a Capture Action to work, it would be nice to be able to configure the action accordingly. Right now the action just fails and is re-scheduled which doesn't do any harm, but it would be "more right" to watch both trigger events.
? This isn't actually a scheduling feature but just let me add it here: It would be pretty cool to have one Capture Action use several backup targets instead of configuring several Capture Actions for the same data source. This way the same data needed to be read only once, and QRecall could more intelligently keep knowledge about which files to capture. An hourly local snapshot, a daily one to a secondary disk and a weekly one to a remote location might just copy the merged result of the hourly/daily Captures instead of traversing the whole disk again.

I do hope I came across friendly and politely yet understandable. If I didn't, don't hold it against me but rather ask for clarification. Bear in mind that English is not my mother tongue and I'm far away from having mastered all of its subtleties. ;)

Best of regards from sunny-yet-cold Germany,
Norbert

Norbert Karls wrote:Forgive me the length of this post, I tried to keep it short but failed.

Norbet, the length is not a problem. It's great to get detailed suggestions.

Let me just address your suggestions one at a time:

• Prioritize Capture Actions: As the QRecallHelper's can be quite memory-hungry I had to limit QRecall to three concurrent actions. At times this can lead to a long queue of waiting actions, especially in the mornings after connecting to the backup drive and network location. I would greatly appreciate being able to tell QRecall to reserve one action slot exclusively for Capture Actions. A Compact or Verify Action can easily run for an hour, and there may be a handful of them waiting. Actual data snapshot Captures shouldn't need to wait for the backup application's internal data management.

It's interesting that you would suggest this. I've actually played around with an experimental scheduling feature that I think might provide what you want. The setting would limit the schedule from running more than one action of the same "type" (capture, compact, verify) at a time. So if three verify actions were queued up, only one would run at a time, even if you allowed more than one concurrent action.

Having said that, I would suggest you experiment with fewer concurrent actions, and most definitely limit the number of actions that use the same destination volume to 1. Trying to run more than one capture action that's reading from the same physical drive can cause "trashing". This can slow down the actions (and your system) and end up taking longer than if you ran the actions one at a time. It's worth testing.

• Arbitrary timeouts for event triggered actions would be great: ... Being able to use not just 1?5min but maybe 10?20min timeouts, things would be much more relaxed.

This is already in the works to support new types of event schedules. It should show up in the next (major) version.

• Non-Capture Actions should be able to be triggered by Capture Actions:

This, and a couple of similar features, are on the to-do list, but still needs some design work. Chance of making it into the next major version is about 50%.

• Multiple event triggers should be supported: When I need to mount a .sparsebundle image as well as connect an external backup drive for a Capture Action to work, it would be nice to be able to configure the action accordingly. Right now the action just fails and is re-scheduled which doesn't do any harm, but it would be "more right" to watch both trigger events.

To some degree, they already are. You can schedule an action to start when the archive volume connects, and then add a "hold while no capture items available" condition that will wait until the capture source item appears. (You can also write the reverse: start when source mounts, then wait for archive volume.) If you're looking for something else, write and explain the situation in more detail.

• This isn't actually a scheduling feature but just let me add it here: It would be pretty cool to have one Capture Action use several backup targets instead of configuring several Capture Actions for the same data source.

Sorry, but QRecall just doesn't work that way. It wouldn't result in any efficiency gain.

When people suggest this, I think what they really want is a "waterfall backup" or "cascading archive" feature, which is still on the drawing board.

I do hope I came across friendly and politely yet understandable.
I'm delighted that you posted, and I hope I've addressed some of your issues.

When I read Norbert's post, it felt quite familiar. I too have some trouble to get "many sources to many archives at the right time" working. I really tried to get there with the many clever schedule options of QRecall (and James' help) ? it did not work out.

Life would be so much easier if QRecall only had one simple scripting command: "run Action #NameOfAction# now". I could easily write a batch that would take care of everything else (mounting/unmounting disks, stopping Time Machine, closing databases, not interfering with my work or the SuperDuper cloning of the same disk at the same time, not running a time consuming verify action for the local archive when I need to capture to the offsite archive first etc.).

Otherwise QRecall is really the best backup tool I ever used (and I used a lot).

Johannes

James wrote:

Norbert wrote:Prioritize Capture Actions?{explain}

{upcoming feature to limit actions to one instance per type}?
?I would suggest you experiment with fewer concurrent actions?{explain}

A toast to the mentioned feature! As my backup chains usually stack merge/compact/verify actions I'm expecting a great deal of effect from this.

My concurrency issue is with memory consumption, not disk performance at all. If I had a more modern Mac supporting 16GiB of RAM instead of only 8 I wouldn't have to impose a limit at all. Restricting QRecall to even fewer concurrent actions would make individual actions wait even longer. If I could reserve one action for captures only and two more for anything else my snapshots could be written more regularly. It's not that anything is broken now, it's just my personal taste not being happy with 20minute snapshots not being taken exactly every 20minutes. What really hurts me is that for hours on end no new snapshots are taken at all once a week because the compact and verify actions all tend to trigger at once (when archive volume connects).

James wrote:

Norbert wrote:• Arbitrary timeouts for event triggered actions
• Non-Capture Actions should be able to be triggered by Capture Actions

?in the works?
?on the to-do list?

Make this a paid upgrade, it will totally be worth it (to me, at least).

James wrote:

Norbert wrote:Multiple event triggers should be supported?{explain}

To some degree, they already are.{explain}

I totally missed that. Thank you for pointing it out, I've already changed my actions to use this.

James wrote:

Norbert wrote:?to have one Capture Action use several backup targets?

?a "waterfall backup" or "cascading archive"?

That pretty much nails it. Have a local archive with very frequent snapshots to undo editing errors or accidental deletes/overwrites of recent work. Have a much larger archive on an external drive with daily/weekly snapshots to recover from disk disaster or harmful OS upgrades or similar. Have a monthly snapshot on tape in the vault just in case the office gets robbed or burns down so you lose only a months worth of progress. I can see how such a topic can need a whole lot of design work so do take your time on this.

Thank you for your quick and helpful feedback,
I wish I still knew anyone who doesn't already use QRecall so i could recommend it. ;)

Best regards,
Norbert

Johannes wrote:Life would be so much easier if QRecall only had one simple scripting command: "run Action #NameOfAction# now".

That's in the works, and I apologize for the delay.

I already had a crude command-line version of QRecall, which I used for testing. My original plan was to polish it up and make it part of the QRecall product. But it soon became evident that a stand-alone command-line tool has a lot of limitations and some security issues. For example, your example of "run Action xxx" wouldn't work with a stand-alone tool; you'd have to reproduce all of the parameters of the action you want to perform on the command line (-capture this/that/other/thing -exclude this -ignore that ...).

So I've scrapped the idea of a self-contained command-line to in favor of a command-line front-end to the existing scheduler and helper architecture. This will allow much more flexibility in what QRecall actions you can perform, won't introduce new security vectors, and opens the possibility of scheduler related commands (i.e. qrecall schedule --holdall 20min).

Unfortunately, this means a complete rewrite of the command-line tool, which will take a little more time. But I think it will be worth it.

QRecall development took a dramatic turn last fall. Like the command-line tool decision, a lot of QRecall's architecture was keeping me from pushing it in the direction I wanted it to go in. There was a long (long!) list of features that couldn't be implemented without breaking the current archive format, so they kept getting pushed to the back.

That dam broke in October. Apple's decision to deprecate much of the filesystem API that QRecall relies on has forced me to rewrite all of the filesystem code using the low-level BSD API. This required significant changes to the archive data record formats. One side effect is that the next version of QRecall will NOT be forwards compatible with previous versions. In other words, once you start using the new version, you can't use that archive with an older version.

But every cloud has its silver lining: This break has allowed me to jettison all kinds of legacy data and code, like aliases and PPC support. Aliases have been replaced by modern URL bookmarks and I can now use the latest compilers. I can finally add support for things like per-item filtering, a more rational method of ignoring changes, support for hard-linked directories, data redundancy, new event schedules, simpler inter-process communications, better logging/reporting, and so much more.

Much of this work has already been done, but a lot still remains. I'm currently writing two books for Apress, which is consuming most of my time, but I assure you that work on QRecall continues.

I have to say that I've been overwhelmed by the incredible support, encouragement, and engagement of the QRecall community. I couldn't ask for better customers.

Norbert Karls wrote:Make this a paid upgrade, it will totally be worth it (to me, at least).

I'm sorry to disappoint you, but the next version of QRecall will be a free upgrade.

James Bucanek wrote:

Norbert Karls wrote:Make this a paid upgrade, it will totally be worth it (to me, at least).

I'm sorry to disappoint you, but the next version of QRecall will be a free upgrade.

That proves it, James is mad, quite mad!

James Bucanek wrote:Much of this work has already been done, but a lot still remains. I'm currently writing two books for Apress, which is consuming most of my time, but I assure you that work on QRecall continues.

That's great to hear about all the upcoming changes and improvements in QRecall! Just out of curiosity, can you san anything about the two books you're working on?

-- Bruce

Bruce Giles wrote:... can you say anything about the two books you're working on?

According to my publisher, I'm supposed to be shouting it from the rooftops.

I recently finished revising David Mark's venerable Learn C on the Mac.

I'm two-thirds done with Learn iOS App Development (no link yet). It should be on the shelves by September.

James Bucanek wrote:?a command-line front-end to the existing scheduler and helper architecture. This will allow much more flexibility in what QRecall actions you can perform, won't introduce new security vectors, and opens the possibility of scheduler related commands (i.e. qrecall schedule --holdall 20min).

May I follow up on this asking if there has been any progress already? I'm in the process of setting up a second Mac with QRecall (shared archives, finally) and that topic popped back into my mind.

Best regards,
Norbert

Norbert Karls wrote:May I follow up on this asking if there has been any progress already?

I?m pleased to write that QRecall development is?once again, thankfully, finally?in high gear.

I'm putting the final touches on the command line tool, which does a lot more than I orignally planned.

I have some new scheduling logic/features that I want to get working before I release it as a beta, along with some massive UI code clean-up to do.

There are also a bunch of random problems, glitches, and crashes that I?m still hunting down. Consequences of all that new code. *sigh*

And I have more testing to do. QRecall needs to be really stable before I spring the next version on my loyal users. This version will NOT be backwards compatible with earlier versions. Once you capture new data using 2.0, you can?t open that archive with an earlier version. So early adopators need to know they?ll be flying without a safety net...