Stephen Murphy wrote:1) If I read the Help file correctly, when I exclude something, and then restore a directory, anything excluded would be deleted. I would like to be able to exclude files or folders in a way that they are totally ignored - neither archived nor restored. An example here would be podcasts. They exist in my iTunes folder hierarchy, and at times they take up a lot of space, so I don't want to back them up - I can always re-download them if I need to. But if I excluded them, and then had to restore the containing iTunes folder, they would be deleted.

You read it correctly. This is an excellent suggestion. I'm working on some new logic to deal with deleted files. It might be possible to extend this to handle excluded items in a special way too.

2) Because of this, to "exclude" certain directories/files, the only way I can see at the moment is to explicitly include all files/folders other than those to be "ignored". So, in the iTunes case, I would have to include, say, every Artist folder at the same level of the hierarchy of the Podcasts folder - that's a lot of work, and it also means that every new folder that gets created would have to be explicitly added (and may not be done so in time for a critical backup). It would be better to have an overall include, excluding ("ignoring") some of the folders/files underneath a certain level.

Or, you can simply capture the entire iTunes folder (excluding Podcasts) and should you ever need to restore it, select all of the iTunes subfolders in the archive and choose Restore. Each individual subfolder will be restored. Since you didn't capture the Podcast folder it isn't in that list and would be ignored. Probably much less work than the other way.

3) My home directory is around 60GB, but I only really need to backup about 12GB of this. So, I'm forced to use the clumsy "include just these folders" approach I've suggested above. The final problem here is that I don't think I can inlcude hidden files/folders (say, a .svn directory in home) in the current system.

There are a number of ways of doing this. Since the .svn directory is a folder, you can use the Go To command in the open files dialog or the Finder (Command+Shift+G) to navigate into the hidden directory. If you can open the hidden item using the Finder or some other application (BBEdit, Hex Fiend, etc.) you can then drag the item's icon from its window bar title into the action.

If you enable hidden files in the Finder, you can drag hidden items directly.

Now that the log files are up, I can answer the primary question.

Frederic Thomas wrote:So I have QRecall set to backup my home to a AFP server every day. It did not run last night, but there's nothing about it in the log? See attached screen shot: it ran the 27th and the next execution is 29th, nothing in log about 28th... (b)

QRecall wasn't running at 1:00 on the 28th. According to the log. The last log record on the 27th was

2007-07-27 19:58:04.202 +0200 #debug# received SIGTERM [3.732883.263.36]

This indicates that something killed the scheduler, probably a shutdown. The next log record is

2007-07-28 02:15:18.717 +0200 #debug# discovered recent 'QRecall Archive.quanta', mod=2007-07-27 01:06:53 +0200, repository=/Volumes/Gandalf/QRecall Archive.quanta [4.732885.385.2]

Which is one of the first things that will happen when you log in. Given that there are no log messages during the intervening period (the scheduler attempts to restart every half hour), my conclusion would be that your computer was off at 1:00 on the 28th. From the help file (Help > Automation > Scheduling):

Actions are not scheduled while the computer is off. When you restart, actions will be scheduled to run at their next calculated run time.

If you want a QRecall action to run consistently at 1:00 every morning, schedule your computer to start up at least 30 minutes before.

If, however, you simply put your system to sleep, any pending actions will start immedately when the computer is awakened again. Again, from the help:

Actions do not run?nothing does?while the computer is sleeping. When the computer wakes up, any actions that would have started while asleep are started immediately.

Frederic Thomas wrote:So I have QRecall set to backup my home to a AFP server every day. It did not run last night, but there's nothing about it in the log? See attached screen shot: it ran the 27th and the next execution is 29th, nothing in log about 28th... (b)

There's no attached screen shot, so I can't comment. It would be better to attach (or send me privately) your actual log file as there is much more information there that might illuminate a problem.

In any case, the capture is set to INTERVAL every 1 day starting from xxx... so why is it not trying to backup ? It did backup on the 27th, more than one day ago... (a)
[we're the 28th at noon as I write this]

Does your action have a condition that says "Ignore if no archive?" This will cause your action to be silently skipped if the archive isn't mounted at the time the action runs. If you want QRecall to scream and wail when the archive is missing, remove that condition.

It's also remotely possible that the scheduler daemon wasn't running at the time the action was scheduled to run. Is there any chance you had just restarted your system?

And lastly, looking at the log, (c), there is no schedule using IncyBack.quanta so why is it looking for it? When is it going to delete alias of long gone archives in its Preferences? At the level just before max, messages about this fill the screen as if this was a terrible problem! (no screenshot of this).

A lot of the log messages in QRecall are there for the purposes of diagnosing the beta. Most of these will disappear in the final release.

- (a) INTERVAL does not seem to work as I would expect of interval

Can't really comment because I don't know what's going on (or didn't go on, as the case may be.)

- (b) If a scheduled backup is skipped, I think a log entry would be good. Again this is all about visibility on what the scheduler is/was/will be up to.

Everything gets logged. Just about the only reason an event like this wouldn't' be logged is if QRecall wasn't running.

- (c) alias removal in pref

The obsolete aliases are only producing annoying log messages. This will go away in the future.

It's a harmless error. I'm planning to remove these kinds of errors in a future version.

These occurs when some process deletes or replaces a file between the time that QRecall reads a folder to determine which files need to be captured, and it actually captures the file. If the file has been removed or replaced in the mean time, the OS returns a "file not found" error, QRecall logs it, and moves on.

For files that are contently being replaced (like the iCal preference file in your example), this is fairly typical occurrence. What QRecall needs to do is find out if the file has been replaced with something else and capture that instead, or record that the file has been permanently deleted.

It has no impact on the integrity of the archive.

(any chance you could allow multiple selection and copying directly from the QRecall access version of the log?):

It's already on my to-do list.

A much more usable solution would be to simply get the actions to wrap up faster.

Frederic Thomas wrote:In the same vein, a "skip today" and "skip forever" button pair in the activity would be nice. If you see QRecall starting to backup a huge file you forgot to skip, you can have it done from there. The "just now" option is useful for laptops. Yes you want the backup to complete before leaving but you can tolerate that the latest Leopard preview disk of 6+ GB is not backed up today cos you're in a hurry and you can always download it later.

That's an interesting suggestion. That would be some logical functionality to add to the status window if it showed pending actions.

Frederic Thomas wrote:To be clear, I am specifically interested in the backup server mode of Retrospect. Please don't take hints from their user interface

The inspiration for QRecall was Retrospect's UI

Seriously, I once overwrote a file and decided that would be easier to retype it than try to restore it using Retrospect. That was the day I decided that there had to be a better way.

Frederic Thomas wrote:Every time something is cancelled by clicking in the little box of the activity window, it takes ages to actually do it.
The compact stage actually does not seem to be cancelable at all.

The only action that can't be canceled is a Merge. That's because the merge is deleting the directory structure of several layers and replacing it with a new layer. It's sort of an all or nothing proposition.

The Compact action is actually the most easily canceled action, with this caveat: If the compact has recovered a lot of space, that space must be overwritten by zeros to maintain the integrity of the archive. If, when you click on the cancel button the status changed to "Erasing free space", then the action has received the cancel request and is cleaning up. If it doesn't, then that's a bug I'd like to know about.

Initially the box seems to accept clicks, but now it does not anymore and it's been running for 4:15...

Once a cancel request has been accepted, the button disables itself. This indicates that the cancel request was received and reflects that fact that you can't cancel an action that has already been canceled.

How much space has the compact recovered and how fast is the drive? If your backup archive is on a USB port or network volume, things that take seconds to complete on a local drive might take minutes.

Now having this choice worries me because it means a HW failure would lead to a destroyed archive. The system should be resilient to that sort of abrupt failures.

QRecall archives are extremely resilient and specifically designed to avoid data loss even in the case of power failure or kernel panics. But one of the cornerstones of this design is that every single bit of an archive file is accounted for and is internally consisitent, which is why the compact must erase any "garbage" left over before it can close the archive.

If the QRecall action/process crashes or is interrupted for any reason, the repair command will be able to recover all of the successfully written data.

QRecall processes also interpret BSD kill signals as a cancel requests, so it's safe to shut down your system without first stopping a QRecall action. As the system shuts down, each running QRecall process will be cleanly canceled, finish up its work, and properly close its archive.

So why can't it do that by itself when I click the cancel button?

A QRecall archive that has not been properly closed cannot be used again until you perform a repair. Repairing an archive is an exacting and time consuming process, which is best avoided.

Ah, yes. (I mentally had the Used and Avail numbers from df swapped in my head).

This could simply be a problem with the base station's disk sharing. You can Verify the archive to ensure that it's sound. You might also make sure you that you've applied the latest firmware upgrade. This has solved several problems for QRecall users with the new Airport Extreme base stations.

An interesting test would be to unmount the drive from the base station and plug it directly into your Mac then see what df, Disk Utility, et. al. says about it.

I have 2 computers and I am using the same key and the same file for both.

I forgot to mention, you should really use two identity keys if you are capturing two different systems to the same archive.

The files from each computer will belong to their respective owner (in the Owners & Volumes drawer) where there's no possibility of confusing the two. Yet they will both share the same data store, with the all space saving advantages.

From your logs, it appears that you captured your home folder to the archive IncyBack.

QRecall captured (read) 41.7 GB worth of files. Of this, 11.5 GB (28%) was duplicate, so the archive only grew by about 30.1 GB.

You can see these numbers by increasing the detail level in the log viewer by one and expanding the line that says "Captured 33360 items, 41.7 GB (28% duplicate)". The details of that message will list exactly how much data was read, how much was duplicate (already in the archive), how much new data was written to the archive, and other miscellaneous details.

The "problems" encountered during the capture are minor, and I intend to remove them from the log in future versions. They occur when QRecall reads a folder to determine what files to capture, then comes back a few moments later to actually capture them. If another process has deleted or renamed the file in the intervening time, QRecall logs a "File not found" error.

I have plans to eventually extent both filtering and capture item selection to include a vast array of rules, probably based on Spotlight queries.

Frederic Thomas wrote:I think the Mac world could do with an alternative to the moribund Retrospect. There's a million of solutions to backup stuff to a local HD, but very few solutions to replicate what can be done with Retrospect backup server: network backup every N hours (or ASAP) of potentially mobile mac (and windows) clients.

Have no fear; I have Retrospect squarely in my sights.

IMHO QRecall is very close to that goal (except for Windows). What's missing is probably better collaboration between instances and "home/server" detection (if the backup volume is on an airport extreme, it works).

QRecall will eventually grow to include direct network support, so that captures and recalls can be performed remotely -- even over the Internet. (Please be patient; there's a lot of work to do between now and then.)

QRecall's duplicate data detection is (roughly) based on the principles of rsync -- although on a much grander scale. I'm sure you can see the potential for extremely efficient network-based captures. Far faster and more efficient than Retrospect or similar utilities.

Frederic Thomas wrote:However it seems 2 QRecall instances cannot access the file simultaneously, is this correct?

Two actions can't modify an archive simultaneously, but QRecall is fully aware when other actions/computers/processes are doing so and will patiently wait until the current action is finished.

Go ahead a schedule your two computers to capture to the same archive. If one doesn't finish before the other is ready to start, it will wait its turn.

I capture five systems to a single archive and all of them start nightly at 2:00 AM. They capture one at a time until all five are done.

Frederic Thomas wrote:Again, I understand the logic. I am suggesting the UI is fixed as staring at a beach ball for 1 minute and 20 second feels very long (progress bar?), especially since the size of the full disk is known...

I would have loved to have used a progress bar, but it's a catch-22. You have to read all of the folders to determine how many folders you have to read....

But I agree that it's the one point that sort of "breaks the flow" of the assistant. I'll look into ways of handling it more efficiently.