Thanks James, not what I wanted to hear but I understand what is happening now.

My main reason for configuring my mac this way is that I am planning to put install Zevo for User directories, this is a ZFS implementation from Don Bradley of <http://www.tenscomplement.com> which, in version 1 won't support booting of Mac OS X.

I am not happy with QRecall running where I can't see it because if I put my Mac to sleep while it is capturing I end up with a corrupted backup.

Thanks anyway

I have just reconfigured my Mac so that the system and user directories are on separate disks. The problem I am now encountering is that QRecall always asks for authorization when I try to capture the system disk. I have gone through an uninstall/reinstall procedure but the problem keeps recurring. Any ideas James?

Adrian

Thanks James

This is exactly what I hoped to hear from you.

Adrian

Hi James

I am not sure if this question has been answered elsewhere and I apologise for asking if it has.

If I select for backup an entire volume containing the OS and user directories, what, if any, files or folders are excluded by default? I ask this because applications such as SuperDuper automatically omit certain items, such as the Volumes folder.

Thanks

Adrian

James Bucanek wrote:
I would recommend that you NOT exclude your home folder. The Exclude items feature is designed to exclude (thus the name) items from ever being captured. QRecall treats excluded items as if they did not exist. When you capture your entire volume and exclude your home folder, you create a layer in the archive where your home folder does not exist. If you restored your system using this layer, you wouldn't have a home folder.

Set one action to capture your entire volume, and a second action to periodically capture your home folder during the day. Now you can recall from any layer and you'll get your system files up to the last day, and your personal documents up to the last hour.

James

As usual, you have hit the nail on the head. It was the exclusion of my home folder in the daily full backup that was causing the problem. QRecall has been running perfectly for several days now, and it's fast.

Thanks again.

Hi James

Thanks yet again for a speedy and detailed reply.

James Bucanek wrote:

FSEvents is not infallible and QRecall only trust it for a limited amount of time. The default is to trust FSEvent change history for 13.9 days, which will cause QRecall to perform an exhaustive search of your entire folder tree about every two weeks. This setting can be changed using the QRAuditFileSystemHistoryDays advanced setting. So the fact the QRecall occasionally runs out and rescans everything isn't, by itself, surprising.

I understand this and could I perhaps make the suggestion that when QRecall decides it is time to go rummaging, it flags up some sort of warning so that this deep search can either be postponed or rescheduled. Would it be possible to have a preference so that the time when this occurs could be set. It could be very inconvenient if QRecall embarks on an action that could last many hours just as you need to shut down your machine.

James Bucanek wrote:

The history of FSEvent information is kept on a per-captured-item basis, and can't always be applied to future captures. This means that QRecall might scan more folders than you expect if you have mixed or overlapping capture actions defined. For example, if you have one capture action that captures your entire home folder and another that captures just your Documents, capturing your Documents folder doesn't help the history that's saved for your home folder. The next time you capture your home folder, it will rescan all of the historic changes in your Documents folder too because it can only use the history information that entirely encompasses the item that it's capturing. I admit that this is confusing, but it has to work that way or QRecall might miss changes.

My present arrangement is to backup my entire system EXCEPT my home directory once per day, and to backup my home directory at intervals of 2 hours. As it happens both actions will occur at the same time (10:00am) so one or the other gets queued. Would it be better to make the full system action backup my home directory too, and make the separate home directory backup skip the 10:00am one.?

James Bucanek wrote:

QRecall will ignore the FSEvent information and perform an exhaustive scan of your folders if the previous layer was incomplete, or contains any information from layers that are incomplete or marked as damaged by a repair. Items marked as "-Damaged-" are always recaptured in their entirety.

This may well be what has been causing me problems because sometimes I have cancelled a backup. Would it avoid the full rescan if the incomplete layer is deleted? prior to the next scheduled backup?

James Bucanek wrote:
If all of the metadata for an item is identical to what's in the archive, the item is skipped. If any changes are noted, the metadata for that item is recaptured. This doesn't mean the entire file is recaptured (that's later), just that the metadata about that file is recaptured so that QRecall always has the latest information about that file.

The reading, testing, and even recapturing of metadata is pretty fast and most folders only require a fraction of a second to determine what items in that folder need to be recaptured.

If QRecall finds any changes in a file's metadata that might indicate its contents could have been modified (creation date, last modified date, attribute modified date, name, extended attributes, number of data forks, length of data forks), it proceeds to recapture the data for that item. This consists of reading every byte of the file and comparing that to what's already in the archive. This is the data phase of the capture, and the one that takes the most time.

Yes I understand this and it's what I had expected QRecall to do.

James Bucanek wrote:If you believe that QRecall is recapturing file items that it should be breezing past, you can find out why with a little sleuthing.

The advanced setting QRLogCaptureDecisions (see the Advanced QRecall Settings post) will log the reason that QRecall decided to capture each item. Note that it only logs the first reason; there could be more than one. This will tell you something about what it is about the item that triggered QRecall's decision to (re)capture the item. Warning: This setting logs a ridiculous amount of information to the log file, so don't leave the setting on once you've found the information that you're looking for.

If you find that all of these files have been really been modified, then I would go hunting for some background or system process that is surreptitiously rummaging around your file system in the background.

This thought had passed through my mind, and the problem only seems to have occurred on the Mac Pro since I started playing around with Path Finder but I can't believe that is tinkering with the file system in such a way that so many files need to be checked. Neither does it explain why my QRecall on my wife's Macbook today suddenly decided it needed to to rummage through 1GB of files and save just 80MB or so of modified files.

Anyway, thanks again for all the information. I will try the QRLogCaptureDecisions idea if the problem persists.

I love this application and I am sure my copy of QRecall and I can come to some sort of mutually acceptable way of working

Adrian

Hmm, bad practice replying to my own post, but I am adding to it rather than replying.

As I understand it QRecall uses FSEvents to monitor changes, but FSEvents only logs the fact that the contents of a folder have changed, and it is then down to QRecall to examine the contents of the folder and decide what needs to be updated in the archive.

I have probably got this completely wrong but from my experience on my mac Pro and indeed subsequently on my wife's Macbook, it would appear that QRecall actually has to compare the archived file to the current one, byte by byte, and it seems to do this even for files that have unchanged created and modified dates. If this is indeed how QRecall works I can see horrendous problems with folders that contain, say, many very large files which rarely change, and perhaps a few small text files which change frequently.

Please tell me I am wrong and why my machine seems to be behaving as I have described.

Almost every time I run a capture from my Mac Pro to my backup drive connected to my Mac Mini I seem to end up with the Mac pro having to download almost every file from the backup in order to compare it to the original, and the number of changed folders seems to be extremely high.

For instance, having performed a capture of my user directory with only a few changes which was done very quickly, when I next tried, barely 45 minutes later having done nothing more than checked my e-mail and browsed a few web sites, QRecall informs me 801 folders have changed, and then embarks on this comparison exercise on files in folders that haven't been opened or changed in years. The archive verifies OK.

Any ideas?

After verify indicated a damaged archive, and the auto repair didn't work. I resorted to the "Copy recovered content to new archive" option, which successfully created a verifiable archive, which works fine. However, as can be seen from the screen grab below, there is an unknown damaged layer that I don't seem to be able to remove. although it doesn't seem to be doing any harm I'd like to be rid of it if possible. Any ideas?

I have now installed the beta on all 3 machines and I have done away with the separate archives for music etc. The speed of the beta has made these redundant, and simplified the management of the archives. At the moment I am doing rolling merges, compact and verify daily, but once I am happy that everything is stable I will ease back on the frequency of these actions.

James

I have a copy of the beta and now that I have everything up and running smoothly with the latest release I'll certainly give the beta a try. Certainly, if the time taken to open a large archive is reduced it would be good to do away with the separate archives. Yes, I have set up regular verifies on all the archives on the Mac mini.

Thanks for your help.

It is barely 48 hours since I started experimenting with QRecall, and I bought a license after just 24 hours. I can only say thank you to James for developing such a wonderfully versatile piece of backup software. I have 3 Macs, one of which, a mac Mini, acts as a small server and to which is attached my Firewire backup drive. The other computers, a Mac Pro and my wife's macbook are connected to the server wirelessly.

We all know wireless connections are relatively slow, especially for running things like the merge and compact functions, but thanks to QRecall I have set these operations to be carried out from the server in the wee small hours when the other Macs are normally disconnected.

To keep the backup processes fairly slick on my Mac Pro, I perform a full backup of the boot volume once a day and 2 hourly backups of my user directory, but I have excluded three large folders from my user directory which only change infrequently, namely Music, Pictures and TV Programmes. These last three folders are backed up just once per day to their own archives. The effect of this is that my full volume archive is now more manageable and so connecting to it is relatively quick, important for the 2 hourly backups, it was taking about 5 minutes to connect when everything was in it.

On the Macbook, I have a similar strategy, but as my wife doesn't have much in the way of big files, her whole disk backup is quite small.

I can't see anything much wrong with this approach but if anyone has any suggestions I'd like to hear them.

Adrian