I agree that QRecall needs to be a little more sophisticated about how it calculates the expected capture and verify intervals. Right now it's a weighed running average, but that obvious breaks down when archives are not captured at uniform intervals, or are not being captured at all.

I don't have a solution for an individual archive, but you can modify QRecall's archive status window behavior globally.

The warnings that a capture or verify have not occurred recently is based on a simple formula, the constants of which you can tweak using advanced settings. I'll describe how the capture formula works and how the adjust it. The verify logic is the same, it just uses a different set of constants.

(1) Every time an archive is (successfully) captured, the interval since the last capture is calculated. This updates the AverageInterval property for that archive.

(2) The AverageInterval is then compared with the global QRStatusMinimunCaptureInterval and the larger of the two values is chosen as the NominalInterval. The QRStatusMinimunCaptureInterval prevents rapid (i.e. hourly) captures from creating a ridiculously small nominal interval.

(3) The NominalInterval is multiplied by QRStatusCaptureWarningIntervals. If the time difference between now and the last time the archive was captured is greater than that product, the status window shows a "warning" (yellow) indicator for that archive.

(4) The NominalInterval is multiplied by QRStatusCaptureTruantIntervals. If the time difference between now and the last time the archive was captured is greater than that product, the status window shows a "problem" (red) indicator for that archive.

QRStatusMinimunCaptureInterval is expressed in seconds and defaults to 21600 (6 hours)

QRStatusCaptureWarningIntervals defaults to 3 (shows a warning when 3x the nominal interval has elapsed without a capture)

QRStatusCaptureTruantIntervals defaults to 7 (there's a problem when 7x the nominal interval has elapsed without a capture)

For an archive that you capture once a day and rotate once a week, you could try setting the warning multiplier to 10 and the problem multiplier to 15, like this:

defaults write com.qrecall.monitor QRStatusCaptureWarningIntervals -float 10.0
defaults write com.qrecall.monitor QRStatusCaptureTruantIntervals -float 15.0

If you capture more often than that, consider setting the QRStatusMinimunCaptureInterval too.

I don't have a solution for an archive that is never captured, except to set the intervals to excessive values (like 1000) so they simply don't occur.

The verify logic works the same, except that is uses these settings:

QRStatusMinimunVerifyInterval is expressed in seconds and defaults to 86400 (1 day)

QRStatusVerifyWarningIntervals defaults to 8 (shows a warning when 8x the nominal interval has elapsed without a verify)

QRStatusVerifyTruantIntervals defaults to 16 (there's a problem when 16x the nominal interval has elapsed without a verify)

Adam Horne wrote:Now I'm getting an error from the Qrecall log:

I'd be interested to know where in the log this message occurs. Rather than playing "20 questions" I'll ask that you send a diagnostic report (Help > Send Report...) and we'll work from there.

Adam Horne wrote:Is this possible on a RAID volume?

It should be. HFS+ encryption is performed at the filesystem plug-in level (that controls how files are organized on a device), while RAID works at the device driver level (that does the work of reading and writing data blocks to hardware).

In Disk Utility, select a volume on your RAID and go to the Erase tab. If one of the choices is Mac OS Extended (Journaled, Encrypted), then you can create an encrypted volume on your RAID.

Andrew Reid wrote:I will test out the software I will try the archive on a NFS volume with and without the sparsebundle and report back.

I look forward to hearing what you discover.

The sparse bundle has appeal as it can be encrypted, and it allows for network efficient offsite backup of the archive . I did read the note about caution in using encrypted sparsebundles as archive volumes, can you expand a bit on that ?

Sparse bundles add another layer of complexity to the filesystem that I prefer to avoid. In theory, it shouldn't make any difference. In practice, layering filesystem technologies like this can cause strange behavour that's difficult to track down.

In my latest post about sparse bundles, a user was getting strange archive verify errors. I was encouraging them to abandon the sparse bundle and use the volume directly?reducing the complexity. As it turned out, the volume/drive itself might have been the cause of the problem. They switched to a different drive and was able to write and verify the archive. I haven't heard back from them, but it probably wasn't the sparse bundle that was the problem.

Andrew Reid wrote:I would like like to use QRecall to backups to an archive stored on a NAS unit.

In general, there shouldn't be any problem. There are numerous QRecall users who backup to various NAS devices.

For various (compelling) reasons I would prefer to have the archive housed in a sparsebundle based volume which is mounted "locally" over NFS to to computer being backed up.

Just be aware that you'll be giving up some performance. NAS adds volume access overhead, and a sparse bundle adds more on top of that.

Also be aware the disk full situations could be a problem. Some NAS and sparsebundle implementations "lie" about how much free space is available on the volume. QRecall uses the volume status information to determine when an archive has filled up the volume so it can stop the capture and gracefully close the archive. If the volume lies, the archive could overfill the volume, damaging the archive and making it difficult to repair.

So to QRecall the archive is just a local volume. I can easily script the mounting of the NFS/sparsebundle combo in cron or with a pre-run script hook if QRecall support those.

Depending on your NAS implementation, QRecall may do this for you already. If the volume mounts as a networked volume, or appears (at the disk driver level) to be a physically connected external drive, QRecall will attempt to mount the volume/drive before an action begins. If QRecall mounted the volume, it will then attempt to automatically unmount it when all of the actions have run. Assuming the drive is compatible with the OS X's network volume or disk arbitration framework, this work has already been done for you.

a) Have I explained this well enough, does it make sense ?

It all seemed perfectly reasonable to me.

b) Does the archive volume even have to support any Apple-centric (HFS) features. What features does the volume need to support.

QRecall makes as few assumptions about the filesystem that the archive is located on as it can. So it should work on any filesystem that OS X supports.

It is, however, still being accessed via the Core Services framework in OS X. Apple has deprecated this framework and QRecall is currently being rewritten to use only the low-level BSD filesystem APIs. This should provide even better compatibility with foreign filesystems in the future, but for now QRecall has problems with some filesystems that aren't compatible with the Core Services APIs. For example, QRecall can't currently store an archive on a ZFS volume from TenOne at the movement because of quirks in the Core Services framework.

Adrian Chapman wrote:Why are these things so damned obvious in hindsight

Because, I fear, it's just complicated.

I've added a to-do list item to log a message when an item has been ignored for too long. Maybe that will help point people in the right direction.

Adrian Chapman wrote:Thanks for yet another in depth explanation of what is going on.

You're welcome. I should have summarized, though. The quick and dirty tip to remember is that if the capture action logs "Locating changes since ..." then it's going to use filesystem change events to speed up its search for items to capture. If it logs anything else, or nothing at all, it doesn't.

I suppose my first question is do I have anything to worry about in terms of my schedules and exclusions? The only thing I ignore changes in is my Virtual machines folder which I archive once per day as the last activity. As I don't use my VMs much would it be better to not bother ignoring them?

In principle, your capture actions are fine: you capture your user files during the day (ignoring the the VM folder) and capture your VM folder once a day.

Except, you're not. You're actually capturing a subfolder in the VM folder once a day. So the top-level VM folder is never captured (while not ignoring changes), which what's causing your ignored history information to accumulate indefinitely. At the very least, I would suggets modifying your once-a-day capture to capture the entire Virtual Machines folder/package.

When using the Ignore Changes In feature of the capture filter, I find this technique the easiest way to manage it:

- Create the action you want to capture your top-level folders/volumes, and set the Ignore Changes In to those sub-items you're not interested in tracking during the day. Set whatever items you want to exclude, etc.

- Schedule that action to run at regular intervals (once per hour, or whatever). Save it.

- Duplicate the previous item, remove all of the Ignore Changes In items, and change the schedule to run once a day (or so). Save it.

Now you're sure that the second action will capture (and exclude) exactly the same items as the more frequently run one, but will NOT ignore the changes any any subitems. This gives the archive a chance to get up-to-date with all of the items once a day.

As you can probably see from the logs, my System is on it's own drive, in fact now an SSD, and the users are on a Softraid RAID 1 array. The system volume and Users directory are saved to separate archives, which I thought to be a good way of doing things, i.e. keeping archives specific to volumes.

I think that's just dandy. Whether you keep one archive per volume, or capture multiple volumes to a single archive, is entirely up to you and how you want to manage your actions and backup space. QRecall is designed to be flexible and tries not to dictate any source/archive relationships.

Adrian,

I've analyzed your log files and it would appear that QRecall is performing exhaustive scans of your directory structure on a pretty regular basis (see attached file).

You have a fairly complicated capture schedule, and that's causing a lot of captures to simply ignore the file system events history and scan the item directories in their entirety.

For QRecall to use filesystem event history, it must first capture an entire directory structure. This creates a history "bookmark" in the layer. The next capture examines that bookmark and uses it to query the filesystem event's manager for changes that have occurred since, and then it only has to review the folders that are known to have changed.

This is greatly complicated, however, when captures overlap (capture a volume, then later capture one or more subfolders), or if items are being ignored (using the new "ignore changes") features. This requires QRecall to determine the largest region of overlap, and it must also replay the history from the oldest captured items in the folders that are being ignored.

All of these conditions require QRecall to request change information for a much larger period of time from the filesystem history manager. If the manager doesn't have that much history, there's no history "bookmark" in the layers that intersect this capture, changes in specific folders have been ignored too long, the geometry of the overlapping items is too complex, any intersecting layers were damaged/repaired, or the volume doesn't support FSEvents, then QRecall simply ignores the filesystem change history and performs a deep scan. It doesn't log any messages about gathering, or ignoring, changes because it's not using change history at all.

I prepared a summary of your captures since April and attached it to this post. The captures marked with "- no FSEvents" are captures that are not using filesystem changes at all (for any of the above reasons), which means they preformed an exhaustive scan of the directory structure. The "- deep scan" captures were using change history, but it expired (QRAuditFileSystemHistoryDays) and a deep scan was performed anyway. All of the unmarked captures used filesystem change history to perform a quick scan.

The report was created by first discarding all captures that reported a warning or error (for simplicity), then finding all captures that reported to be using filesystem change history, those that explicitly ignore it (trust expired), and then those that don't log any messages regarding filesystem change history at all. The later are marked "- no FSEvents" and would have performed a deep scan.

Adrian Chapman wrote:QRecall is definitely not performing regular exhaustive scans, at best it is once per month and in May it didn't do it at all.

That's sounds unusual. It would be helpful is you sent a diagnostic report to review.

I should add that I have had cause to perform repairs a few times so would this reset the count down for FileSystemHistoryDays?

That could be a factor. Some repair conditions cause the history information in a layer to be discarded (because it can't be trusted now). This will cause a deep scan of the captured items, but will require a little sleuthing to find. It results in a capture action that doesn't log anything about the change history. So you're looking for a capture action that doesn't say either "Filesystem change history..." or "Locating changes since...".

Adrian Chapman wrote:I have a feeling that QRecall isn't performing an exhaustive scan as often as it should.

I hate that feeling.

What am I looking for in the logs which will confirm it is doing this approximately once a week with the default setting of 6.9 days?

The message "Filesystem change history expired" will appear in the log when the archive have been relying on change history information for longer than the trust limit setting. This will replace the normal "Locating changes since..." message.

The log entry is classified as "minutia," so you'll have to set your log window detail to its highest setting to find it. If you want to find them all quickly, run the detail slider to max and enter "history" in the search field.

Also is it possible to force an exhaustive scan?

If the last layer(s) containing the same items being captured now was incomplete (or damaged), QRecall forces the current capture to ignore the change history and perform an exhaustive scan. In this situation the message "Filesystem change history ignored" appears in the log.

To force a scan, start a capture, wait for a few items to be added, and then stop it. Start the same capture again and it perform an exhaustive scan of the directory. You can then merge the two layers, if you like neatness as much as I do.

(Don't cancel the capture before it's had a chance to add at least one new item to the archive. If the capture doesn't capture anything, no new layer is created, which means there won't be an incomplete layer to trigger to deep scan.)

Chris Petit wrote:I was wondering if two computers with two different identity keys can capture to an archive at the same time.

Not at the same time. Only one action can update an archive at a time.

If not, what is the best method for allowing them to take turns backing up automatically?

Set both computers to backup to the same archive. QRecall will arbitrate and make them take turns.

CaB Studios wrote:We're just south of Birmingham in the UK. You just had to rub it in and mention 'sunny'! Wettest summer ever!

Thankfully we're all feeling very proud of the best Olympics ever and Andy Murray finally winning a major! Woohoo!

I still have synchronized swimming saved on my DVR. Trying to make it last a little longer...

I'm thinking of rotating 2 x 4TB FireWire 800 drives to perform the archives so I can take them offsite. Does this sound sensible in terms of best practice?

There are action schedule features designed specifically for that. There are basically two approaches:

The first is to create capture actions that run on a regular schedule for archive A. Add a schedule condition to ignore the action if the archive isn't available. Duplicate all of those actions, then set the target of the duplicate actions to archive B. Now you have two concurrent set of actions that will run automatically, but only those that apply to the currently connected archive will run. You can then rotate your drives on whatever schedule you like.

The second alternative is a little more immediate. Create event-based schedules that run the capture action when the archive is mounted. Then your capture A action will start immediately when you plug in backup drive A. Eject A, put it in a safe place, plug in backup drive B, and the B capture starts immediately. Event actions can optionally be set to repeat if you leave the drive plugged in.

CaB Studios wrote:I'll have to wait for this mammoth 700GB to backup first. It's done 260GB so far and has been running for 3hrs 55mins.

Yes, a 700GB capture is going to take awhile.

If QRecall ever gets in the way to doing something, you can always stop the capture and reschedule it to run later. It will leave an incomplete layer in the archive, but the next capture will still take advantage of the work that's been done and pick up where the last capture left off. (You can merge the two layers if you don't like incomplete layers in your archive.)

Are you guys in the US I take it?

Sunny Arizona!

Then maybe I've misunderstood the problem.

If you scheduled an action to run, and it didn't (nothing in the log when set to the highest detail setting and the filters set to show all activity), then please send a diagnostic report (Help > Send Report...). It may contain some clue as to why the actions didn't run.

CaB Studios wrote:

Move the detail slider in the log window to the right. My guess is the missing actions will appear in the log.

No one was logged in at 2am.....

QRecall doesn't normally run actions when the user is logged out.

If you want it to, set the Start and run actions while logged out option in the Scheduler preferences. This installs the scheduler process as a system daemon that runs even while you're logged out.

You can actually schedule an action to run when you log out (or in), if that's useful. There are also power management options that will wake/start your computer before an action runs, or shutdown/restart/sleep your system after it's finished.