(Replying to my own post...)

James Bucanek wrote:If you restored your system using this layer, you wouldn't have a home folder.

It also occurred to me that this is one more way in which QRecall will be forced to recapture everything in your home folder.

When you capture your entire volume and exclude your home folder, it's just as if your home folder doesn't exist. The next time you capture your home folder, QRecall sees a brand new folder with thousands of items in it! It has no history or metadata to compare with, so it performs a first-time capture of every file.

Adrian Chapman wrote:Would it be possible to have a preference so that the time when this occurs could be set.

I'll add that to the wish list.

It could be very inconvenient if QRecall embarks on an action that could last many hours just as you need to shut down your machine.

That's what the Stop and Reschedule action menu item (in the monitor window) was designed for. If it's time to shutdown, stop and reschedule the action to run in a few minutes, or at least enough time to shutdown. When the system reboots, the action will pick right back up where it left off.

My present arrangement is to backup my entire system EXCEPT my home directory once per day, and to backup my home directory at intervals of 2 hours.

I would recommend that you NOT exclude your home folder. The Exclude items feature is designed to exclude (thus the name) items from ever being captured. QRecall treats excluded items as if they did not exist. When you capture your entire volume and exclude your home folder, you create a layer in the archive where your home folder does not exist. If you restored your system using this layer, you wouldn't have a home folder.

Set one action to capture your entire volume, and a second action to periodically capture your home folder during the day. Now you can recall from any layer and you'll get your system files up to the last day, and your personal documents up to the last hour.

I have plans for a new "Ignore" feature that will do what you're trying to use the Exclude filter for, but it hasn't been implemented yet.

This may well be what has been causing me problems because sometimes I have cancelled a backup. Would it avoid the full rescan if the incomplete layer is deleted? prior to the next scheduled backup?

If that's what's causing the rescan, then yes.

This thought had passed through my mind, and the problem only seems to have occurred on the Mac Pro since I started playing around with Path Finder but I can't believe that is tinkering with the file system in such a way that so many files need to be checked.

Hard to say. Since Path Finder (ironically, one of my first Apple ][ applications was a program called Pathfinder for ProDOS) is a file browser, it might (like the Finder) store information in invisible files (like .DS_Store) or use extended attributes. Any changes like this would cause QRecall to recapture items.

I love this application and I am sure my copy of QRecall and I can come to some sort of mutually acceptable way of working

Greg,

QRecall 1.2 requires OS X version 10.5 or later.

Supporting 10.4 has become increasingly difficult and my installed base of 10.4 systems is currently less than 7%. In all likelihood, future versions of QRecall will require 10.5 or later.

Archives created with 1.1.4 can be used by all later versions of QRecall. QRecall 1.1.4 can still use archives created/updated by 1.2, within the limits outlined in the release notes. Most notably, 1.1.4 can not open or recall from archives that are significantly larger than 2TB.

Adrian,

All good questions. My short answer is that I don't know why the operating system is telling you that hundreds of folders have changed (beyond the obvious reason that hundreds of folders did, in fact, change). QRecall only recaptures items under certain circumstances (which I explain later), but it should not try to recapture an item that hasn't been touched in anyway.

Let me see if I can accurately explain exactly what QRecall does and why.

Recapturing items can be largely divided into three phases: Folder changes, metadata, and file data.

In the first phase, QRecall requests the list of potentially modified folders from the FSEvents service. As you correctly observed, FSEvents only produces a list of folders that might contain changes. It's QRecall's job to examine the contents of those folders to find out what, if anything, has actually changed.

While that sounds simple enough, these things never are and there are a few caveats that you want to be aware of:

FSEvents is not infallible and QRecall only trust it for a limited amount of time. The default is to trust FSEvent change history for 13.9 days, which will cause QRecall to perform an exhaustive search of your entire folder tree about every two weeks. This setting can be changed using the QRAuditFileSystemHistoryDays advanced setting. So the fact the QRecall occationally runs out and rescans everything isn't, by itself, surprising.

The history of FSEvent information is kept on a per-captured-item basis, and can't always be applied to future captures. This means that QRecall might scan more folders than you expect if you have mixed or overlapping capture actions defined. For example, if you have one capture action that captures your entire home folder and another that captures just your Documents, capturing your Documents folder doesn't help the history that's saved for your home folder. The next time you capture your home folder, it will rescan all of the historic changes in your Documents folder too because it can only use the history information that entirely encompasses the item that it's capturing. I admit that this is confusing, but it has to work that way or QRecall might miss changes.

QRecall will ignore the FSEvent information and perform an exhaustive scan of your folders if the previous layer was incomplete, or contains any information from layers that are incomplete or marked as damaged by a repair. Items marked as "-Damaged-" are always recaptured in their entirety.

So, now that QRecall has its marching orders, it's time to examine the individual items in each folder. This it the metadata phase. QRecall starts by comparing the metadata for each item with what's in the archive. "Metadata," for those new to this, is "information about information." In this case, the metadata of an item is things like it's name, created date, last modified date, ownership, permissions, extended attributes, launch services attributes, custom icon, and so on. It's basically everything that the operating system knows about the item except its actual contents.

If all of the metadata for an item is identical to what's in the archive, the item is skipped. If any changes are noted, the metadata for that item is recaptured. This doesn't mean the entire file is recaptured (that's later), just that the metadata about that file is recaptured so that QRecall always has the latest information about that file.

The reading, testing, and even recapturing of metadata is pretty fast and most folders only require a fraction of a second to determine what items in that folder need to be recaptured.

If QRecall finds any changes in a file's metadata that might indicate its contents could have been modified (creation date, last modified date, attribute modified date, name, extended attributes, number of data forks, length of data forks), it proceeds to recapture the data for that item. This consists of reading every byte of the file and comparing that to what's already in the archive. This is the data phase of the capture, and the one that takes the most time.

If you believe that QRecall is recapturing file items that it should be breezing past, you can find out why with a little sleuthing.

The advanced setting QRLogCaptureDecisions (see the Advanced QRecall Settings post) will log the reason that QRecall decided to capture each item. Note that it only logs the first reason; there could be more than one. This will tell you something about what it is about the item that triggered QRecall's decision to (re)capture the item. Warning: This setting logs a ridiculous amount of information to the log file, so don't leave the setting on once you've found the information that you're looking for.

If you find that all of these files have really been modified, then I would go hunting for some background or system process that is surreptitiously rummaging around your file system in the background.

Gary K. Griffey wrote:I am getting a QRecall application crash in beta versions 1.2v6 and now in the just released 1.2v8. It occurs when the Re-Index operation is performed.

Thanks, Gary.

This is a known bug. It occurs with the Repair command too. And sometimes the first time you choose either of these commands nothing happens, but the second time it works. There's also a related bug that can cause the QRecall application to crash when closing an archive window.

These are caused by a change in OS X that happened around version 10.6.2 which changed the order of events that occur when opening and closing a window. While arguably an improvement, it trips QRecall up.

I've been ignoring this issue because the QReacall user interface is being largely rewritten as I write this. The new code should replace the buggy code and (hopefully) won't have any of the same problems.

Greg Morin wrote:Does this mean it is possible for two different computers to write to the same archive (say on a shared mount point on a server) simultaneously?

No. Only one action (capture, merge, compact, delete) can modify the contents of an archive at a time.

Any number of processes can read an archive simultaneously. So it's possible to browse, recall from, and verify an archive all at once.

If not, what would happen if User A tries to open the archive to write to it while User B is in the middle of an archiving process using that archive?

User A will automatically wait until user B is finished. If you're opening the archive to view its contents, QRecall will present a dialog that will politely wait until the archive is available, or until you decide to give up and close the window. Similarly, if have an archive window open and start a (local) action that wants to modify it, QRecall will automatically close the window so the action can proceed.

Gary K. Griffey wrote:So...just to summarize...(I promise )...previously verified layers in the archive were indeed corrupted...but not via the last Capture operation...through some other, as of yet, unrecognized "event"...that is my final "takeaway" from your comments...correct?

Correct. And should have let you write the reply; that was so much more succinct.

If so..this makes me feel completely confident in your product once again going forward...expect several $40 licenses fees from my clients in the near future...

Always good news.

(Yikes, this really should have been it's own thread.)

Gary K. Griffey wrote:Thanks for the info...possibly, you could assist me in better understanding the available Repair options...it seemed to me that after executing the Repair operation...previously verified layers in the archive were indeed compromised by this subsequent re-Capture operation...which is very much contrary to what you have stated...possibly (very likely) a user error on my part when running the Repair.

None of the above; the previous verified data simply stopped verifying. A capture action only adds to an archive, it doesn't rewrite portions of the archive that have already been captured. If a previously captured and verified data record later fails its verification, then it's because some external cause (magnetic media failure, a glitch in the drive controller, OS bug, whatever) either lost, changed, or overwrote that data. The last thing to happen to an archive isn't necessary the cause.

The various repair options (which I'll explain in a moment) aren't to blame, nor can they pull valid data out of thin air.

As you recall...I ran the Repair operation against the archive after the verify failed...

That's absolutely the correct thing to do.

I ran it using only the first default option..."Use auto-repair information"...you further stated...from examining the log files that I attached...that the Repair was successful. However, after the Repair operation completed...I opened the archive...and saw 3 of the 4 layers now showing "Damaged Files"...which turned-out to be the large virtual machine package...(the most important reason for running the backup in the first place, by the way).

This is also correct. The archive is now "repaired" in that it is valid and internally consistent. However, data was damaged. Lost data can't be magically reinvented. QRecall examined the records and concluded that the data block beloning to two versions of your VM file were lost. So the two file records that contained the damaged data blocks were expunged from the archive, and the folders that contained those files was marked as "-Damaged-" indicating where the data loss occurred.

Thus it appeared to me that the VM package in layer 2 and 1...taken 3 and 4 weeks ago respectively...had been compromised...and thus my assumption that previously verified backup layers had now been corrupted.

That's correct. The data you captured a few weeks ago became damaged at some point. The verify alerted you to the issue, and the repair recovered the files and folders that weren't damaged.

Looking at the log, only a single data record was corrupted in your archive. Most likely, this impacted a single data block belonging to different versions of the file in the two earliest layers. That data didn't belong to later versions of the file, so the subsequent layers were unscathed.

Should I have run the Repair with different/additional options selected?

Not really, unless you're desperate.

The repair options are:

Copy recovered content to new archive: Use this only if you do not want to touch the damaged archive in any way. It's useful for testing repair settings (since it doesn't change the original archive) or repairing an archive on read-only media.

Recover lost files: If directory records in the archive are destroyed, it may leave "orphaned" file records. That is, file records that aren't contained in any folder. This option scrapes the archive and assembles all orphaned files together in a group of special "recovered" folders. Note that this may resurrect files previously deleted by a merge action.

Recover incomplete files: If a data block belonging to a file is lost, the file is deleted. With this option turned on, the file is kept, marked as "damaged" and all of the remaining (valid) data block are assembled into a file. The file is still incomplete—the lost data is still lost—but the remaining data is recovered. The file probably isn't usable, but might contain usable data.

This last option is probably the one you're most interested in, but it still can't recover the entire file. It can only recover the portion of the file data that wasn't lost.

Would this have preserved my 1st and 2nd layers?

No. Lost data is still lost data. You can't get it back once it's gone. Using the last option, you can get everything else in the file that wasn't damaged back, but that's usually of limited value.

Gary K. Griffey wrote:By the way...I ran Disk Utility on the target laptop's hard drive...the one that experienced the corrupted archive...and there were no issues with the drive at all. I also checked disk permissions...no issues. So...I guess at this point..there is really no known explanation for what occurred with this particular incremental Capture that evidently corrupted the entire historic archive.

It's often hard to isolate the cause of a single, or even a series, of random data failures without more information. The basic problem is that "stuff happens." Data storage and transfer is not nearly as perfect and repeatable as most people assume. Data in magnetic media—despite all of the clever tricks employed by modern drives to avoid it—gets lost from time to time. Data flying through USB, Firewire, SATA and over WiFi doesn't always arrive the way it was sent. And consumer-grade dynamic RAM is susceptible to the occasional bit-flip and corruption by cosmic rays.

The "problem" with QRecall is that it is almost alone as a backup solution in that it attaches 64-bit checksum to every record of data it creates, and verifies the integrity of that data every time it reads it. When QRecall reports damaged data, it inevitably results in the initial impression that QRecall is failing or is somehow failing to protect your data, when in fact it's most likely the media/controller/interface/RAM/CPU/OS that's damaging the data. What is (or should be) frightening is that so many other so-called "reliable" backup solutions make no attempt whatsoever to protect against, detect, or report any data loss. Solutions like Time Machine simply copy files and hope for the best. They couldn't tell you if your files were successfully and accurately copied if you wanted them to.

I guess that is the issue that I see moving forward using QRecall for backups with my clients...it would appear that each and every incremental Capture basically puts the entire historic archive at risk of total loss...

Not true at all. QRecall is specifically designed to protect against partial loss of data in an archive. Damaging part of an archive in no way impacts the integrity, or recoverability, of the rest of the archive.

this is unlike many other imaging products that I have used in the past where incremental backups are always written to a new physical file...

QRecall doesn't write files, it writes data records. Each data record is small, self contained, and independently verifiable. When the data in an archive is damaged, the repair process reads and verifies every record. It then reassembles the valid records into a usable archive again.

It doesn't matter if a million records were written to a single file or a million individual files. (Writing to a single file is more efficient and ultimately safer, which is why QRecall does it that way.) Either each individual record is valid or it isn't. QRecall does not relay on the file system's directory structure to organize its information.

and thereby do not jeopardize the integrity of the historic backups currently in existence.

The integrity of the archive's history is never in jeopardy. QRecall's layer records are specifically designed to resist corruption through partial data loss. It employs a method of "positive deltas" that re-record critical directory structure information in every layer. So the loss of data in an older layer won't impact the structure or integrity of subsequent layers.

I realize, of course, that I could simply make a copy of an existing archive each and every time before a subsequent Capture is performed...

No need, QRecall's already doing that behind the scenes. Most actions begin by duplicating key index files within the archive (if you peek inside an archive package during a capture or merge you'll often see temporary "_scribble" files appear). New data is appended to the primary data file. If anything goes wrong (power loss, OS crash, ...) the partially modified files are summarily discarded and the primary data file is truncated at the point before the action began. The result is an instant rewind to a valid archive state. You'll see "auto-repair" in the log when this happens.

Don't get me wrong...I think your product has some great features...and I do appreciate your thoughts and guidance.

I hope some of this technical information will help explain the extraordinary efforts that QRecall takes to preserve your data, and detect when that data has been lost or damaged.

Gary K. Griffey wrote:I observed a system log message being repeated literally thousands of times during the capture operation.

Welcome to beta testing.

The beta version of QRecall spits out tons more console and log messages than the release version, mostly so I can diagnose problems reported by beta testers. In this case, it's the QRecall Spotlight plug-in, which is normally quite laconic.

But you make an interesting observation. (And see, one that you wouldn't have made if I left those messages off!) Normally, Spotlight shouldn't reindex the archive until the capture (or whatever) is finished. I'm surprised that it would repeatedly be reindexing the archive while a single capture was in progress.

But it's hard to tell from the fragment that you've sent. All of that activity is a single reindex. You'd need to look for multiple occurrences of the message "mdworker[xxxx] _RepositoryNamesImporter /Users/Gary/Downloads/melsLaptop.quanta/filename.index" during the course of a single capture.

If that was happening, then that's something I need to look into.

Sheldon Furst wrote:
Might I suggest that the program defaults be set somewhere in the mid range for the Archive settings so that new users don't feel as I did?

That's a tough one.

It used to be set on low-to-middle by default, but people with some really large archives were getting abismal performance so I set them to off by default.

On the other hand, the duplicate data performance for large archives has improved radically since those days, so maybe the time has come to turn it back on for new archives again.

Sheldon Furst wrote:How can I make QRecall "recall" a file back to its original location? When I do a recall it always asks me where I want to recall it to. Most of the time I want to put it back where it came from.

Use the "Restore" command.

A Restore is just a recall with the destination set to the item's original location.

See Help > QRecall Help > Capture Basics > Restore for the details.

Cody Frisch wrote:I just find it interesting that here is a program that seems to be written by one guy, and its far more capable than applications written by companies with several employees. (Not to mention much less expensive.)

I'm glad to hear that. I would certainly like to have several programmers working on QRecall; I know that I'd like to add features faster.

One thing I was thinking could be useful, is compression rules on an archive. Exclude certain file types (mp3, mov, mp4) from being compressed at all since they aren't that efficient to compress again. Trade off by focusing the compression on those things which can really benefit.

I seriously considered this when developing the compression feature. I know that Retrospect (and others) have compression filters. However, I ultimately rejected it for two reasons: (a) I think it's too complicated and (b) QRecall doesn't work the way that other backup programs do.

Hopefully (a) is obvious, but (b) requires a little explanation.

Instead of turning compression on/off on a file-by-file basis, QRecall measures the cost/benefit of compression on a block-by-block basis. When you have the compression level set low, QRecall uses a fast-but-not-necessary-efficient compression algorithm on every block of data. It then examines the results of that compression. If the compression really didn't save much, it throws it away and stores the uncompressed copy of the data record. The theory is that the little bit of data saved doesn't make up for the amount of CPU required to decompress it.

In this way, QRecall is self throttling, automatically opting to compress data is the highly compressible but not compressing data that isn't. This is smarter than file-by-file filters because it can even decide that some parts of a file are compressible while others aren't.

At higher compression settings, the compression algorithm used is slower and more CPU intensive, and the threshold of for accepting compressed data is lowered.

One of the big differences between QRecall and other file-based backup software is that the data in files is compared to that data already in the archive. Using Retrospect (or Time Machine, or whatever), if you edit the metadata of an MP3 file the software will copy the entire file again. If compression is enabled, it will compress the entire file again.

QRecall, on the other hand, reads the MP3 file and compares it to what's already in the archive. Most of the file hasn't changed, so almost no new data is compressed. If the duplicate data in the file wasn't compressible, then QRecall has already determined that (when it captured the data the first time), so no compression or decompression is performed. If the data was compressible, then it only needs to be decompressed (much faster than compressing) in order to compare it with the source data before a decision is made not to add it to the archive.

So in the end, I feel that QRecall's compression features is much easier to configure, smarter, and ultimately more effective than hand-tweaked per-file compression filters.

Cody Frisch wrote:This is what I have wanted my backup software to always do.

Awesome.

I'd definitely love to see more filter capabilities, especially finder labels.

So would a number of other people; it's a common feature request. I have advanced filtering features tentatively scheduled for the 1.3 release (currently working on 1.2). A bit off in the future, but I'm sure I'll get there.

Gary K. Griffey wrote:Yesterday, after adding the 4th layer...I ran a QRecall Verify...as I always do...and the verify failed...

That's correct. The verify detected corrupted data in the archive and/or on your hard disk.

I tried to Repair the archive...but this also failed.

Actually, the repair was successful. QRecall did log warnings about the problems it found and what items were affected, but the repair finished successfully as was confirmed by the verify action that you performed afterwards.

Note that many (many!) archive corruption errors are often the side effect of a corrupted volume structure. I would encourage everyone to use Disk Utility to repair the volume containing the archive before repairing the archive. If your volume has cross-linked file allocations (for example), repairing will just set the archive up for future failure.

In looking at the Repair log entries...it appears that the rather large (about 30 GB) virtual machine disk file caused the error.

Also correct. If you did not select the option to recover damaged files, then the damaged version of that file has been deleted from your archive.

I know from other threads that you have written...that a virtual machine must be suspended or shutdown to make a valid backup of it.

That's very true. There is no backup system that can correctly copy a file that is being actively modified.

In this case, however, the laptop that contained the virtual machine was operating in target disk mode...and was mounted to my second laptop where QRecall is installed...thus...not only was the virtual machine shutdown...but OSX was quiesced on the source drive as well.

The problem wasn't that the file was being modified, but that QRecall detected that data previously stored in the archive failed its validity check(s). This can happen for a score of different reasons (data corrupted during transfer to the drive, random data loss on the drive, intermittent RAM errors, ...). But it has nothing to do with the source file or what condition it was in.

I also applaud the rigor of your backup methodology, but I personally think it's a little overkill. While it's true that you can't make a "perfect" copy of your boot volume while OS X is running, QRecall works really hard to successfully perform live captures and recalls from/to your startup volume. It's certainly problematic, and you definitely want to quit as many applications as possible (certainly any VM and disk images that you might be writing to), but it's not absolutely necessary to shutdown the entire OS to make a decent backup. QRecall has the ability to capture while you're logged out, and you can even schedule captures to run while you're logged out (or not run while you're logged in). Just food for thought.

I only mention this because I firmly believe that the back up strategy that works best is the one that gets used, and the one that gets used is usually the one that runs automatically, independent of the user. I'd be much happier with an imperfect backup that occurs every day than a perfect one that I get twice a week—if I remembered to do it. You also might consider a two-tiered backup strategy. Make regular (even hourly) captures of your regular documents, excluding things like your movie library and virtual machine images, and then continue with your complete backup strategy on a weekly or bi-weekly basis.

Now, I did run a Compact operation on the archive during the week with the 3 existing layers...and I also changed both the Compression and Shifted Quanta detection on the archive...but I never had issues doing this before to an existing archive.

That shouldn't have had any bearing on the problem you described.