Prion wrote:there are several incomplete layers of capture after a certain date.

An incomplete layer is just that: QRecall stopped capturing to the archive before it was finished. It just means that one or more items modified before the date of the layer were not captured. The missing items will be captured by the next (complete) capture.

1) Any ideas what may have caused the problem technically and how to avoid this in the future?

An incomplete layer is created when the QRecall capture is stopped before it can finish. There are a variety of events that can cause this to happen:
o You stopped the capture action directly (clicking on the stop button or sending the process a TERM signal) or indirectly (log out, shutdown, restart).
o The capture ran into a problem that prevented it from finishing, like running out of disk space.
o The capture's action has a condition that automatically stops the action at some time.

If this started to occur after you enabled Time Machine backups to the same device, you might have run out of disk space. This will prevent both QRecall and Time Machine from completing their backups. Of course, QRecall tells you about this, while Time Machine just deletes data or stops.

2) QRecall can backup over WiFi, can it not?

Yes.

3) Delete the incomplete layers? Or do they still contain valuable information?

The layers contain the items that QRecall could capture before it was stopped. Whether that's "valuable" is entirely up to you. The only danger of incomplete layers is when recalling; if you recall an incomplete layer you may be missing items that were modified before that layer date, but were not captured.

If incomplete layers bother you, simply merge the incomplete layers with the next complete layer in the archive. You'll discard any intermediate versions of items that were captured in those incomplete layers, but at least you won't have any incomplete layers in the archive.

Prion wrote:There may be a simple thing causing all this but I am at a loss.

I wouldn't bet on "simple."

There are a bewildering number of issues that can cause an archive to become unusable. The most common is a corrupted volume structure. This can cause the operating system to report I/O errors, mysterious "Cannot convert FSRef to Path", data validation errors, and items to disappear in the Finder—I'm not sure if this is what you meant when you wrote "Archive disappeared."

Once QRecall detects any inconsistency in an archive it will refuse to use it until it can be repaired. So the general formula to dealing with these kinds of problems is:

(1) Repair the volume that contains the QRecall archive using Disk Utility.
(2) Repair the QRecall archive using the File > Repair... command.

The original/root cause of the problem can probably be found in the QRecall log. It will most likely be the first error or warning that immediately precedes the spat of problems you're having now. If you get stuck trying to figure it out, send a diagnostic report (Help > Send Report).

Prion,

I received your diagnostic report. The "Unable to read extended attribute data" message occurs when QRecall tries to get the attributes of your remote volume—the volume containing the archive. QRecall uses some attributes of the archive volume to determine compatibility with advanced features.

I've never seen this happen before. I suspect that it's an idiosyncrasy of the TimeCapsule.

Anyway, it's harmless. The extended attributes of your archive volume's root directory aren't important to QRecall.

Prion wrote:ah, bummer, although I half expected that. Does the same hold true for the TimeCapsule itself? Never tried that one because it keeps being used for TimeMachine backups of the family's laptop.

To the best of my knowledge, all of the volumes in and connected to a TimeCapsule are accessed the same way.

Regarding the extended attributes I must admit I never noticed the little details slider in the upper right corner of the log window. It helps a lot to pull it all the way to the right

It reads:

xattr:com.apple.system.Security
errno: 1

Errno 1 is "Operation not permitted" which is pretty generic. Since QRecall is trying to read a security attribute, I assume it has something to do with the security services, but I have no idea what and I've never had that happen here.

If you send me a diagnostics report (Help > Send Report) I might be able to tell more. Let me know if repairing the volume makes any difference.

Prion wrote:- QRecall cannot find the archive when I try to capture to the drive connected to the TC while logged out

That's expected; OS X can't mount a network volume unless you are logged in.

A Time Machine volume appears as a networked volume. Networked volumes require a user and password to mount, which requires that either you enter the name and password via a dialog or you save the name and password on your keychain. Both of those solutions require that you be logged in.

- when logged in, the capturing takes place normally, however, the log specifies that "Unable to read extended attribute data". This does not happen when plugging the drive directly into the MBP

That message says that the operating system reported that a file or folder has extended attributes, but when QRecall tried to read them it returned an error instead. You might trying repairing the volume and see if that makes any difference.

The file that encountered the problem, the name of the extended attribute, and the error returned by the operating system are included in the details of the message. If you need help deciphering the cause or its implications, send a diagnostic report (Help > Send Report) and I'll take a look at it.

Daniel Newman wrote:Thanks for your reply. So basically I need to install QRecall on each computer then. The backups go into one archive, but segmented by different owners (i.e. each computer) and the backups instigated by each computer individually. Is this correct?

Perfect.

So there is no real network clients or cannot be instigated by one computer/server only?

Not in the current version. The current version shares archives via file sharing. There's no "backup server" in the traditional sense.

Daniel Newman wrote:So far in my testing QRecall seems to be a fantastic and flexible backup Program.
I am seriously considering to purchase and use it for backing up a small office, which includes of 2 main computers and a server.

Daniel,

You do need three keys, if you want to capture all three computers (two main and one server) to the same archive. Install one key on each computer.

Every item captured in an archive belongs an owner. An identity key identifies the owner. To see the owners in an archive, open an archive and choose View > Show Owners and Volumes.

Let's say you install three different identity keys on three computers and name them "A", "B", and "Server." When computer A adds its files to the archive, they appear in the archive under owner "A". Similarly, the same files captured using the server appear in the archive under owner "Server."

An identity key uniquely identifies an owner and is used to "tag" the ownership of every captured item. You can change the name of an identity (QRecall > Preferences > Identity Key). Its name will change in the archive, but it's still the same identity/owner and any items captured using that identity key still belong to that owner.

You only need an identity key to capture items. If the server isn't capturing its own files (just hosting the archive file and running maintenance actions), it doesn't need to have an identity key.

Tip: If you're using the Capture Assistent to set up your actions, delete the merge and compact actions created for the two networked computers. Those two only need capture actions. If they are all sharing the same archive, only one computer needs to be performing the regular maintenance (merge and capture) actions, and the one directly connected to the drive containing the archive is the most efficient.

Axel Munker wrote:I don't know if it's only me,

It's not just you...

QRecall sends signals to the operating system while it's working to keep the computer awake. This was done because a few early QRecall users were complaining that their computer was going to sleep in the middle of an action. I've never seen it happen here, and I suspect that it depends on certain combinations of OS and hardware. These "stay awake" messages have the unfortunate side effect of also waking up your monitor.

You can turn this feature off with an advanced QRecall setting. Close QRecall, open a Terminal window, and issue the following command:

defaults write com.qrecall.client QRRunningMakesSystemActive -boolean false

In version 1.2 I've put this option in the QRecall preferences window, making it much easier to set.

Charles Watts-Jones wrote:OK I understand this and even suspected it though it doesn't explain why two verifies (2 & 5 January) started at 6:55. Or does it?, it's too late here to do the maths.

The action is probably scheduled to start at 6:10 but isn't starting until 6:55. The action might be waiting for the archive or maybe the computer is asleep and that's the time you have it programmed to wake up.

By the way my own machine uses the same verify routine and has had no troubles (so far as I'm aware). Crazy world, eh?

It is. And as soon as I find a better one, I'm moving.

Charles,

You've stumbled into the crazy world of human dates and times.

An interval schedule (like every 3 days or every 90 minutes) is based on fixed time intervals from a particular point in time. You selected 28-Jul-2008 7:10 as your anchor time. QRecall then calculates 72 hour intervals from that time.

Unfortunately, 7:10 in July is an hour different from 7:10 in January because daylight savings time shifts the clock between summer and winter. QRecall is calculating an exact number of 72 hour intervals from 28-Jul-2008 7:10 and that occurs at 6:10 this month.

Solutions:

You could pick a starting time in the winter (say 1-Jan-2010). That would start the verify at 7:10 in the winter and 8:10 in the summer.

I'd suggest using a Daily schedule and choose two days a week to perform the verify. The Daily schedule always uses the localized time for "today", so the actual time shifts automatically according to your local time zone. For example, if you moved from France to Mexico, your capture will still run at 7:00 Mexico time (CST), whereas your current interval schedule would start running your verify at 1:10 at night (but still 7:10 in France in the Summer).

(I wish I could go back in time, find the guy who invented daylight savings time, and imprison him on a desert island where he can't do any harm.)

Bruce Giles wrote:If I boot from the external drive, running Tiger Server, will there be any problems with using that system to restore Leopard Server onto the internal drive?

I suspect that this won't work. While the file system differences between Tiger (10.4) and Leopard (10.5) are not at huge as those between Leopard and Snow Leopard (10.6), Leopard introduced access control lists (ACLs) that I believe are critical for a few key OS files. Tiger lacks the APIs for restoring ACLs, so QRecall will ignore ACLs captured by a Leopard system.

My suggestion would be to install a minimal Leopard system—it can be a 10.5 client, it doesn't have to be a server—on an external drive, copy over a copy of QRecall, bo use that to restore the 10.5 server.

If you don't have a spare external drive, install Leopard on the Xserver, install QRecall, perform a "live" restore, and then immediately reboot.

Ralph Strauch wrote:I'm continuing to have problems backing up to a network drive connected to an Apple Airport Extreme. I run scheduled partial backups on my wife's iMac at 1am and on my MacBook Pro at 3am, into the same archive. Both have Qrecall 1.1.4.5 installed. Up till 12/22 both seemed to be working fine. On 12/22 the iMac log shows a "Scheduler Version mismatch" and "Autoupdate system components," so that may have been when I updated the iMac to 1.1.4.5. From the log, it looks like I had updated the MBP on 12/19.

Ralph,

This is indeed the message that you get when QRecall updates itself. The log files you uploaded confirm that this is the date you updated QRecall on that system.

On 12/23, the 1am iMac backup failed due to "disk full -- Capture ran out of disk space." The disk had more than 20gb free and the backup would have been less than 100mb. The 3am MBP backup later that night ran fine. Later that day I merged a bunch of layers to free up additional space in the archive, then turned the iMac and AE off and left for a vacation. When I came back on 12/31 I started things up again.

You've run into another known problem with OS X and the Airport basestation. Basically, some Apple File Servers don't handle file pre-allocation requests correctly. The bug is in older versions of OS X, the Airport basestation, and some NAS storage devices.

Your "disk full" problems can probably be solved by setting the QRFilePreallocateDisable expert setting in the Terminal:

defaults write com.qrecall.client QRFilePreallocateDisable -boolean true

You'll probably want to do this on all of your systems, particularly those encountering the out-of-disk-space problem. This problem was recently mentioned in the Preallocation failed thread.

Most of what you describe after that makes sense. The file allocation bug caused QRecall to abort abnormally, leaving the archive in an invalid state. It may have also caused the volume allocation map to become corrupted—operating systems have notorious difficulty dealing with disk-full conditions. Repairing the volume and then repairing the archive probably cleaned everything up.

After the repair I have two distinct entries in the "owners and volumes" list for my MBP hard drive. One contains only recent layers which I haven't merged yet and the other contains all the layers that have been merged at some time with another layer. I also had one damaged layer at the end, with no date and only damaged content. I then did a manual full backup of the MBP. Both that and the scheduled partial backup look fine. I haven't yet reattached the drive to the AE for a backup of the iMac.

The two volumes in the Owners & Volumes drawer is odd. This usually happens when you resize/reformat a volume. Even though it has the same name, QRecall identifies it as a different disk drive. I'd be interested to know if QRecall continues to add layers to the "new" volume or the old one.

The damaged volume/layers in the repaired archive can be ignored or deleted. In your case they are simply artifacts of the repair process.

Richard,

Sorry for the delay, I didn't notice that you'd posted a follow up.

The mini: Given the current architecture of QRecall, the mini is going to have an awfully hard time capturing and maintaining an archive over 1TB. I don't know how much memory it has, but that's probably the limiting factor. When archives get large, capturing involves a lot of small drive/network reads which incur a lot of latency and virtual memory swapping. As I mentioned earlier, I'm in the process of reengineering the database lookup code so that's it more efficient with large archives, but that's not a reality yet.

The iMac: As for the other problems you're having, I suspect I/O problems possibly related to using a WHS as your archive volume. I really can't hazard a guess as to what those problem might be, but you shouldn't be getting the errors and problems that you're reporting.

In all cases, I'd really appreciate it if you could send me diagnostic reports (Help > Send Report) from each QRecall installation. That would let me look at your hardware configuration and the details of the reported errors.

I'm sorry that you're initial encounter with QRecall wasn't more pleasant. I can't personally test every possible combination of computer and network, so user feedback and diagnostic reports are immensely helpful in improving QRecall for everyone.

Incidentally, my iMac will not blank the screen while QRecall is running, which is annoying. It blanks as soon as the job completes.

See the QRRunningMakesSystemActive setting on the Advanced QRecall Settings page.

Richard,

First, a confession. I'm fully aware that QRecall has performance problems when the size of the archive begins to exceed 1TB. The basic problem is that QRecall must compare every block of data being added to the archive with the data that's already stored in the archive. This various indexes and tables used to do this are very efficient when the archive is modest (500GB or less), but begin to degrade when the archive exceeds 1TB. Addressing this performance issue is one of the main goals of QRecall 1.2.

So the fact that things are slow isn't surprising. By the way, you never mentioned how big your archive is.

However ...

Richard Morris wrote:I continually get "A storage or disk error occurred......the archive is probably damaged" error messages, particularly from the iMac. The Mini seems much less prone to problems and after several attempts I succeeded in backing it up.

Slow is one thing, but you shouldn't be getting corrupted archives. Please send a diagnostic report (Help > Send Report) from both systems so I can take a look at the cause of these errors in more detail. There may be something else going on.

In desperation I have broken the backup job into 4 actions and the second one just failed.

My suggestion would be not to try to subdivide the capture (although there are good reasons to do that too), but to limit the amount of time the capture works by setting a stop condition. QRecall captures are always incremental, so I recommend setting up a single capture that starts late and night and automatically stops in the morning. You can do the same for other long-running actions, like a compact.

The idea is that the capture might not finish, but it will save its work and finalize the archive. This is important because the auto-repair feature works by reverting changes back to the last completed action. By successfully adding data in smaller increments, any possible failure in the future doesn't have as much to recapture. The next capture will always pick up where the last one left off.

Running a rebuild/reindex after a failure takes 24 hours so not something that is practical to do after every second backup attempt.

When you get a "archive is probably damaged" message, are you letting another capture or verify action run first before trying to repair the archive? Depending on what kinds of problems you're encountering, QRecall is able to auto-recover from most kinds of capture failures. It does this automatically at the beginning of the next action.

The verify action/command is particularly useful in this respect. A verify will first auto-repair the archive before the verify begins. If a damaged archive can be auto-repaired, the verify will do it. Just watch the verify progress. Once it starts verifying the contents of the archive, it has successfully auto-repaired the archive (assuming it needed it) and you can stop the verify. If the archive can't be auto-repaired, it will immediately report a problem.

Is any one else successfully backing up to multi TB archives, which after all, are not that uncommon these days?

[Raises hand] I keep a 1.6TB archive here for testing. I use it mostly to stress test QRecall.

It seems very slow once the archive is large.

I freely admit that my 1.6TB archive is as slow as molasses on a cold day.

Do these speeds sound right for other users? The LAN can easily handle the top speed of a USB drive (30+MB/s).

My speeds are better than yours, but you're never going to get stellar performance from this arrangement. (That's not to say that it couldn't be better). A 30MB/s transfer rate is not going to be terribly fast for a large archive for a number of (technical) reasons. Add in network and file server latency and it's going to slow that down even more. So given these circumstances, that sounds about right.

Any suggestions would be welcome.

My suggestions would be (1) create time-limited captures, (2) run verify after a failure (or just let the next action run as scheduled) to see if the archive can be auto-repaired, (3) send in a diagnostic report, (4) schedule a verify to run once a week, and (5) try capturing to separate archives.

The last one is really one of desperation, but will probably avoid most of the problems you've encountered. You could, for example, set up four archives: one for each internal drive and one for each external drive. The real question is how much duplicate data you have between the four drives. If you have 100s of GBs of data duplicated on the two Macs, then I can see how you'd like to have a single archive. However, if most of data on each system is unique, then most of the duplicate data is going to be from one capture to the next, not between systems. In the latter case, multiple archives will be nearly as efficient and much faster

I suppose it is possible there is some interaction with the WHS going on but it is patched to date and has performed flawlessly for the last 18 months.

That's possible, but I need to see more evidence. I'm naturally skeptical of claims about "flawless" performance because most drive and network errors go unnoticed.

For anyone who's interested, QRecall currently imposes a 2TB limit on the size of archives. This limit is imposed by the size of the data structures needed to manage an archive of that size. I plan to increase this limit greatly in the future, but it will require a 64-bit version of QRecall.

The message "waiting for permission to open archive" usually means that some other process has the archive open. It could be another backup program, Spotlight, the QRecall application (an open archive window), or a QRecall action running on another computer.

One way of finding out which process has an archive file open is to use the lsof command, like this:

sudo lsof /Path/to/folder/containing/archive

There are two bugs to look out for. The first is Apple's filer server. It occationaly loses contact with a client and thinks the client still has a file open when it doesn't. The solution is to restart (stop/start) file sharing.

The previous version of QRecall had a bug where closing an archive window didn't completely close the archive file. Any actions that tried to start would wait. I you think this might be the case, just close the QRecall application and open it again.

In very (very) rare circumstances, the archive's lock file doesn't get cleared. If you've checked all other possible sources (using lsof) and the action is still stuck, post again or send a diagnostic report (Help > Send Report)