John,

Congratulations, you've discovered a new problem! That's not what most customers want to hear, but there it is.

During a capture, QRecall uses a feature of the operating system to pre-allocate file space on the volume. This protects it from "painting itself into a corner" by ensuring that there's always enough free space on the volume to close the archive and write all the necessary index files.

For some reason, QRecall's attempt to pre-allocate space for one of your index files is failing. This is from your log file:

2009-08-27 18:39:14.027 -0400 #debug# preallocation failed

2009-08-27 18:39:14.027 -0400 #debug# API: FSAllocateFork

2009-08-27 18:39:14.028 -0400 #debug# OSErr: -48

2009-08-27 18:39:14.028 -0400 Details Length: 36864

2009-08-27 18:39:14.028 -0400 Details Pos: 1081392

2009-08-27 18:39:14.028 -0400 Details File: /Volumes/Xsan1a/Edit/EVS/QRecall/ProjectsA.quanta/package_scribble.index

2009-08-27 18:39:14.028 -0400 #debug# RepositoryPackage received kPreallocationFailedNotification

2009-08-27 18:39:14.442 -0400 #debug# caught exception

2009-08-27 18:39:14.442 -0400 Details disk full

The error that QRecall is getting back from the operating system when it tries to preallocate a mere 36K of additional disk space is -48 (duplicate file error), which frankly doesn't make any sense at all. QRecall assumes that you've run out of disk space simply because the error was returned from the pre-allocate request, which is just about the only reason that call should ever fail.

I suspect that this is a bug in Apple's server OS or XSAN implementation. The FSAllocateFork has had a checkered history of bugs (I've filed at least three), some of which have were fixed in OS X 10.5 (client). Other's are still outstanding.

QRecall already has a workaround for OS X 10.4, where the FSAllocateFork function doesn't work correctly at all. I could send you a special version of QRecall that uses that workaround all the time, or a special version that doesn't attempt to pre-allocate at all. With 5TB of free space, you clearly don't need this mechanism; it's really to protect archives on really small volumes, like USB thumb drives.

If you're interested in testing a special version, let me know an I'll code it up and send it to you.

John,

One more question: Are you running QRecall on the Xsan metadata controller or from one of the clients? If so, what's the version of Mac OS X running on the controller (the diagnostic report will tell me what OS is running on the client)?

John Heagy wrote:Hmm... our Xsan is certainly not limited to any file size and is fibre attached, so it's a local HD as far as any apps are concerned. It does use a clustered file system as any San would.

I tried another smaller archive and it complained of lack of disk space when attempting to archive 300GB to 5TB of free space after writing only 5.8GB. I suppose it doesn't like writing to an XSan.

That's possible. If you're not running into the 4GB file size limit, then something else is amiss. Send me a diagnostic report (Help > Send Report...) and I'll look into it further.

Are you aware of any customer using QRecall to archive files to an Xsan volume? I my test case it's from one Xsan volume to another.

Not that I'm aware of, although I know many people are using various NAS drives without any problems. And I capture to a 3TB RAID system daily.

John Heagy wrote:Any issues backing up data from Xsan volumes?... or as archive destinations?

QRecall is specifically designed to capture files on local, HFS+ formatted volumes. It works, with limitations, capturing other filesystems (MS-DOS, UFS, etc.). It can capture files on remote/networked volumes, but will be limited to the security and access restrictions imposed by the server.

The archive itself is simple and only uses flat POSIX-compatible files. You should be able to create an archive on any filesystem supported by Mac OS X that supports large files (a SAN device, a USB thumbdrive, ...). I will note that QRecall does not play well with WebDAV volumes. Search the forum archives for more info.

Seems to be working fine archiving a small Project folder. FCP, PSD AE etc...

I then pointed it to all our projects, about 700GB, and eventually got a out of disk space error... it had 20TB available.

A common problem with many external and networked volumes is that they may be using a volume format or file server that does not support files greater than 4GB. What you'll see is QRecall capture about 3.9 GB of data and then stop with an error that it ran out of disk space. (I plan to include a test for this specific situation in a future version, but for now this is the behavior.)

You need to determine if the volume format and file sharing protocols you're using supports files larger than 4GB. If not, then looking into reformatting the volume (preferably HFS Extended) or upgrading your file server software/firmware. For example, the original Apple Time Capsule requires a firmware upgrade to work with 4GB files.

I'm using the 12day demo.

A trial key is fully functional.

Any way to include or exclude file types by extension?

Not in the current version. This is on the to-do list for an upcoming release.

Bruce Giles wrote:Are you allowed to talk about compatibility of QRecall with Snow Leopard yet?

With Snow Leopard rolling off the docks already, I can't see how talking about it could do any harm.

Does it work? Any problems?

The current version (1.1.2) has a number of serious problems under Snow Leopard. I've fixed those and have been testing a 1.1.3 version for about two weeks now. I just made some minor tweaks today. If testing this evening goes smoothly, I'll roll out the new version tomorrow.

I'll make sure that's one of the options.

Wayne Fruhwald wrote:There are times when I would like to merge two layers preserving those files (e.g. only remove the older version of files that exist in both layers).

Wayne,

I'm working on a new "Keep Last Version" feature that will preserve the last version of a deleted file (limited to some user-defined amount of time). I think that will give you what you're looking for.

David Kaminsky wrote:I'd like to have a "merge trial" function which would tell me how much space will be saved by doing a particular merge.

David, that's an interesting suggestion. I've also had many requests for statistics about how much duplicate data is contained in each layer, and in the archive as a whole.

I've put off implementing any of these because the calculations required are rather expensive. It requires building a map of all the data references in the archive, then isolating those used by one layer but no others. The "Searching for free space" phase of the capture action that occurs following a merge is (essentially) the same calculation. As you've probably already experienced, that can be a very time consuming process—often requiring 10 minutes or more to complete.

Nevertheless, I've had enough requests for this kind of information that I'm considering implementing some form of free/duplicate space calculator in the browser. With multi-core CPUs and multi-GB RAM becoming the norm, maybe it will actually turn out to be a usable feature.

Charles,

I'm sending you a special version of the QRecallKickStart deamon. This is the system component that causes launchd to start your scheduler process as soon as you boot (rather than waiting for you to log in the first time). Hopefully, it should shed some light on what's going on—or not going on.

Bruce wrote:I think your software would be my favourite backup-solution if it would encrypt the backuped data.

Bruce,

This is an often requested feature and is currently planned for the next major release. In the mean time, several users are backing up to archives they created on encrypted disk images.

Chris Caouette wrote:Maybe having a skip free space search can become preference in a future update?

It's already on the to-do list.

Chris Caouette wrote:Why is my scheduled backup spending about 20 minutes searching for unused space? Should I reindex?

Chris, reindexing won't help. When QRecall captures new items, it tries to identify and reuse any wasted space in the archive. This obsolete space is created when you merge layers. Merging may result in records of files, directories, or individual blocks of data that the archive no longer needs to keep.

Following a merge, the next capture or compact action begins by cross-referencing every record used in the archive in order to determine isn't being used. If the archive is particularly large, complex, or access to the archive is slow (say, via a network), then this can take some time.

If the amount of time spent locating free space is becoming excessive, here are two solutions:

Merge less often. A merge is what causes records to be freed and triggers the search for free space during the next capture or compact action. If you haven't just merged, neither of those actions have anything to find. Set the schedule of your merge actions to run, say, once a week.

Turn off the the free space search for capture actions. Turn off the free space sweep by issuing the following command in the terminal (see Advanced QRecall Settings). Make sure you schedule a compact to occur at least once a week. The capture will skip looking for empty space, even after a merge, but the compact still perform its normal maintenance.

defaults write com.qrecall.client QRCaptureFreeSpaceSweep -boolean false

The second solution is particularly well suited for situations where you capture from a remote computer (say a laptop) to an archive on a desktop via a network. The laptop can capture quickly, and the desktop can schedule local compacts to keep the archive tidy and efficient.

Charles Watts-Jones wrote:FYI there were no com.qrecall files in the /Library/LaunchDaemons, /Library/LaunchAgents, and ~/Library/LaunchAgents folders. While the /Library/Application Support/QRecall and the ~/Library/Application Support/QRecall folders existed, they were empty.

That's the way it should be. When you use Shift+Option+Command+Q to Quit and Uninstall QRecall, it should remove all of those components. I just wasn't taking any chances.

Well, it appears from your log that everything was reinstalled correctly but your scheduler still isn't getting started when you start up. Why I can't say. Launch your Console application and look in your console and system logs for anything from the QRecallKicker (just search for "qrecall"). If you find any messages, send them to me. If you don't, I'll send you a special version of the QRecallKicker daemon that logs extra detail to the console log.

The QRecallKicker daemon is a system daemon that (should) poke the launchd service into starting your scheduler when your computer boots. My guess is that either it's not running, or launchd isn't starting your scheduler when it gets poked.

Two quick questions: Do you have your system configured to auto-log in, or boot to the login screen? And is your /Users folder on a different partition than your main /System folder?

Duly noted. I've already added it to the wish list.

All good suggestions!