I'm curious to learn if your QRecall installation was successfully pre-authorized. Open up the QRecall > Preferences... and in the Authorization tab, make sure it says "QRecall has been preauthorized to use administrative privileges when capturing and restoring items."

I'd also like to see your log file. If you are using the current beta, choose Help > Send Report... command. If you are using the release version, zip up the file ~/Library/Logs/QRecall/QRecall.log and either post it to the forum or send it directly to me.

Thanks

Bruce Giles wrote:I can send you the log if you're interested.

Yes, thank you. Try the new Help > Send Report... command.

If you've reindexed your archive there shouldn't be any missing names in the index. There's a recently discovered bug that could result in empty (null) names appearing in the index, but I'll have to go back and find out if that bug was in the latest public beta.

The effect of missing names is harmless, but thanks for the heads up. If you have a chance, please send a diagnostics report (Help > Send Report...) or e-mail the relevant log files.

Can I use QRecall to make a non-encrypted backup of files from my home folder when that folder is open (i.e., accessible)?

In principle, the answer is yes. I'd suggest configuring the QRecall scheduler to NOT start or run actions while logged out. If you are just capturing your own documents, you probably don't even need to authenticate it to run with administrative privileges.

However, I strongly suggest that your download QRecall and try it with a trail identity key first. I have not tested QRecall recently with File Vault and Apple has made a large number of security changes that impact what applications are allowed to do under File Vault. So try it out first and make sure it works on your system.

I have a long standing to-do item entitled "Make QRecall File Vault Friendly" which has been around for some time. I've been putting this off until I add archive encryption to QRecall, which would make it much more appropriate for using in conjunction with File Vault users.

Frankly, I wish I hadn't encrypted my home folder with FileVault. I don't have enough space to un-encrypt it

If you have the external disk space, you could probably copy or capture enough to your external drive to un-encrypt your home folder and then restore the documents.

Ralph,

Thanks running the reindex again and the for the diagnostic report.

The problem is a duplicate directory record. This is a subtle problem that was only recently brought to my attention by another user. The most recent versions of QRecall (version 1.1.0(7) and later) have new diagnostic code to detect and correct this situation.

My suggestion is to repair the archive. This should detect and repair the duplicate directory records. If you end up with a layer that's marked "Damaged", merge it with a subsequent valid layer to eradicate any trace of the anomaly from the archive.

Ralph,

If you have the time, please reindex the archive with QRecall 1.1.0(lucky 13).

I expect that you'll encounter a similar problem, but this version dumps more information to the log. I'm basically trying to figure out if the problem is in your archive (any why) or if it's a subtle flaw in the reindex procedure.

To upgrade:
- Download 1.1.0(13)
- Open your Applications folder
- Drag your existing QRecall application to the trash
- Open the 1.1.0(13) disk image
- Copy the new QRecall into your Applications folder
- Launch the new version

Reindex the archive, then choose Help > Send Report... to upload your log files.

Thanks,

Ralph Strauch wrote:After backing up with v1.1.0.12 the archive needed reindexing so I did that.

The message you got was a little misleading. The long version of that message would read "the names index has gotten out of synchronization with the archive because it was updated by an older version of the application that doesn't know anything about the names index. The names index has been erased. If you want to take advantage of the performance improvements provided by the names index, the archive will need to be reindexed."

When the process finished i got an error message telling me it had failed. Looking at the log it appears that the failure occurred early in the process, but it didn't stop the process or notify me till the end.

The problem that stopped the reindex occurred at the very last instant. The warning message at the beginning was because you started a reindex and then stopped it, leaving the index files completely scrambled.

My compressed log file is attached.

Next time try the new Help > Send Report... command.

I guess the thing to do now is to repair the archive, but I'll wait on that till I hear for sure from you.

Hold on for a bit, if you can. The error that you got is extremely puzzling and I'd like to do a little investigation first.

Ralph Strauch wrote:I downloaded the beta tonight and opened it from the dmg to take a look at it and see what was different. I thought I'd wait for the release version, though, to make the switch. I ejected the dmg and tried to back up using v1.0.1 and it told me I had the wrong scheduler and needed to quit and restart to reinstall the scheduler.

The problem is that the new version installed its own scheduler as soon as you launched the application. The older version won't uninstall the newer components because that could disrupt the installation for other users.

So then I decided just to install the beta and run from that. Now I'm getting the message that qrecall can't contact the scheduler, and restarting qrecall doesn't help.

I'm not sure what's going on now, but the scheduler is probably only half installed.

Try uninstalling QRecall (hold down the Shift and Option keys and choose QRecall > Quit and Uninstall). Then relaunch QRecall and reauthorize it (QRecall > Preferences... > Authorization). If that doesn't solve the problem, toggle the "Start and run actions while logged out" option twice. This will uninstall and reinstall the scheduler. (A restart afterwards won't hurt either.)

If you're still having problems using 1.1b12, send a diagnostic report and I'll look into it (Help > Send Report...).

What do I have to do to reinstall the scheduler so I can use the app again?

As a general rule, if you want to bounce around between versions of QRecall first uninstall the currently active version (QRecall > Quit and Uninstall) then launch the new one.

Got it. Thanks.

Ralph Strauch wrote:I currently use Apple's Backup program to maintain an offsite backup of important files on dot Mac (soon to be mobileme), and I'm wondering if qrecall might do a better job.

Alas, .Mac/MobileMe and QRecall are not a good mix. There's a long-winded discussion about why in the Online storage thread. But the short version is this: MobileMe is based on WebDAV which does not play well with QRecall.

Would the processing reduction gained by using smaller archives make it worthwhile to have separate archives for different groups of files, particularly for groups of files that I might want to update less frequently?

Probably not. There's not much overhead associated with the data in an archive that isn't being updated. The overhead of managing multiple archives probably outweighs the overhead of a lot of data that doesn't get updated that often.

Capture compression should reduce the amount of backup data to be transmitted, but would it also impose additional interchanges between qrecall and the archive that would negate these savings?

Compress doesn't involve any additional communications. Overall, less data is transferred to a compressed archive than an uncompressed one. The penalty for compressing an archive is computational: Every time data is accessed from the archive it has be be decompressed and all new data has to be compressed. That just takes a lot of additional CPU cycles.

Would merging and other management processes be inordinately time-consuming over the internet?

Yes. The best strategy would be to avoid them, or run them very infrequently (like once a month), if your only connection to the archive was over a slow communications link.

Do you have any other thought pro or con about using qrecall over the internet?

If the file server you are storing the archive on is using a "real" remote file system protocol (like an AppleShare server) then you'll probably be OK capturing a modest amount of data and scheduling infrequent merge and capture actions.

However, if you are using any WebDAV based service (MobileMe, Amazon SSS, AOL, ...) then it's impractical to use QRecall on the remote archive directly. Instead, consider getting a small USB thumb drive or small external drive and performing your captures to that. Once a week, or once a month, upload the archive to your off-site data store.

Marc Farnum Rendino wrote:I hadn't heard this before; where can I find more info?

That's a good question. To some degree, I'm guilty of repeating the conventional wisdom of the backup industry. This has been considered a "fact" for more than a decade, but finding a reliable source to corroborate it is difficult.

There's really very little information on the effects of long-term storage of HDDs. Even Google's massive study (Failure Trends in a Large Disk Drive Population) essentially ignores the issue of shelf-life because no one uses hard drives that way.

It's still a widely held belief that hard drives have a shelf-life of about 5 years. I'm very confident that hard drives have much shorter shelf-life than optical or tape media, but knowing exactly how much is problematic. A big confounding issue is that hard drive technologies continue to evolve. Drives made 5 years ago might have a shelf-life of 5 years, but drives made today might have a shelf-life of 20 — or 2.

Alexandra Morgan wrote:Alternatively, could you have QRecall run on the machine to which the backup drive is attached, instead of the machine to be backed up? In other words, the archive is local, the item to be captured is a network volume.

Yes, but with limits. The operating system doesn't have unfettered access to files on a remote volume. Everything accessed via a networked volume is constrained by the permission and security limits of the file server. Thus, it would be impossible to capture a user's operating system remotely — at least not in a form that would let you restore the volume and have it boot. If you just want to capture regular user documents, it shouldn't be a problem.

Can QRecall automatically mount a network drive to back it up?

QRecall won't automatically mount the volumes that contain items to be captured. However, you could probably hack something up without too much trouble: Create any kind of script that would cause the volume of the remote user to mount (just opening an alias to a folder on that volume would do the trick). Then schedule the capture to run after that, or even schedule the capture to run when that volume mounts (see event schedules).

Context: I'm dealing with a small workgroup, with about 10 to 13 desktops, and I have 3 large hard drives to back up onto, all between 500 and 750 GB each (because you get the most storage per $ in this size range).

I deal with this by having each client capture to its own archive. Not as much space savings, but all clients can capture simultaneously. The clients only have a single capture action. All merge, compact, and verify actions are set up to occur on the computer hosting the archive volumes.

... it appears that a network hiccup on a client machine resulted in a very corrupted archive.

Try the beta version. It has a new automatic repair and recovery feature that should instantly, and transparently, recover from 99% of problems that would result in a corrupted archive. This feature was added principally to deal with intermittent network communication failures.

Irk, I so want to get away from Retrospect already!!!

I'm right there with you.

W. Darson wrote:ow. so my questions are, what is "closing" an archive

Closing an archive encompasses a number of tasks. Mostly it involves updating the archive's various indexes.

and how long should i expect it to take?

That's a hard one to answer. The short answer is "it takes at long as it takes."

In your instance, I suspect that QRecall is preoccupied updating the quanta index. This is the largest index and, since you just captured 47GB of new data, it has a lot of new data to index. Unlike the actual capture phase, updating the quanta index involves millions (in your case hundreds of millions) of tiny reads and writes.

When capturing to a local hard disk, I'd expect a capture of 47GB to take another 10 minutes or so to index all the new data. I don't know anything about your network or your NAS drive, but how fast it can process these small read and write requests will substantially impact the speed at which it will take to finish the index. For example, if your network+NAS combination were working at 1/20 the speed of a local hard drive it would take 3-4 hours to finish indexing. If it works at 1/100 the speed (which is not unheard of) it could take 15-20 hours.

Start up the Activity Monitor and see if QRecall is working. You should see substantial network activity (lots and lots of small packets). If you do, I'd suggest letting QRecall continue working. If not, let me know and we can look into the situation in more detail.

The good news is that this doesn't have to be done again. On the next capture only the newly added data will added to the quanta index, and will be much faster.

Rodd Zurcher wrote:Only question I have (and I don't think it's beta related) is how long it takes a capture to "locate free space".

I'm backing up two machines to the same archive. At first I thought the locating was fast when the same machine captured to the archive without the other in between (Capture A, Capture A; not Capture A, Capture B, Capture A).

But I think an intervening Merge/Compact has more impact; but I'm not sure.

It has everything to do with the Merge.

A merge removes two or more layers and replaces it with a composite of those layers. When its done, it may have removed items that may, or may not, share common data with other files still in the archive.

The "locating free space" phase will occur on the next capture or compact following a merge. What QRecall is doing is something called "mark and sweep" in computer jargon. It first "marks" all of the quanta that are used by at least one file and then "sweeps" all unmarked quanta into the trash bin.

I'm backing up to an archive over AFS. Using both ethernet and WiFi. Of course the wifi is slower; but works.

How long this takes depends primarily on the total number of items records in the archive and fast it can read them. Reading millions of file records over a WiFi connection will take awhile.

You can mitigate this by merging less often and only on the machine with the fastest connection.

Schedule a merge followed by a compact on the one computer with the fastest access to the archive and remove the merge and compact actions from all other computers using that archive. If the AFS server is a computer (not a Time Capsule or other NAS), install QRecall and schedule the merge and compact on that computer. You do not need an identity key to schedule and run maintenance actions.

If you have the disk space to spare, consider merging only once a week or add a condition to the merge action so that it only runs when the free disk space falls below some reasonable margin.

Is there anyway you could index the free space instead of "locating" it?

That's exactly what it is doing. It's "locating" the unused data so it can add it to the index of free space.

Also could you add to the log the amount of time spent creating the scribbles, and locating the free space (to the normal logs; not #debug).

That's a good idea. I'll add some message indicating how long it took to locate the free space. I might even be able to log the amount of free space it found. The timing of scribble files, however, is somewhat meaningless as this happens in parallel with the capture. It's only an issue if the capture finished before the initial copy does.

Thomas Traeufer wrote:The program, despite not working, also created thousands of small, invisible .PBsyncDB files. I didn't like that and now my question would me whether QRecall does something similar.

No.

QRecall doesn't make any changes to the documents or folders it is capturing and it doesn't add any files as it goes. It will happily capture from a read-only device.

Everything captured by QRecall is stored in an archive. An archive package consists of a single data file containing all of the archive information, accompanied by a small number of index files.

The only other files it creates are a handful of support files (mostly in ~/Library/Preferences/QRecall and ~/Library/Application Support/QRecall), some log files, and some support components that get installed into the operating system.