Christian Roth wrote:While the initial capture took its time (about 9 hours), the second capture now takes over 15 hours and I am about half way through it.

15 hours seems like a long time, but it's not unheard of. It depends mostly on how much new data is being added to the archive and the speed of the volume containing the archive.

Any capture that adds a lot of new data (i.e. tens of GBs) can spend a considerable amount of time organizing and reorganizing the archive database. This is usually encountered during the "closing archive" phase of the capture, but it can happen in the middle of the capture too. The result is that QRecall appears to be spinning its wheels (sometimes for hours) while it updates and sorts its index of quanta.

Other confounding factor: The speed and latency of the archive volume (particularly networked volumes and USB drives). How much RAM the system has (less than 1GB will cause QRecall to run much less efficiently). Running other applications at the same time (causes virtual memory swapping and reduces the amount of RAM available for buffering data).

I checked in Intstruments and find a lot of very small (in size) accesses using PBWriteForkSync and PBReadForkSync. Should this be happening?

Probably. My guess is that QRecall is capturing a lot of small (&lt32K) files to a relatively large (&gt100GB) archive and/or your system doesn't have a lot of RAM. (A semi-informed guess based on your I/O trace.)

I guess that the constant head movement slows down the capture significantly, but maybe it's ok.

Excessive head movement is generally bad, and really slows things down, but in a few cases is unavoidable. The real killer is when QRecall decides that it needs to resize its lookup tables. This can thrash the archive volume for hours while it copies really tiny bits of data from one index to another. Indexes are resized exponentially, so this will only occur a few times during the life of the archive; but it is especially likely to occur during the first few large captures.

As you see, there are many 12-byte-long reads and 4, 5 or 6 byte writes. Is this supposed to be handled by the OS' caches or would these accesses (since synced??) actually result in hard drive access?

When using Instruments or iosnoop, you're tracing the requests that come from the application, not the physical I/O to/from the drive. All of these small reads and writes are (typically) buffered by a RAM cache that tries to minimize the actual I/O to the volume. Of course, there are limits and the more RAM you have the more efficient this buffering will be.

Just wondering if that behaviour is expected and normal.

Unless QRecall gets stuck or stops with an error, I'm inclined to assume that it's doing the best it can with what its got. Let the capture finish -- send a diagnostics report afterwards if you like -- and then see what the performance of subsequent captures is like.

gery houiller wrote:I do synchronize 2 Xserve servers with 1.5 TB of remote data by an ADSL line.

That's a lot of data to try to capture via a DSL connection. Right now, QRecall only writes to mountable volumes. So you would have to mount the volume with the archive using some form of remote file services, such as AppleShare IP.

Is that your software can handle as much data and how
long does it take to verify the state files a knowing synchronize
it has 30 gigabytes of data per month amended.

It can handle that much data, but there are too many variables to say how long it will take. My only suggestion would be to get a trial identity key and run a test.

Is it necessary to check the status of each file every day or do you use the tool of verification of time machine in the mac os 10.5.4 to avoid to scan all files.

QRecall uses the same file system change event technique that Time Machines uses under OS X 10.5. However, this can't be trusted indefinitely. There are a number of subtle situations where OS X 10.5's change information can be inaccurate. So approximately once a week, QRecall will ignore the change information and perform a deep scan of every item on the volume. This period can be adjusted.

David Cretney wrote:Using the drop down arrows to navigate the library is cumbersome, I'd sure like to be able to navigate by double click and only use the task bar button for restore- or make it configurable.

There are two alternate browsing modes that I've had on my to-do list for eons, but I've never gotten much feedback about the browser so I've left it be for now. That, and any change to the browser would be a huge amount of work.

I've considered a single-window browser. Opening a folder would replace the contents of the window with the contents of the opened folder. This is, more or less, how single-window browsing in the Finder works. But then there are navigation issues (i.e. how do you get back).

Another option, and the one I'm most fond of, is to implement a column browser mode.

Anyone have any input or reaction to these suggestions?

ubrgeek,

I just looked at the log file you sent and it's a bit of a mystery. The capture is fails to write to the archive with OS error -51. This is an "invalid reference number" error.

When an application opens a file, it gets a "reference number" which it then uses to read and write to that file. When the file is closed, that reference number becomes invalid.

I can't think of any good reason why this might be happening. Some file/operating systems have limits about how many files can be open at once. I'm wondering if the SMB drivers on the PowerMac is losing track of what files it has open? Or maybe the volume is getting disconnected and reconnected (this happens all the time on my WiFi), and after reconnecting it has forgotten what files it had open? (I'm mostly thinking out loud here.) I'll post this question on one of the technical forums and see if anyone has any good ideas.

ubrgeek wrote:Can I send along the log file and see if you have any ideas?

Always. Just choose Help > Send Report... and I'll take a look at it.

David Cretney wrote:Did switching from trial to paid lock me out of captures that were created while I was using a trial key?

No, they're still there. They just belong to a different user.

An identity key is just that: It identifies the owner of the captured items. When you changed keys, you changed identities. Items captured before September 1 belong to the trial key owner. Items captured after belong to your permanent key.

Open up the Owners and Volumes drawer (View > Show Owners & Volumes). There you will see the set of owners and volumes captured in the archive. Select a volume to switch between them. The layers that contain items belonging to that owner & volume are active. The layers that do not contain any items belonging to the selected owner and volume are grayed out.

Welcome, David.

David Cretney wrote:I really hope the development continues. It appears the 1.0.1 has been around for awhile now.

If 5 months is "awhile now," then I guess 1.0.1 is getting long in the tooth. sigh Internet time moves so quickly.

Despite the developers intentionality in creating a simpler user experience, I guess it has partly been the change of vocabulary, and partly just the change in doing things

I've struggle mightily with the terminology of QRecall, and that's one of its consistent criticisms. The problem is that QRecall does do things differently. So the choice is to reuse teminology that's inappropriate, or introduce new terminology that's unfamiliar. I've decided that I can't win for losing.

but I haven't always found that QR does what I would have hoped or expected.

That's exactly what these forums are here for. If something's confusing, inconsistent, or unexpected, please share those experiences. It will either improve QRecall or your understanding of QRecall.

Steven M. Alper wrote:When I restarted the icons for my external Firewire hard drive's two partitions had been changed to generic icons.

That's suspicious.

However this morning I was notified that QRecall was unable to find its archives on that disk.

QRecall remembers the location of both archives and items to be captured using aliases. These are the same alias records created by the Finder when you make an Alias file.

I suspect that AppleJack has changed something that OS X was using to identify known volumes. This would also explain why your icons changed (since OS X now thinks these are new, never before seen, volumes). You might check to see if you have any other aliases to items on your external volume. They might be suffering from the same problem.

You can probably rectify this by editing your actions. Open the action, reselect the same archive (creating a new alias to the archive) and save the updated action.

/private/var/db/volinfo.database

Yikes! /var/db/volinfo.database is not a transient file. I would definately alert the AppleJack authors about this one.

One of the important bits of information stored here is which volumes use POSIX Ownership and Privileges and which ones don't. Erasing this file could cause volumes that previously enforced file ownership and privileges to suddenly stop using them and visa versa.

I don't see anything QRecall related in there, but what do I know.

I don't either. I think AppleJack just gave OS X a case of amnesia and it no longer remembers volumes that it's seen in the past.

Chris Caouette wrote:I am always nervous about a backup to an unknown file format.

I sympathize with your concern.

QRecall archives use a proprietary file format. There is no commercial filesystem available that provides the kind of storage organization that QRecall needs. So it simply isn't possible to provide the kind of features that QRecall does with an existing filesystem. (In an interesting footnote, Apple couldn't implement Time Machine with the standard HFS+ filesystem either. Apple had to alter the HFS+ volume format to get Time Machine to work.)

Is there a way to recover if the backup file gets an error?

Absolutely. QRecall includes a comprehensive Repair command that will systematically scan the raw data blocks of an archive, clear out any damaged data blocks, and reassemble all valid data back into a functional archive.

QRecall archives, unlike standard filesystems, were specifically designed with data recovery in mind. Filesystems are designed to be fast. QRecall archives are designed to be robust and compact. The archive contains several levels of redundant information so that the loss of archive data limits the loss of information to the fewest number of captured items as possible.

In fact, I would go so far as to encourage anyone who's interested to put this to the test. Create a QRecall archive, then use a Hex editor or some other method to poke holes in the archive data, run a Repair, and see what happens.

Archives also contain internal consistency checks that standard filesystems don't have. Every block of data in an archive includes a checksum, which is verified every time that data is read. There have been more then one QRecall user who discovered a drive that was occasionally returning invalid data because QRecall, and QRecall alone, complained when reading from that volume.

What happens if Dawn to Dusk stops production on Qrecall?

I will be very sad. The QRecall application, however, will continue to run. If the future of the application puts your data at risk, you can always recall it to a traditional filesystem and re-archive it using other means.

I had this happen already with a nice backup program that was going great then suddenly the company shut down.

I feel your pain.

I am not having a ton of luck with Time Machine (I have 3 computers and the drive is a 1TB on a AEBS). Qrecall looks like the answer but I am just making sure I have my rear covered.

You're welcome to try QRecall free of charge to see if it meets your needs. If you have questions, problems, concerns, or suggestions, the forums are always open.

A QRecall archive will not span volumes. The very nature of how QRecall works requires that entire contents of the archive be available while capturing new data, so typical volume spanning won't work.

You might find that you can split up your source material and capture one portion to an archive on one drive and another portion to a second archive on a second drive. QRecall's duplicate data detection and compression[1] might allow you to store half of your data plus quite a bit of history on a single volume. You can obtain a free trial identity key to see just how efficiently QRecall can pack data on your drive.

Backing up each data set to its respective volume can be completely automated.

[1] Photo collections are already pretty well compressed, so compression won't help much for this type of file.

Bruce Giles wrote:Thanks for the explanation -- that mostly makes sense. But one thing still confuses me -- the part about the scheduler assessing the state of mounted volumes. When I started up this morning, the external drive WASN'T mounted. It wasn't even plugged in yet.

This is where it gets complicated for the scheduler.

Information about disk and volumes is provided to applications via a series of events that the application can subscribe to. Once an application subscribes, it receives a disk mount event when you plug in a volume and disk unmount event when you eject it.

The action event schedules are driven by these events. When OS X sends a disk mount event, QRecall sees if any actions need to start. When the scheduler receives a disk unmount event, it sees if there are any actions that need to be stopped.

But there are no events associated with volumes that are already mounted (or not mounted, which is the more perverse case) when the application first starts. So some code would have to written to examine the state of the mounted volumes when the scheduler starts and compare that against assumptions about the state of mounted volumes carried over from when the scheduler last shut down.

If I added a "ignore if no archive" condition to each of the actions, would that help?

That will fix the problem. Conditions are evaluated when the action is started, so as soon as the action is started it will check its conditions and skip the action (assuming the volume isn't online).

A better term would be "lingering" actions.

You have five actions defined, all scheduled to run "When archive volume connects," each with a staggered start time (1 minute after archive connects, 2 minutes after archive connects, ...).

Here's what happened.

On August 22 at 16:03:19 you connected the volume named Backup Disk. This triggered the schedule of all five actions:

A Capture was scheduled to run immediately.
A Merge was scheduled to run at 16:04.
A Compact was scheduled to run at 16:05.
A Compact was scheduled to run at 16:06.
A Merge was scheduled to run at 16:07.

All of the actions also have repeat schedules. Once the capture started at 16:03 was finished, it was rescheduled to run again in 24 hours (Aug-23 16:03). Ditto for the remaining actions.

At 16:07 you logged out and shutdown. During the shutdown, the scheduler made a note of all the actions that were currently scheduled to run.

On August 29th you rebooted. The scheduler started up and immediately discovered that it had five actions that were supposed to have been run on August 23rd with various start times between 16:03 and 16:07. The scheduler then started them all immediately.

The perfect storm of events that led up to this were:

- A bunch of actions that were automatically scheduled to run at some distant time in the future.
- A scheduler that was shutdown before the "Backup Disk" volume was unmounted (the scheduler never saw the volume go off-line which would have canceled the repeat schedule)
- The scheduler starts up again long after the previously scheduled actions were supposed to run so that the actions are started immediately.

I'm not saying that it should work this way, but that's what happened. What probably should happen is that the scheduler should assess the state of the mounted volumes first before starting any previously scheduled actions. But there are coding challenges that could make that difficult.

The preferred method of removing QRecall is to hold down the Option+Shift keys and choose the QRecall > Quit and Uninstall command.

Alternatively, you can follow the manual uninstall procedure documented in the Help. This will eradicate every trace of QRecall from your system.

I don't know how AppZapper works, but I'm sure it has no way of knowing how to correctly uninstall QRecall.

The QRecall monitor runs as a user agent (in Leopard). All that's need to remove is to trash the com.qrecall.monitor.plist file in ~/Library/LaunchAgents. It's presence will most certainly have no impact on Time Machine.

The Apple engineers think that the problem is in the file access mode translation.

Some archive files are opened by several threads simultaniously. When a file is open by multiple processes, the access modes all have to be in harmony. When the access modes are translated into HFS or AFP access modes, QRecall is happy. But when the access modes are translated into SMB access modes, they apparently collide.

QRecall version 1.1.0(27) opens its index files using more restrictive access modes. Hopefully, these will translate into equally compatible SMB access modes.

If you want to try this again, download 1.1.0(27) and give it a try.

Extended attributes are "named metadata associated with a filesystem object" such as a file, directory, symlink, etc. Each extended attribute consists of a unique name and a small bit of data that can be attached to any file or folder. This bit of data will (assuming something doesn't strip it off) stay with that item wherever it goes.

To give you an example, when you download a file from the Internet in OS X 10.5 (Leopard), a tiny chunk of data is attached to the file with the name "com.apple.quarantine". com.apple.quarantine is an extended attribute attached to the downloaded file. Whenever you open a file or application that has a com.apple.quarantine extended attribute, the Finder will warn you that you're about to open something downloaded from the Internet. If you say that's OK, the com.apple.quarantine extended attribute is removed.

Tiger had the ability to use extended attributes, but made little or no use of them. Leopard has begun to make use of extended attributes and their use is growing daily, both by Apple and third party applications.

Other examples: The extended attribute "com.apple.metadata:kMDItemWhereFroms" records the URL where an item was downloaded from. The "com.apple.TextEncoding" extended attributes identifies the encoding of a text file so the application doesn't have to guess. "com.apple.diskimages.recentcksum" records checksum calculations which help disk images mount faster.

Some extended attributes, like "com.apple.FinderInfo" and "com.apple.ResourceFork" mirror legacy item metadata (which QRecall already captures).

You can see the extended attributes of an item using the -@ switch of the ls command (in conjunction with the -l switch). So, to see the extended attributes of the items in your Documents folder, open up a Terminal window and type

ls -l@ ~/Documents

Note: I've uploaded a newer version of the QRTouchXAttrItems tool that does a better job of ignoring extended attributes that QRecall already captures.