Chris Caouette wrote:Not to sidetrack the discussion too much but i'm starting anew with my archives now that Lion is out...and I'm not worried about older versions of files as my current setup is all I need.

What would you like a QRecall menu extra would do?

Steve,

You are having a "bad data" day.

Recap

From your description and the logs, I see that you had a capture fail on July 12 due to the discovery of a corrupted data record. You repaired the archive, which encountered and fixed a corrupted data record, and then went on to preform several more captures without incident.

You then copied the archive to an new volume and performed a couple of more successful captures before encountering the same date record corruption problem on July 15. I see in the logs that you then repaired the (new, copied) archive, but there's not much after that before the report ends.

Bad News

First, the (possibly) bad news. The position of the corrupted data record that caused the capture on July 12 to fail is not the same data record reported by the repair that followed. The record that caused the capture to fail was suddenly OK, while some other record was now found to be bad. (Data just doesn't magically fix itself.) This means that the data corruption does not appear to be on the physical disk. Instead, it's occurring somewhere else, most likely on the trip from the platter to your RAM.

After transferring the archive to the new disk, a capture encountered a new data record that was corrupted. The repair that followed reported no less than 14 damaged records.

Good News

Now that sounds bad, but there's potentially good news here. The bad record that caused the latest capture to fail was also one of the 14 records reported by the repair process. This means the damaged data existed on the physical drive.

So it might be that during the copy from the old drive to the new drive, even more data was mis-read from the old drive and that corrupted data was written to the new drive. If that's that case, and the problem that was causing the old data to be corrupted isn't also afflicting the new drive, then the new archive should be OK now that it's been repaired.

If you encounter a data corruption problem during an action and want to distinguish between data corruption on the physical drive and data corruption caused during transfer, try verifying the archive a few times. If you have a data loss on the physical drive, the verify will stop at the same record position every time. If, on the other hand, the verify is successful, or stops with errors at different records positions, then you are experiencing transient data transfer problems, most likely caused by a flaky disk controller, data bus, motherboard, or possibly even bad RAM.

Anything I can do to help debug this issue?

Steve,

Please send a diagnostic report (Help > Send Report). The report will include all crash logs in addition to other information about what QRecall was doing at the time.

Steve Mayer wrote:I've got an archive whose first layer shows up as "Unknown, Unknown, - Damaged-". I've tried repairing the archive, but this layer stays there

A layer that says it is "-Damaged-" means that it encompasses one or more items that were lost during a repair. This doesn't mean the layer itself is damaged, it means the layer has lost information due to damaged data in the past. Thus, the "damaged" moniker will never go away, no matter how many times you repair the archive.

and I'm not able to delete it or merge it.

You should be able to merge a "-Damaged-" layer just like any other layer. If the items that were damaged have been completely recaptured in subsequent layers, then the "-Damaged-" label will go away (because the newly merged layer is no longer missing any files). If not, the newly merged layer will still be marked as "-Damaged-" because it still contains items that were lost at some point.

Is there a way to remove this layer?

Merge it with a subsequent layer.

It is causing my rolling merge to fail (I believe) with an unknown envelope error.

No, an envelope error is causing your merge to fail. An envelope (checksum) error means that a data record in the archive has become corrupted and means the archive needs to be repaired. I recommend using Disk Utility to repair the volume containing the archive, and then repair the archive itself (using the default repair options).

Secondly, attempting to use the "Copy recovered content to a new archive" option in the Repair menu creates a new empty archive but then proceeds to do nothing that I can tell.

Very strange. That sounds like a bug, which I'll look into.

Is my best option just to delete the damaged archive and start over?

Not necessarily. A "damaged" layer just means that data was lost at some point. It doesn't mean your archive is damaged (well your's sounds like it is, but for a different reason). What is does mean is that you should be mindful when recalling items from this layer because it's missing files.

Gary K. Griffey wrote:I guarantee you...I do not have any partition on my mac named "Griffin"...

No, but I do.

It's the result of a placeholder path that I set up during initial development of the browser. In the case of the reindex, the root path doesn't get set until the index is finished and the archive opens again. Until then, it displays its default path. I never noticed it, because the default path got picked up from my development system, so it's exactly what I would have expected to see.

I just deleted the default path, so now the folder control won't display anything until the reindex finishes and the archive opens.

Ralph,

Please send a full diagnostic report. There's usually some additional log messages, near the ones you copied, that will tell which API QRecall was using and what the error code was.

Thanks!

Bruce,

Yes, there's a new advanced setting that will let you "trick" QRecall into thinking your system has a different amount of physical RAM than it's actually equipped with. Set it with the defaults command in the terminal. For example, to have QRecall size its memory usage to 2GB of RAM, use this command:

defaults write com.qrecall.client QRPhysicalMemoryMB -integer 2048

Note that QRecall will ignore any setting smaller than 512MB or larger than 6GB.

Bruce Giles wrote:With b38, I'm seeing the return of a bug I thought you had squashed.

Not squashed, just semi-random. I have not addressed this bug because I plan to replace it with a different mechanism soon.

I didn't try to do a capture before correcting the preference, so I don't know if it was purely a cosmetic issue, ...

It's purely cosmetic. If you had quit QRecall and relaunched it, I'll wager a dozen Krispy Kreme doughnuts that the window would say that it was now pre-authorized.

When you clicked the "Pre-Authorize" button, it launches a process to permanently authorize the helper tool. This normally involves prompting you for your authorization. It didn't because it was already pre-authorized, so it essentially did nothing and then updated the display.

Gary,

Please send a diagnostic report. If the monitor process is stopping for some reason, that might show up in the log.

Everyone,

I has hoped to have the new beta up a few days ago, but some last minute bug and UI changes ran it down to the wire. The new version (1.2.0b37) is up. Choose QRecall > Check for Updates... if it doesn't prompt you automatically.

Sorry for cutting it so close to the expiration date.

P.S. For those using permanent identity keys, you can choose to ignore the "Beta expired" message. Only a beta identity key expires with the application. Permanent keys continue to work with whatever version you are running.

Neil,

Here's the short lesson:

Merging does, indeed, throw stuff away: a merge keeps the new stuff and throws any old stuff away. But that's a good thing over time, otherwise your archive grows forever (or gets unmanageably complex).

Ignore the plain merge action, that's for geeks.

A rolling merge action works on the idea that you want to keep recent fine-grained changes (the file you changed Tuesday morning vs. the version you saved Monday afternoon), but don't care about fine-grained deltas that are months old (two report documents saved a day apart, some six months ago).

The rolling merge actions lets you choose blocks of time relative to today. These are, in order, a number of days to keep only the last version of each item that day, then a number of weeks to keep only the last version of each item that week, and so on with a number of fortnights, months, and finally years.

Here's an example. If you choose to keep 7 day layers, 8 week layers, and 12 month layers, here's what happens when the rolling merge runs:
- All layers captured today are left alone. (This is the minimum "ignore" period described later.)
- The layers for the previous 7 days are organized into 7 groups. All of the layers within a single day are merged into one layer, keeping only the last version of each item captured that day.
- The layers for the previous 8 weeks are then organized into 8 groups. All of the layers within each week are merged into one layer, keeping only the last version of each item that week.
- Finally, the layers for the previous 12 months are organized into 12 groups ... and you get the idea.

You can choose whatever time periods you want, and make them as small or big as you like. The action is very flexible.

There's a special "ignore" time period that extends the period of time that layers are not merged. So if you want to keep every hourly merge for the past two weeks, set the ignore setting to 14 days. The rolling merge wil then start grouping layers starting 15 days ago.

Your first merge will take some time, because you've got a lot of layers to merge. But after that the rolling merges run pretty quickly as there are usually only a few layers to merge most days. I typically schedule my rolling merge to run once a week, followed by a compact action. The compact action will recover disk space freed by the merge.

Neil,

Another random thought: I noticed from your screen shot that you have a lot of layers in your archive. And I mean a lot of layers. Even some of my biggest, oldest, archives only approach 300 layers, and your archive has almost four times that.

I did some earlier stress testing with QRecall with archives up to 1,000 layers, but have largely used much more shallow archives in performance and regression testing.

The number of layers in an archive adversely affects its performance, and the number of layers in your archive could be causing some serious performance degradation. For example, reading the file records for a particular folder requires a lot of short reads from the archive. For external hard drive this is really fast, but for many network volumes (i.e. a Time Capsule), a lot of short reads over the network can be really slow. Now for a handful of files in a dozen or so layers, the added time might not be noticeable. But for 1,000 layers this could add up to huge delays.

So I have two questions.

First, when you verify the archive is the data transfer rate what you would expect? I ask because verify uses a special DMA mode that reads data as fast as the source can supply it, regardless of individual record size. So if verify is fast, but recalling and browsing are really slow, then it might be the number of layers that's the problem here.

Secondly, do you really need all of those layers? The idea behind the rolling merge is that you keep fine-grained changes for the past week or two, but then merge those in to more compact and efficient deltas as time goes on, first into daily layers, then weekly layers, and finally only keeping one layer per month.

Eric,

Since you've seen the earlier post, I won't repeat all that. To answer your specific questions, when QRecall can't read an extended attribute it simply notes that in the log and keeps going. QRecall considers this to be minor problem, which is why it's a "caution" message, it's not even a warning. If there was something else about that item that couldn't be captured (its directory information or any of its data), QRecall would have logged a much stronger message.

In your particular case, the system is returning BSD error 22 (invalid argument). This is a nonsensical error in this context, which often means that there's something wrong with the volume's directory structure. Try some of the remedies in in the earlier post, such as repairing the volume or making a copy of the item and stripping it of its extended attributes.

Neil,

338GB over 25 hours (yikes!) is about 3.8 MB/second. This is miserable throughput for a gigabit ethernet connection. It is, however, pretty close to the transfer speed that you'd get using AirPort.

Are you absolutely certain that you don't also have AirPort enabled on your MacBook Pro? With both an AirPort and an ethernet connection active, it's not that hard for the system to connect to the file server over AirPort. It will then stick with that interface until the volume is unmounted.

If you're certain AirPort isn't the issue, then the problem could be with the Time Capsule itself. There is no reason why a Time Capsule can't read or write to a volume at nearly 10x the speed you're seeing.

Some other random thoughts:

The fact that the QRecall monitor numbers are not changing doesn't (conclusively) mean that QRecall is completely stuck. It's possible that it's reading a very large empty region of the archive, which at your throughput speeds could take a while. Activity monitor is the easiest way to see if the QRecallHelper process is still using CPU time and if there's steady activity over the network.

I would suggest (a) repairing the volume and then (b) repairing the QRecall archive. If the Time Capsule volume is internal see Repairing your backup disk. If it's an external volume, disconnect the drive from the Time Capsule and connect it directly to the MacBook Pro. Repairing will be vastly faster and more reliable with a direct connection.

Finally, if the MacBook Pro is losing connection with the Time Capsule and the volume is unmounting during a QRecall action, this is a "very bad thing" (from QRecall's perspective). It also makes me think this is an AirPort problem; wired ethernet connections are typically pretty solid. Disconnects will play havoc with the data integrity, and could lead to corruption of the volume structure, which opens another can of worms.

Hello Neil,

Good timing. I just spent over an hour iChatting with another user about QRecall's UI and how it might be made more accessible.

Neil Lee wrote:An example - see this screenshot of my archive. I know you can turn this off, but what exactly does this communicate?

The timelines impart a lot of information (which is, in itself, part of the problem). A timeline shows you, for a given item, how many different versions of that item exist in the archive and when they were captured. Now that's pretty important information, if you want to know how many versions of file you have and when they were captured.

All of those lines are meaningless, at least to the untrained eye. I see the idea behind what they're supposed to signify, but at this scale with these many connections, they doesn't actually add anything meaningful to the experience - it's infojunk.

I have to disagree that it's "infojunk," as you say. I do agree that there's a problem if you don't know what the interface is trying to communicate. Basically, I'm trying to display all of the details of a third dimension (different versions captured over time) in a two dimensional interface. No other backup system that I know of tries to do this. Time machine, for all its snazzy UI goodness, doesn't even try. It merely shows you the items as they existed at a particular time, but won't give you the history of any individual item.

It's doubly compounded by the fact that you're in column view. When there was only list view, the timelines were manageable. But in column view, you can put considerably more items on the screen at once. Each timeline imparts a lot of information, and a lot of timelines start to overwhelm the interface.

But, as you pointed out, there's always the option of turning off the timelines if you not interested in the individual history of every item.

Finding a file within the archive is similarly overcomplex: the search field seems to imply I can search my archive, but searches there do nothing.

Ah, that's because search is currently unimplemented. See the Known Issues section of the QRecall 1.2.0(35) beta release notes.

It's possible I completely misunderstand how the archive view is supposed to work, but I guess that's specifically my point - the use is opaque to the user.

That's a very valid point, and something I've struggled with since day one. The basic concept of multiple versions of items over time is really hard to convey in an interface. That's why I came up with the Time View, which is as close as I've been able to come to graphically presenting the actual structure of the archive.

Constraining the time span of the view is also confusing, specifically because the "handles" that you drag to narrow down the time span are out of view. You have to scroll to find them, and if I didn't know they existed, I'd never find them.

That's a good point, and something I want to address.

Overall I'm happy with the mechanics of how Qurecall works - it's restored a lost Users folder and restored a full install perfectly, albeit very, very slowly -- the Users folder restore took 2 days, for example[!].

That's very strange. Recalling in almost always faster than capturing. I'd be very curious to know why your recalls are taking so long.

I hate criticizing a UI without offering some suggestions, and I'm happy to collate a list of stuff I've noticed.

I don't mind criticism, and I don't expect my users to design the UI.

My basic suggestion is that the default UI should be optimized for the primary use case for each task.

I agree.

- Chances are if you open an archive it's to find or restore files. This is difficult with the current UI
- If you need to constrain the time span, it shouldn't require physical motion (scrolling and dragging) - give me a date picker or something more effective

I agree that finding the shader handles in the current UI is a bit awkward. But more to the point, the most typical task is to identify an item to recall and then rewind the archive to a specific version of that item. Previous versions of QRecall had a set of "VCR" buttons that would allow you to move backwards and forwards in time, but stopping only at specific versions of that particular file. The current rework of the UI has lost this feature and I'm working on something to replace it.

- Search should provide more feedback - when you perform a search it looks like nothing is happening - which from my tests can sometimes be true?

Well, when the search is working again you can tell me if the feedback is sufficient.

- The view when you have multiple backups in a single archive is confusing. In the screenshot above, it's not clear why I have two instances of my system, and the presence of "Unknown" is unnerving. Where did that come from? What does that mean?

You have captured to this archive using multiple identities (identity keys). Each identity key you use creates a unique owner that keeps everything belonging to that owner separate from all of the items belonging to other owners. This allows you to safely store the backups of two computer systems in the same archive; nothing will get confused, even if the hard drive and every file name is the same.

You have an "Unknown" owner because (at some point) you repaired the archive and the QRecall recovered files but couldn't determine which owner they belonged to; these recovered files are assigned to a special "Unknown" owner.

If one of these owners is now really old/obsolete, or you want to get rid of the files belonging to "Unknown", you can select one of them and use the Archive > Delete Item... command. This will delete all of the items that belong to that owner from the archive.

These thoughts are still pretty scattered, but I wanted to at least get the gist of them out. I'm happy to spend a bit more time diagramming how I think the userflows should work but wanted to see what everyone's thoughts were.

I really appreciate your thoughts and the feedback. I'm acutely aware of some of QRecall's UI deficiencies, and I'm determined to correct them in this release.