I ran a compact action on a 1tb archive yesterday (2014-12-01 20:40:22), which freed 106gb of space but quit before actually moving files to compact the archive. I then attempted a backup (2014-12-02 03:00:00) which didn?t run, with some connection problems between the drive and the computer which apparently corrupted the archive. My memory isn?t precise here, but I think the drive somehow unmounted itself from the computer. I repaired the archive (2014-12-02 08:03:54) and that repair ended with the drive again unmounted from the computer and the log reporting ?Data Problems Found?

Not knowing what else to do, I repaired the archive again (2014-12-02 13:33:43), and this time the repair seemed to work, though the log again listed long lists of ?Data Problems Found? (2014-12-02 15:11:27). I followed that by another backup that completed successfully.

My primary question now concerns the integrity of the archive. Does the fact that the second repair completed successfully imply that my archive is intact, or does the long list of ?Data Problems Found? imply that the repair process may have discarded parts of the archive? I can live with it either way, since I have another good archive of the same computers, but I want to understand what happened.

This is the same archive that I?ve had intermittent problems with in the past when backing up over a network, but in this case the backup drive was directly connected to the computer during all these operations. I don?t know what caused the instability with the connection between them. This drive is less than two months old, so I don?t have a solid history with it yet, but it seems to be working OK. This experience does make me feel good about having two different back sets that I swap between, though.

I?m submitting a report on these activities. The date/times in parens above are from correspond to the qrecall log entries for those particular actions. Things seem to be back to normal now and I have no urgent need for a response, so let this be a low priority for you.

Ralph

Ralph,

You can confirm the integrity of the archive by performing a verify. If the verify doesn?t complain, then everything in the archive is valid.

During the repair, however, you might have lost specific files or folders. If your drive is spontaneously disconnecting, then the archive probably has missing or incomplete data records that weren?t written before the drive disconnected. The repair will detect this. Look in the log file for items like ?Damaged directories,? ?Damaged files,? and ?Incomplete files.? These will list all of the files and folders the repair process couldn?t reassemble. These are the items you should be concerned about.

You can also look in the archive browser for layers marked as ?-Damaged-?. If the repair found any inconsistencies or missing data in a layer, it will mark that layer as ?damaged?. It will remain ?damaged? until it?s merged with a later layer that contains fresh captures of all of the suspect items. (The ?damaged? state also directs QRecall to force a recapture of those items during the next capture action.)

Since your repair followed an unsuccessful compact action, the repair will also report a bazillion ?Duplicate record? errors. That?s because the compact works by copying records from higher positions to lower positions in the archive file. If you interrupt the compact, it can leave a copy of thousands, even millions, of records at two positions in the archive. These can be ignored. The repair action simply logs the inconsistency and then ignores the duplicate.

In the logs you sent, I do see a handful of missing or damaged items, mostly in your iTunes media backup directories. These should be recaptured, but I?m guessing the rest of your archive is intact. I can?t tell for sure because the log records uploaded by the diagnostic report is incomplete. (The diagnostic report only uploads the last few megabytes of log records, and the repair action is so full of ?duplicate record? entries, that it doesn?t go back further than the last repair you performed.) If you want me to investigate this further, please email me your complete ~/Library/Logs/QRecall.log file.

Thanks for the feedback, James. I still have a couple of questions, primarily to fill in my understanding of how Qrecall works.

After the repair I ended up with two layers marked "damaged" and one marked "incomplete," if I'm remembering it correctly. Merging each with the next later layer I ended up with two "damaged" layers, which seemed to absorb the following good layers into the damaged layers rather than the reverse. (I'm not sure now if one of the was the originally "incomplete" layer, or if they were both damaged layers.) Rereading your post, it looks like that was because the subsequent layer didn't yet contain all the damaged files, and that there's no real downside to leaving them that way. Is that conclusion correct?

After my uncompleted Compact action, the unused space shown in the Status window jumped from less than 10mb to 106gb. If I understand Compact correctly, that was space that had been occupied by previously deleted files which the compaction process had erased, leaving scattered free space throughout the archive, and if the process had completed properly, those spaces would have been closed up by shifting everything to fill them in. Currently, though, qrecall simply uses that as available space within the archive. If this is correct, is there anything to be gained by running Compact again to close up those spaces, or is it just as efficient to simply let Qrecall fill them as over time?

Ralph

Ralph Strauch wrote:After the repair I ended up with two layers marked "damaged" and one marked "incomplete,?

An ?incomplete? layer is one where the capture was interrupted (stopped/canceled) leaving some items uncaptured. This is mostly a warning that the items in that layer don?t fully include all of the changed items at that time. If you subseqently merge this layer with a more recent (complete) layer, this warning goes away becuase the more recent layer will contain the most recent items—and all of them.

A ?damaged? layer occurs when the repair action finds missing or inconsistent information about files or folder in that layer. When you merge it with a more recent layer, QRecall looks for successfully captured items in the more recent layer that supersede the damaged one in the earlier layer. If all of the damaged folders and items have been successfully recaptured, then the ?damaged? status should go away. If there are still any files or folder that QRecall is not satisfied have been completely recaptured, the merged layer retains the ?damaged? status.

Now I say ?should? because I think there?s a bug in this logic. QRecall is very conservative in how it determines that items have been successfully recaptured. If there?s any doubt, it retains the ?damaged? status on those items, and that layer. In my testing here, I?ve never been able to trick QRecall into thinking that a layer is still ?damaged? after all of the suspect items have been recaptued, but occationally users report that this is the case. (This has been a difficult issue to verify and debug becuase the information I need to verify the bug has to be gathered before the merge is performed, and no one reports this issue until after the merge.)

If you have a persistent ?damaged? layer, review the log for the repair action. It will identify exactly which folders and files it found problems with. After your next capture, simply browse the archive and verify that those items were successfully recaptured in the latest layer. If they were, you're in fine shape.

After my uncompleted Compact action, the unused space shown in the Status window jumped from less than 10mb to 106gb. If I understand Compact correctly, that was space that had been occupied by previously deleted files which the compaction process had erased, leaving scattered free space throughout the archive, and if the process had completed properly, those spaces would have been closed up by shifting everything to fill them in. Currently, though, qrecall simply uses that as available space within the archive. If this is correct, is there anything to be gained by running Compact again to close up those spaces, or is it just as efficient to simply let Qrecall fill them as over time?

That?s all true. QRecall will reuse that scattered free space for new captures. But just like volume fragmentation, the process eventually results in thounds, even millions, of tiny little blocks of space that can?t be effectively reused. These eventually become a drag on almost every action, since QRecall has to keep track of them all but can?t do much with them. Once you get your drive-unmounting issue resolved, I?d suggest taking another shot at compacting the archive, espeically with a 106GB free.