Jeff,

In all likelihood, this is a bug. There have been a number of recent changes in how passwords are handled/obtained, and I wouldn't at all be surprised if something fell through the cracks.

Please send a diagnostic report; open the QRecall application and choose Help > Send Report. This will contain a lot of low-leve information that will help me diagnose the problem.

Ralph,

I'm traveling, so I can't take a detailed look at you logs until I get back tomorrow, but just a few notes until then....

From what I've seen, it's unlikely to be the archive itself. The errors I mentioned in the previous post are things like " device not ready" and "no such directory". These are not data or I/O errors, but rather indicate a device that has disconnected.

DiskWarrior, disk utility, et. al. are wonderful tools, but only address the volume directory structure. They say nothing about the reliability of the drive, and certainly don't speak to whether the drive has, or could, disconnect at an inappropriate time.

Finally, the "Problem closing archive" can be a bit of a red herring. Whenever there's a problem with an operation, the action will stop and then try to close the archive. If the problem was something that prevents QRecall from accessing the archive files, you'll also get a "Problem closing archive", in addition to the original problem.

More tomorrow...

Ralph,

Everything I see in the log would indicate I/O problems with the drive.

The capture starts at 12-10 10:35 and runs just fine until 10:44:21 when QRecall encounters a "Device Not Configured" error trying to write to the archive. This is a fatal I/O error that's usually associated with a device that has gone off-line. There's no way to recover from that kind of error, so the capture bails. During its exit, it tries to close the archive but encounters another I/O failure, "Operation timed out". That's when it logged the "Problem closing archive" message.

You then started a repair at 17:09. That ran swimmingly until about 21:42. That's when the error correction messages start. These are not unexpected. Since the capture was interrupted while writing to the data files, it's likely that there's a mismatch between the data that was written and the error correction codes. So what undoubtedly happened is that some of the data doesn't match some of the correction codes. This is fine, and the repair will reconstruct the codes as needed. This is confirmed, to a degree, because the invalid correction codes were for an offset very close to the offset being written when the capture failed.

Once the repair was past that bad patch, the repair resumes running and encountered no other issues until it was almost finished at 12-11 02:49. At this point, everything goes south. As it tries to finalize the archive it encounters a string of "No such volume", "No such directory", and "Device not configured" errors, as if the volume had been spontaneously unmounted or powered down. Needless to say, the repair wasn't successful.

Ralph Strauch wrote:You?ve said elsewhere that the invalid data will be discarded and that the archive will contain only valid files, and I?m a bit confused about the result of that process. Will I lose files which were found to contain invalid data, or only the versions of those files which contained the invalid data?

During a repair, QRecall scans every byte of your archive's repository.data file. Normally, every byte is accounted for. If the repair can't make sense of a string of bytes, this range of the file is erased and you get a "Data problems found" message in the log.

The question, however, is whether any of this "damaged" data means anything important. If the bad data was originally a data or file record, then it will result in the loss of one or more files. That, in turn, could leave one or more folders missing a file. You need to look beyond the "Invalid data" messages in the log and look for folder and file specific messages. For example, if a bad byte was found in the middle of a data record, that would result in the loss of a file. That lost file will record a "Discarded incomplete file" message in the log, along with the details of which file that was lost.

There are no such messages in your log, so none of the "bad" data in your archive was related to anything important. This isn't entirely surprising. After compact and merge actions finish, there are lots of regions in the file that are left empty or contain unreferenced data, the loss of which won't have any consequences for your archive.

As I said in the thread on Encryption, <http://forums.qrecall.com/posts/list/567.page>, I?ve been mounting my backup drives on the MBP because my Airport Extreme can?t read a FileVault encrypted drive. But if this problem is caused by backing up one computer to a drive mounted on another perhaps shifting from FileVault to Qrecall encryption and going back to mounting the drive on the AE would be a solution. What do you think?

Technically, there is very little difference between these two solutions. In both situations, you have an external hard drive mounted on a server (either the iMac or the Airport Basestation), running an AppleShare server, being accessed over a network by a remote client (the MacBook Pro).

The advantage of using the Airport Basestation is that the basestation runs 24/7, so you don't have to worry about when the iMac is running.

The advantage of using the iMac is that actions will be faster on the iMac (because it's directly connected to the volume) and will be more reliable (local busses almost always have lower error rates than LANs).

This should be fixed in 2.0.0b26 (just released).

Quick update: I think I found the problem with the archive window not opening. It's definitely a display problem and (as I suspected) occurs after the archive has been opened, but before the window is displayed.

Still trying to determine the underlaying cause, but at least I have a bead on it.

Ming-Li Wang wrote:Sorry for not making it clear in the first place: by "it didn't work" I meant nothing happened after I double-clicked the archive in the "Open Archive..." dialog (or clicked the "Open" button after selecting the archive). The dialog would close and that's it. Nothing else happened.

Tried it a few times just now, and made sure to wait more than 15 min. for it. Still nothing. Had to double-click in the Finder to open in able to conduct the next test.

Opening a document in QRecall (heck, practically every Cocoa app) via the open command vs. double-clicking a document in the Finder uses almost identical code; there are only tiny differences. So for one to work and the other not is profoundly strange.

If a stale lock is the cause, wouldn't it also prevent the archive from opening right away from Finder?

No, this has nothing to do with stale locks. (If it was a stale lock, you'd see a dialog that the archive was being modified.) What I do suspect is happening is that the archive browser window is failing to display for some reason, but the archive is still being opened. Then, as long as the QRecall app is running, this prevents other actions from modifying the archive—very much an active lock.

That gave me an idea: after canceling the delete job and quitting QR to close everything, I reopened QR and made a manual capture from the Actions window. The capture waited about 5 min. before going into action. The is probably due to the stale lock you mentioned.

That sort of confirms my suspicions that the failed attempt to open the archive earlier was keeping the archive locked for modifications. By quoting the QRecall app, the lock on the archive was abandoned, allowing the other actions to (eventually) break the lock and proceed.

I recommend running a verify to determine if the archive is OK or needs repair. I can't tell from the information here.

Also, send a diagnostic report form the MBP so I can look into the details.

Ming-Li Wang wrote:Today when trying to open an archive from "File - Open Archive ..." dialog, it didn't work (it's been a while since I last opened the archive, so it's no longer on the recent archives list). Double clicking on the archive in Finder did work, but then it can't be closed, either by clicking on the window control button or with the "File - Close" command (or cmd-W, for that matter).

What exactly do you mean by "it didn't work"? Looking at the logs, I don't see anything that would indicate a problem opening the archive. Was the window just blank? Was it taking forever to display the layers? ... ?

Had to quit QR to close it. I've tried it several times with the same result.

Looking at the log, I did discover a bug that can occur when closing an archive (either to run a command or to close the window). That bug, inadvertently, left the shared read mode set for the archive, which is I think the cause of your later difficulties.

I even tried it once on another desktop (running the same beta version, but on Yosemite); same result.

Yes, once the archive has a stale lock on it, any QRecall process that tries to access it is going to get stuck.

The problem is not limited to opening and closing. I can't delete any item on it, nor can I remove any layer, merge layers, compact the archive or combine volumes. (The archive has two volumes of the same source directory after I installed El Capitan, so I want to combine them.) The action would appear to be starting, but would be "waiting for archive" to no end.

Be patient. Once an archive acquires an orphaned access lock, it take a while for QRecall to determine that the lock is stale and reset it. It does this automatically, but it takes about 10 minutes for QRecall assure itself that there are no other processes accessing the archive at a the same time.

After clicking "Stop" to cancel the action, there's an entry in the log that says "Problem encountered while closing archive", so it appeared to be related to the closing problem after all.

That's actually a message from the browser, and is related to the bug I mentioned earlier (which I just fixed).

Verifications (numerous times) found no error. Regular captures seem to be fine as well. I also successfully compacted it from the Actions window without opening the archive, but the action waited more than 10 min. before really doing anything.

I suspect all is well now. You waited the magic 10 minutes for QRecall to determine that it was safe to access the archive. Once it had, the access lock for the archive was reset and actions will now proceed normally.

I have never benchmarked FileVault, so i can't comment on its performance relative to QRecall's. I would expect, however, that they would be comparable. QRecall uses the encryption engine built into OS X, which is what I assume FileVault uses.

There are several aspects of QRecall's encryption that differ from whole disk encryption:

QRecall allows archives to be encrypted on volumes that don't support encryption.

Each archive has its own key. A single volume can contain multiple archives, each encrypted with a different key.

Other processes that have read access to the archive won't see any plaintext data.

An archive key is a completely random AES key, not derived from a password, so it has maximum entropy. This means that the encryption is stronger than the password (and immune to password crackers).

Because the key isn't based on a password, the archive's password can be changed without re-encrypting the archive.

Adrian Chapman wrote:It is always a Macbook archive that shows up as not being run for x days, but when I check the actions have been running correctly. I should add that all my verify, merge and compact actions run from my iMac.

Adrian,

Can you clarify a few things? Are you capturing to the same archive on the NAS, or separate archives for the iMac and Macbook? Is the status message you're talking about the one on the Macbook or the iMac? Does the iMac and Macbook show different status message for the same archive? Is the NAS connected to the Macbook all of the time, or only during the capture? Does the Macbook capture occur when it's logged in or out?

When archives are shared across networks, keeping the status information up-to-date across all participants gets a little tricky.

David Ramsey wrote:There was an interrupted backup 10 days ago. But every backup since then has gone perfectly (I checked the log file).

I think I've discovered where the confusion lies. You have (or had) two archives. You were capturing to the older archive, but then you deleted all of the actions for that archive, created a new archive (with a different name), and created new actions for the new archive. The new archive is working just fine, but the old archive is no longer being captured to.

That's the reason it's status is giving you that warning.

Now, if the old archive is really dead and/or gone, then you can remove it from the status window by right/control+clicking on its status and choosing the Forget command. The status window will stop tracking that archive (unless you perform another action on it, then it will start monitoring it again).

Very odd.

If you haven't already done so, please send a diagnostic report. Then, if you have the time, please locate the following files and send copies to support@qrecall.com:

1) The files in the folder ~/Library/Preferences/QRecall/Status

2) Locate the archive that's being captured, right/control+click on its icon and choose Show Package Contents. Find its status.plist file.

Send me those files and I'll have a look.

Bob Tyson wrote:James, here a new report. I read your latest reply in the Forum. The Status Window now shows red for the 1TB-xx archive, on the first line with the note ?Captured 2 days ago?.

That indicates that this archive hasn't performed a capture in 2 days. QRecall monitors the frequency at which you normally capture items to an archive, and raises a warning (in the status window) if that frequency suddenly drops off. In this case, I suspect that you regularly capture to this archive more than once a day. The status window is saying that nothing has been captured in over 2 days now, which might indicate a problem.

For a damaged layer, if I perform a merge, does this mean that only latest versions of files will be retained?

That's correct. Each layer embodies the changes that occurred since the previous layer was captured. When you merge two or more layers, only the latest version of those items is retained in the resulting layer.

Our concern is that files from earlier layers be retained (so long as not damaged) in order to maintain the integrity of the archive and not lose older 'original' photographic images.

Since you didn't use any of the special repair options, none of the items in the layer are damaged. Said another way, following a repair (with default options) all the items remaining in a layer will be intact and complete. The issue is items that might have been lost due to corrupted data.

In the browser, you can narrow the focus to a single layer using the layer shades and review just the items in that layer. (Tip: select the layer and choose "Shade Unselected Layers".) The items you now see in the browser are just the items captured in that layer. They are all fine, but there may be items that are missing because they were lost during the repair—and we have no idea what those are, because that data was corrupted.

The rest of your log looks good. I see nothing alarming.

One last remark, I truly appreciate your patience and support. Perhaps it helps that I'm posting at hours when you may be sleeping or at least not at your screen, and vice versa. Kind of moderates the pace?

It does, but maybe that's a good thing.

Bob Tyson wrote:Monitor 2015-11-28 03:15:41 Minutia Capture history for archive 1TB_FOTO_ARCHIVIO_PORTABLE_HD.alias appears to have been deleted
Monitor 2015-11-28 03:21:37 Minutia Capture history for archive 1TB_FOTO_ARCHIVIO_PORTABLE_HD.alias appears to have been deleted
Monitor 2015-11-28 04:23:43 Minutia Capture history for archive 1TB_FOTO_ARCHIVIO_PORTABLE_HD.alias appears to have been deleted
...

That's a bug. In this case, QRecall discovered a status file for a version of 1TB_FOTO_ARCHIVIO_PORTABLE_HD that had at some point been deleted, or overwritten. It's supposed to log this message once and delete the (orphaned) status file. The delete step was broken.

QRecall 2015-11-28 09:12:40 Warning Report generator exited with status 15; please report this to the developer

Odd, and I'll look into it, but I'm not overly concerned about it. Apparently an problem occurred trying to sample some of QRecall's background processes in preparation of uploading a diagnostic report. But the report came through, and was intact, so I'm guessing this was a transient problem.

As I in part noted with the report, here are things that now don't quite look right:

1. Yes I clicked in the Status Window to have QR 'forget' the 1TB_FOTO_ARCH - xx history since that archive no longer exists and I have created a new archive for that external HD. But having the log show the message repeatedly like this?

Unfortunate coincidence, and unrelated to the real problem.

2. Archive 1Tb-xx shows in the status window as amber but I can't see why. No pending actions; a Verify went to completion today. Still amber.

Expand the status for 1Tb-xx and see which status is amber?capture, verify, or size?and what the status message is. The status indicators change for a variety of reasons, many unrelated to the health of the archive.

3. Archive MILLIE-xx is in the green on the Status Window but opening the archive shows that layer 0 is damaged:
Layer 0 -- Date UNKNOWN -- Size UNKNOWN -- Items DAMAGED.
---->> How should I regard this? What should I do to recover the data in this layer?

A damaged layer indicates that one or more items in that layer was, at some point in time, lost during a repair. It's a concern only if you recall items from that layer, as some items may be missing or incomplete. Eventually, the missing data will be replaced when that layer is merged with subsequent layers, and the "damaged" indicator will go away.

Your other issue, regarding not being able to repair your other archive, has also been fixed in 2.0.0b25, which should appear on the server shortly. Update to the latest version and try that repair again.