These are minor issues that I don't see as problems, but things you might be interested in.

I've had a couple of backups fail recently due to hash map problems -- failure to open the hash map in one case and failure to close it in another. In both cases I reran the backup and it ran without problems.

Several backups recently have reported an inability to open an email attachment for backup. The unopenable files were ones that the Sophos Quarantine Manager reported as malware, so this was a good thing. It's not something I've seen until recently, though.

Ralph

Ralph Strauch wrote:I've had a couple of backups fail recently due to hash map problems -- failure to open the hash map in one case and failure to close it in another. In both cases I reran the backup and it ran without problems.

This could be caused by virus scanning software. As QRecall updates the index files in its package, the virus scanner will see it as a new file and attempt to scan it. Depending on the timing, this may prevent QRecall from relocating the file before the scan is complete. I've had a number of users encounter this, and the only solution (for the current version of QRecall) is to prohibit the virus scanning software from scanning the volume that contains the archive.

Several backups recently have reported an inability to open an email attachment for backup. The unopenable files were ones that the Sophos Quarantine Manager reported as malware, so this was a good thing. It's not something I've seen until recently, though.

You Quarantine manager might be preventing any software from accessing the file, which would include QRecall. I'd be very interested in seeing your QRecall log file where these events occurred. So at your convenience, please send a diagnostic report.

I don't routinely run any virus scanning software, but do use Sophos Anti-Virus to scan incoming email for malware, and that's what picked up the viruses. I doubt that that was involve in the hash map problem. I'll send diagnostic reports from both my MBP, where the issue with qrecall not being able to process PC virus files occurred, and also from from my wife's iMac, covering a couple other bad backups this week.

The 1am scheduled iMac backup on 11/28 failed due to a permissions problem. Backups on the MBP are scheduled for 3am, but normally run when I open the lid in the morning. That morning at 7am the next MBP backup failed, logged as an incorrect index.

I wasn't paying much attention to the computer on Thanksgiving and didn't notice these failures, which led to another 1am iMac backup failure on 11/29, logged as "archive file out of sequence," followed by a 7:30am failure on the MBP logged as "header file length invalid."

I finally noticed the problem on Friday (11/29) afternoon and repaired the archive, after which both computers were successfully backed up. I'm sending you reports from both.

Question: The logs on both computers told me that the index is incorrect and needs to be recreated. I just repaired the archive, since I've found that reindexing always seems to require a repair anyway, so the reindex step is just a waste of time. Does it ever make sense to do a reindex rather than just doing a repair?

Ralph

Ralph Strauch wrote:The 1am scheduled iMac backup on 11/28 failed due to a permissions problem.

And I find that strange. Going back through your earlier log records, I find quit a number of QRecall actions that fail due to permission errors, or some variation thereof. The errors I see are "refnum," "permission," and "EACCES" errors. None of which make much sense, and makes me suspect the file server or the operating system's interaction with it.

I wasn't paying much attention to the computer on Thanksgiving and didn't notice these failures, which led to another 1am iMac backup failure on 11/29, logged as "archive file out of sequence," followed by a 7:30am failure on the MBP logged as "header file length invalid."

The earlier capture (on 11/28) had completed, but while the archive was being finalized, that permissions error prevented QRecall from correctly closing the archive.

Whenever QRecall opens an archive, it verifies that all of the files in the package are current with one another. Since the package index files couldn't be finalized on 11/28, the files were out of order and couldn't be trusted. Both the MacPro and the MacBook refused to use the archive.

Question: The logs on both computers told me that the index is incorrect and needs to be recreated. I just repaired the archive, since I've found that reindexing always seems to require a repair anyway, so the reindex step is just a waste of time. Does it ever make sense to do a reindex rather than just doing a repair?

Looking back through your logs, I can only see one Reindex operation, and that one was successful. There are many problems that a reindex won't fix (invalid data, bad checksums, and so on). However, there are a lot of issues, like an out-of-sequence index file, that the reindex will correct.

The advantages of a reindex are speed and the promise that no data in the repository.data file will be altered. The amount of improved speed is variable. If you're reindexing a file on a local hard drive, you might see significantly better performance from a reindex over a repair. If you're access the volume over a network or slow external drive, you probably won't see any advantage at all and a repair would be your best choice in all cases.

The second advantage is obscure. If you're super-worried about not losing any data in the archive, a reindex will not alter any data in the main repository.data file. A repair, on the other hand, will erase any data it doesn't like or doesn't appear to belong to any file (depending on what repair options you choose), which could result in loss of data.

I'm continuing to have problems backing up my MBP over wifi. I got a new retina MBP this week and migrated to in, then connected my backup disk directly to consolidate a bunch of volumes that had accumulated over time from updates, etc., and do my first backup of the new machine. That went fine. Since then two backups over wifi have failed due to hash map problems. So the problem appears to be with my network, but I've been using qrecall this way since it was in beta and the problem is fairly recent. Do you have any suggestions of things I might do to tweak the network? I'd hate to have to go to manually mounting the backup disk in order to use it.

Ralph

To update my previous post, yesterday I again mounted the backup drive directly to the rMBP and ran a backup, without first repairing the archive. The backup ran fine, and took 6.5 hours to back up 203gb with 99% duplication, out of 450gb on the computer. I guess it was backing up so much because the computer is new, even though the same data had been previously backed up from a different computer.

So it does look like the hash map problems I'm having are network related, though the network setup is the same as I've been using for years. Do you have any suggestions of ways I can troubleshoot this further? I'd rather not have to mount the backup drive to the laptop for backups.

Ralph,

What is your drive being served by when on the network? What protocol? What version of OSX?

Steve

Ralph,

Looking at the latest diagnostic report you uploaded, I see three additional capture actions, all of which failed because of I/O errors that occurred trying to read or write to the archive.

You said that you're access the archive on a shared network volume via WiFi. The three most likely causes of I/O errors in this situation would be:

(1) The physical drive is experiencing hardware/controller problems

(2) The volume directory structure of the archive volume is corrupted

(3) You're encountering network connectivity failures (you'r loosing the WiFi connection)

It's my understanding that you also capture to this same archive from other computers, without any problems, so 1 and 2 are unlikely, but I would still run a volume repair on the archive volume and restart the file server to just to make sure.

If the I/O errors are being caused by wireless network communications, then you can fiddle with that. Try changing the channel on the base station in an attempt to reduce interference, adjust the transmitter power, and so on. (A network scan utility is handy for finding out what channels other WiFi zones in your vicinity are using and pick a channel as far from those as possible.)

As an experiment, you can connect your MacBook to the base station using an ethernet cable and see if the captures are successful. That would duplicate the fileshare/network arrangement and eliminate only the WiFi as a variable.

P.S. Given the number of failed captures, I'd recommend running a verify on the archive first, and repair it if any anomalies are found.

Thanks for the suggestions, James. I'll experiment some more. I think that one or two of the failures you noted resulted from my disconnect a backup in process by inadvertently shutting the laptop lid. Sometime a backup in process seems to survive this, and other times it doesn't.

My major concern is the the "failure to close negative hash map" errors, which do seem to be network related. While the log then reports "The archive may need to be repaired," when I've tried to rerun the backup without doing a repair it does seem to work.

Steve asked about my network details. My wifi is served by an Apple Airport Extreme (802.11n) which is several years old. The archive is mounted on an iMac connected to the AE via ethernet. Both computers are running Mavericks. I've read that Apple switched the network protocol in Mavericks from AFP to SMB2. I know nothing about network protocols so don't know if this switch could be involved, but I think my hash map problem started occurring after my upgrade to Mavericks.

I'm due to retrieve my off-site backup tomorrow and swap it out for this one, so that will let me see if the drive mechanism may be involved.

Thanks for the continuing support; I appreciate it.

Ralph

Ralph Strauch wrote:I think that one or two of the failures you noted resulted from my disconnect a backup in process by inadvertently shutting the laptop lid. Sometime a backup in process seems to survive this, and other times it doesn't.

When the laptop wakes up again, it becomes a race condition between the TCP/IP stack trying to reconnect with the server and reestablish the session it had and the next filesystem request timing out. The filesystem tries to give the network stack time to recover, but it doesn't always happen that way.