Bought and Installed QRecall over the weekend; been testing it. Ran into this problem.

Ran a full capture of my entire volume; minus some selected folders and items. I did the backup to an encrypted sparse (not-sparsebundle) disk image on a network drive.

It was slow; but not horribly so. Capture completed fine; took like 8 1/2 hrs. There were some warnings about "missing file icons". Don't believe those are bad; although the error says IO Exception.

Here's a portion of the log for this capture:

2008-02-19 22:28:46.225 -0600 Details Helper running with administrative privileges [2.733091.345.1.6]

2008-02-19 22:28:46.376 -0600 Details archive: /Volumes/Eidetic/Eidetic.quanta [2.733091.345.4]

2008-02-19 22:28:46.885 -0600 Details Amethyst [2.733091.345.6.1.1]

2008-02-20 00:49:10.354 -0600 Details error obtaining icon [2.733091.345.7.1]

2008-02-20 00:49:10.354 -0600 Details IO exception [2.733091.345.7.1.1]

2008-02-20 00:49:10.354 -0600 Details File: MGI ZOOM Viewer Installer [2.733091.345.7.1.3]

2008-02-20 00:49:10.404 -0600 Details missing file icon [2.733091.345.9.1.1]

2008-02-20 00:49:10.404 -0600 Details IO exception [2.733091.345.9.1.1.1]

2008-02-20 00:49:10.405 -0600 Details File: Amethyst:Users:HIDDEN:Documents:Downloads:MGIViewerBuild250C:MGI ZOOM Viewer Installer [2.733091.345.9.1.1.2]

2008-02-20 00:51:19.803 -0600 Details error obtaining icon [2.733091.345.10.1]

2008-02-20 00:51:19.803 -0600 Details IO exception [2.733091.345.10.1.1]

2008-02-20 00:51:19.804 -0600 Details File: Changing Info Color.app [2.733091.345.10.1.3]

2008-02-20 00:51:19.848 -0600 Details missing file icon [2.733091.345.9.2.1]

2008-02-20 00:51:19.848 -0600 Details IO exception [2.733091.345.9.2.1.1]

2008-02-20 00:51:19.848 -0600 Details File: Amethyst:Users:[HIDDEN]:Library:Application Support:DVD Player:Scripts:Applets:Changing Info Color.app [2.733091.345.9.2.1.2]

2008-02-20 00:51:19.867 -0600 Details error obtaining icon [2.733091.345.11.1]

2008-02-20 00:51:19.867 -0600 Details IO exception [2.733091.345.11.1.1]

2008-02-20 00:51:19.867 -0600 Details File: Loop Movie.app [2.733091.345.11.1.3]

2008-02-20 00:51:19.871 -0600 Details missing file icon [2.733091.345.9.3.1]

2008-02-20 00:51:19.871 -0600 Details IO exception [2.733091.345.9.3.1.1]

2008-02-20 00:51:19.871 -0600 Details File: Amethyst:Users:[HIDDEN]:Library:Application Support:DVD Player:Scripts:Applets:Loop Movie.app [2.733091.345.9.3.1.2]

2008-02-20 00:51:19.889 -0600 Details error obtaining icon [2.733091.345.12.1]

2008-02-20 00:51:19.889 -0600 Details IO exception [2.733091.345.12.1.1]

2008-02-20 00:51:19.889 -0600 Details File: Preview Movie.app [2.733091.345.12.1.3]

2008-02-20 00:51:19.892 -0600 Details missing file icon [2.733091.345.9.4.1]

2008-02-20 00:51:19.893 -0600 Details IO exception [2.733091.345.9.4.1.1]

2008-02-20 00:51:19.893 -0600 Details File: Amethyst:Users:[HIDDEN]:Library:Application Support:DVD Player:Scripts:Applets:Preview Movie.app [2.733091.345.9.4.1.2]

2008-02-20 00:51:29.764 -0600 Details error obtaining icon [2.733091.345.13.1]

2008-02-20 00:51:29.764 -0600 Details IO exception [2.733091.345.13.1.1]

2008-02-20 00:51:29.764 -0600 Details File: Enigmo [2.733091.345.13.1.3]

2008-02-20 00:51:29.770 -0600 Details missing file icon [2.733091.345.9.5.1]

2008-02-20 00:51:29.770 -0600 Details IO exception [2.733091.345.9.5.1.1]

2008-02-20 00:51:29.770 -0600 Details File: Amethyst:Users:[HIDDEN]:Desktop:Enigmo.app:Contents:MacOS:Enigmo [2.733091.345.9.5.1.2]

2008-02-20 05:57:27.896 -0600 Details captured: 170.9 GB (183,554,343,659 bytes) [2.733091.345.15.1]

2008-02-20 05:57:27.896 -0600 Details written: 152.7 GB (164,005,185,768 bytes) [2.733091.345.15.2]

2008-02-20 05:57:27.897 -0600 Details duplicate: 18.5 GB (19,901,327,724 bytes) 10.84% [2.733091.345.15.3]

2008-02-20 05:57:27.897 -0600 Details rate: 390.3 MB/min [2.733091.345.15.4]

2008-02-20 05:57:27.897 -0600 Details files: 1,261,795 [2.733091.345.15.5]

2008-02-20 05:57:27.897 -0600 Details folders: 262,433 [2.733091.345.15.6]

2008-02-20 05:57:27.897 -0600 Details icons: 1,312 [2.733091.345.15.7]

Immediately ran another capture; just too see how each incremental would take. Took ~45min.

Here's a portion of the log for this capture:

2008-02-20 07:10:31.740 -0600 Details archive: /Volumes/Eidetic/Eidetic.quanta [2.733092.1227.4]

2008-02-20 07:26:55.824 -0600 Details Amethyst [2.733092.1227.6.1.1]

2008-02-20 07:53:26.566 -0600 Details captured: 274.8 MB (288,150,811 bytes) [2.733092.1227.7.1]

2008-02-20 07:53:26.566 -0600 Details written: 26.5 MB (27,795,776 bytes) [2.733092.1227.7.2]

2008-02-20 07:53:26.566 -0600 Details duplicate: 249.7 MB (261,854,093 bytes) 90.87% [2.733092.1227.7.3]

2008-02-20 07:53:26.566 -0600 Details rate: 10.3 MB/min [2.733092.1227.7.4]

2008-02-20 07:53:26.567 -0600 Details files: 10,271 [2.733092.1227.7.5]

2008-02-20 07:53:26.567 -0600 Details folders: 361 [2.733092.1227.7.6]

2008-02-20 07:53:26.567 -0600 Details icons: 1 [2.733092.1227.7.7]

I then let the scheduler run at 3am the next capture; followed by a verification action.

Here's that run's portion of the log:

2008-02-21 03:00:00.927 -0600 Details archive: /Volumes/Eidetic/Eidetic.quanta [2.733093.3757.4]

2008-02-21 03:20:00.040 -0600 Details Amethyst [2.733093.3757.6.1.1]

2008-02-21 03:39:24.024 -0600 Details package list exceed 63K elements [2.733093.3757.7.1.1]

2008-02-21 03:39:24.025 -0600 Details Data exception [2.733093.3757.7.1.1.1]

2008-02-21 03:39:24.025 -0600 Details Path: [HIDDEN] [2.733093.3757.7.1.2]

2008-02-21 03:39:26.776 -0600 Details package list exceed 63K elements [2.733093.3757.8.1]

2008-02-21 03:39:26.776 -0600 Details Data exception [2.733093.3757.8.1.1]

2008-02-21 03:39:26.776 -0600 Details package list exceed 63K elements [2.733093.3757.9.1]

2008-02-21 03:39:26.776 -0600 Details Data exception [2.733093.3757.9.1.1]

2008-02-21 03:39:26.781 -0600 Details package list exceed 63K elements [4.733091.234.3560.1]

2008-02-21 03:39:26.781 -0600 Details Data exception [4.733091.234.3560.1.1]

2008-02-21 03:39:26.788 -0600 Details package list exceed 63K elements [3.733091.281.4078.1]

2008-02-21 03:39:26.788 -0600 Details Data exception [3.733091.281.4078.1.1]

The error seems to be a directory with greater than 63k files.

Then the verify following had these errors (again from the log).

2008-02-21 03:39:26.902 -0600 Details archive: /Volumes/Eidetic/Eidetic.quanta [2.733093.3772.5]

2008-02-21 03:39:26.905 -0600 Details header file length invalid [2.733093.3772.8.1]

2008-02-21 03:39:26.905 -0600 Details Data exception [2.733093.3772.8.1.1]

2008-02-21 03:39:26.905 -0600 Details Expected: 164038508712 [2.733093.3772.8.1.2]

2008-02-21 03:39:26.906 -0600 Details EOF: 164110092832 [2.733093.3772.8.1.4]

2008-02-21 03:39:26.906 -0600 Details header file length invalid [2.733093.3772.9.1]

2008-02-21 03:39:26.906 -0600 Details Data exception [2.733093.3772.9.1.1]

2008-02-21 03:39:26.907 -0600 Details Expected: 164038508712 [2.733093.3772.9.1.2]

2008-02-21 03:39:26.907 -0600 Details EOF: 164110092832 [2.733093.3772.9.1.5]

2008-02-21 03:39:26.919 -0600 Details header file length invalid [3.733091.281.4080.1]

2008-02-21 03:39:26.919 -0600 Details Data exception [3.733091.281.4080.1.1]

2008-02-21 03:39:26.919 -0600 Details Expected: 164038508712 [3.733091.281.4080.1.2]

2008-02-21 03:39:26.920 -0600 Details EOF: 164110092832 [3.733091.281.4080.1.6]

2008-02-21 03:39:26.920 -0600 Details header file length invalid [4.733091.234.3561.1]

2008-02-21 03:39:26.921 -0600 Details Data exception [4.733091.234.3561.1.1]

2008-02-21 03:39:26.921 -0600 Details Expected: 164038508712 [4.733091.234.3561.1.2]

2008-02-21 03:39:26.922 -0600 Details EOF: 164110092832 [4.733091.234.3561.1.6]

2008-02-21 03:39:27.042 -0600 Details archive: /Volumes/Eidetic/Eidetic.quanta [2.733093.3773.5]

2008-02-21 03:39:27.045 -0600 Details header file length invalid [2.733093.3773.8.1]

2008-02-21 03:39:27.045 -0600 Details Data exception [2.733093.3773.8.1.1]

2008-02-21 03:39:27.045 -0600 Details Expected: 164038508712 [2.733093.3773.8.1.2]

2008-02-21 03:39:27.045 -0600 Details EOF: 164110092832 [2.733093.3773.8.1.4]

2008-02-21 03:39:27.047 -0600 Details header file length invalid [2.733093.3773.9.1]

2008-02-21 03:39:27.047 -0600 Details Data exception [2.733093.3773.9.1.1]

2008-02-21 03:39:27.047 -0600 Details Expected: 164038508712 [2.733093.3773.9.1.2]

2008-02-21 03:39:27.048 -0600 Details EOF: 164110092832 [2.733093.3773.9.1.5]

2008-02-21 03:39:27.066 -0600 Details header file length invalid [3.733091.281.4082.1]

2008-02-21 03:39:27.066 -0600 Details Data exception [3.733091.281.4082.1.1]

2008-02-21 03:39:27.066 -0600 Details Expected: 164038508712 [3.733091.281.4082.1.2]

2008-02-21 03:39:27.066 -0600 Details EOF: 164110092832 [3.733091.281.4082.1.6]

2008-02-21 03:39:27.067 -0600 Details header file length invalid [4.733091.234.3562.1]

2008-02-21 03:39:27.067 -0600 Details Data exception [4.733091.234.3562.1.1]

2008-02-21 03:39:27.067 -0600 Details Expected: 164038508712 [4.733091.234.3562.1.2]

2008-02-21 03:39:27.068 -0600 Details EOF: 164110092832 [4.733091.234.3562.1.6]

2008-02-21 03:39:27.190 -0600 Details archive: /Volumes/Eidetic/Eidetic.quanta [2.733093.3776.5]

2008-02-21 03:39:27.193 -0600 Details header file length invalid [2.733093.3776.8.1]

2008-02-21 03:39:27.193 -0600 Details Data exception [2.733093.3776.8.1.1]

2008-02-21 03:39:27.193 -0600 Details Expected: 164038508712 [2.733093.3776.8.1.2]

2008-02-21 03:39:27.193 -0600 Details EOF: 164110092832 [2.733093.3776.8.1.4]

2008-02-21 03:39:27.194 -0600 Details header file length invalid [2.733093.3776.9.1]

2008-02-21 03:39:27.194 -0600 Details Data exception [2.733093.3776.9.1.1]

2008-02-21 03:39:27.194 -0600 Details Expected: 164038508712 [2.733093.3776.9.1.2]

2008-02-21 03:39:27.195 -0600 Details EOF: 164110092832 [2.733093.3776.9.1.5]

2008-02-21 03:39:27.199 -0600 Details header file length invalid [3.733091.281.4084.1]

2008-02-21 03:39:27.199 -0600 Details Data exception [3.733091.281.4084.1.1]

2008-02-21 03:39:27.199 -0600 Details header file length invalid [4.733091.234.3563.1]

2008-02-21 03:39:27.199 -0600 Details Expected: 164038508712 [3.733091.281.4084.1.2]

2008-02-21 03:39:27.199 -0600 Details Data exception [4.733091.234.3563.1.1]

2008-02-21 03:39:27.200 -0600 Details Expected: 164038508712 [4.733091.234.3563.1.2]

2008-02-21 03:39:27.200 -0600 Details EOF: 164110092832 [3.733091.281.4084.1.6]

2008-02-21 03:39:27.200 -0600 Details EOF: 164110092832 [4.733091.234.3563.1.6]

Thoughts?

Could the 63k error have corrupted the archive; or is something else going on? It's asking me to re-index/repair... what should I do?

rbz

The "missing file icon" messages aren't serious. They just mean that when asked for the icon of a file, the operating system returned an error instead of an icon. This typically means inconsistencies in the launch services database.

The $64,000 question (no pun intended) is, "do you have a directory with more than 63,000 files in it?" I'm honestly not sure if I've ever tested that case.

The "package list exceed 63K elements" just means that a package list exceed 63K elements. Package lists are lists of packages; They could represent a list of file or directories, but are most often used to list the data blocks that belong to a file. When they get too big they transform into a tree of packages lists, so they should never exceed the 63K limit.

There's a couple of potential problems here. It could be that some geometry of your hard drive has found a flaw in QRecall's package list management, be it a directory with 63,000 files or who knows what. However, it could also be a problem writing to a disk image. It would be easy to test that by repeating your test writing directly to a volume instead of a disk image on a volume.

I'm going to assume that the 63K error occurred during the capture, which means that the archive is most likely damaged. If you want to continue to use this archive it will have to be repaired.

It would also be helpful if you could send me your complete log file, via private mail if you prefer, following the repair. I'm interested in exactly what the complete order of events was and what problems the repair finds.

Rodd,

Thanks for sending me your logs and additional information privately. The problem is indeed related to a single folder with over 100,000 files.

The solution was remarkably simple: I just delete the code that throws an error when there are more than 63,000 items in a package list. It turns out this was old code, left over from the days when package lists couldn't exceed that number. But package lists were reworked ages ago and can now grow to more than 30,000,000 records.

If you have a folder with more than 30,000,000 files, then we might be in trouble.

Please download QRecall 1.0.0b4 and let me know how it does. To replace your current version with the new one:

- Open your Applications folder
- Drag your existing QRecall to the Trash
- Open the disk image
- Copy the new version from the disk image to your Applications folder
- Launch the new version

I also wanted to make one comment about performance. I appreciate what you're trying to accomplish by storing the archive on an encrypted disk image, but speed is not one of the benefits.

Besides writing to a network volume, every access has to go through the disk image driver which is notoriously slow. Add to that the encryption/decryption overhead and you have a recipe for truly mediocre performance.