(Not sure whether this is related with this topic: http://forums.qrecall.com/posts/list/913.page)

Since changing two Volumes from hfs+ to apfs I have two archives that frequently fill up my hard disk for apparently no reason.
After some investigation here are my findings:

Both archives report significant lower size in Finder than within QRecall (about 40 GB vs 120 GB).

Free Space of the disk indicates that Finder is wrong (like in the other thread)

Compacting the archive brings the archive back to the small size given in Finder

Trying to automate the solution with a compact action (condition: ignore if archive size is <60GB) failed. The action is not triggered (Finder size is 40 GB, Qrecall reports 72 GB today).I would assume that the condition is looking at the Finder size.

this did not happen when the Volmes were hfs+ formatted. Source volume and archive volume are both apfs.

Both archives are triggered several times daily, have regular merge actions and mostly have small layer sizes.

I'll send a diagnostic report in a few minutes.

Thanks for looking onto the issue.

Johannes,

You are the third user that has reported something like this. And I'm beginning to suspect it's a bug in APFS. It also makes no sense to me that compacting the archive would make any difference.

I would be interested in getting some allocation information about the archive files by running the 'ls -lskn' Terminal command on the archive (particularly at a point in time when the archive size and Finder size disagree), like this:

ls -lskn /Volumes/YourVolume/Path/To/Archive.quanta

Secondly, I'd be interested to know if the disk repair tool in Disk Utility produces any anomalous output when you repair that volume.

Finally, I'd be curious to know if there are any snapshots of that volume. You can find out by issuing the command:

tmutil listlocalsnapshots /Volumes/YourVolume

James,

thanks for looking into this.

Here is part 1 of the answers:

Archiv 1:
Finder size: 42,94 GB
Qrecal size: 82,62 GB
Output of ls -lskn:

total 80155512

       60 -rw-r--r--  1 501  0        59224 10 Jul 18:06 displayname.index

    15920 -rw-r--r--  1 501  0     16301912 10 Jul 18:06 filename.index

        4 -rw-r--r--  1 501  0           56 10 Jul 18:06 fill.index

   196612 -rw-r--r--  1 501  0    201326648 10 Jul 18:06 hash.index

      576 -rw-r--r--  1 501  0       588449 10 Jul 18:06 hash_adjunct.index

     1068 -rw-r--r--  1 501  0      1088080 10 Jul 18:06 layer.index

    16388 -rw-r--r--  1 501  0     16777220 10 Jul 18:06 negative.index

        4 -rw-r--r--  1 501  0         1489 10 Jul 18:06 outline.index

   485932 -rw-r--r--  1 501  0     30081072 10 Jul 18:06 package.index

      756 -rw-r--r--  1 501  0       770428 10 Jul 18:06 package_adjunct.index

 76945184 -rw-r--r--  1 501  0  40124391024 10 Jul 18:06 repository.data

    38792 -rw-r--r--  1 501  0     39183976 10 Jul 18:06 repository_4k.checksum32

  1224892 -rw-r--r--  1 501  0   1253888000 10 Jul 18:06 repository_p4w8k32m2.0.anvin_reed_sol

     2208 -rw-r--r--  1 501  0      1224500 10 Jul 18:06 repository_p4w8k32m2.0_4k.checksum32

  1224892 -rw-r--r--  1 501  0   1253888000 10 Jul 18:06 repository_p4w8k32m2.1.anvin_reed_sol

     2208 -rw-r--r--  1 501  0      1224500 10 Jul 18:06 repository_p4w8k32m2.1_4k.checksum32

        4 -rw-r--r--  1 501  0          122 10 Jul 18:06 sequence.index

        4 -rw-r--r--  1 501  0         1814  5 Jul 09:33 settings.plist

        4 -rw-r--r--  1 501  0          874 10 Jul 18:06 status.plist

        4 -rw-r--r--  1 501  0         3872 30 Jun  2018 view.plist

Archiv 2:
Finder Size: 491,5 MB
Qrecall Size: not available (opening the Archive presents an error as reported), probably 100 GB or more

total 151814832

        4 -rw-r--r--  1 501  20        240  8 Jul 17:00 displayname.index

       36 -rw-r--r--  1 501  20      33144  8 Jul 17:00 filename.index

        4 -rw-r--r--  1 501  20        520  8 Jul 17:00 fill.index

    24580 -rw-r--r--  1 501  20   25165880  8 Jul 17:00 hash.index

        4 -rw-r--r--  1 501  20         82 24 Mai 09:00 hash_adjunct.index

        4 -rw-r--r--  1 501  20       3152  8 Jul 17:00 layer.index

        4 -rw-r--r--  1 501  20        770 11 Jun 23:00 outline.index

   339172 -rw-r--r--  1 501  20    1278000  8 Jul 17:00 package.index

      148 -rw-r--r--  1 501  20     147532  8 Jul 17:00 package_adjunct.index

151398188 -rw-r--r--  1 501  20  412997368  8 Jul 18:00 repository.data

      200 -rw-r--r--  1 501  20     201660  8 Jul 18:00 repository_8k.checksum32

    26224 -rw-r--r--  1 501  20   25812992  8 Jul 18:00 repository_p8w8k16m2.0.anvin_reed_sol

       16 -rw-r--r--  1 501  20      12604  8 Jul 18:00 repository_p8w8k16m2.0_8k.checksum32

    26224 -rw-r--r--  1 501  20   25812992  8 Jul 18:00 repository_p8w8k16m2.1.anvin_reed_sol

       16 -rw-r--r--  1 501  20      12604  8 Jul 18:00 repository_p8w8k16m2.1_8k.checksum32

        0 -rw-r--r--  1 501  20          0  8 Jul 18:00 sequence.index

        4 -rw-r--r--  1 501  20       2038 10 Jun 18:22 settings.plist

        4 -rw-r--r--  1 501  20       1016 10 Jul 18:06 status.plist

No snapshots on both Volumes.

I'll try Disk repair later (have to save boot from an other account)

Johannes

What appears to be happening is that the repository.data file?the file with all the important data?is taking up more physical (allocated) space than it actually contains.

In the case of Archiv 1, the repository.data file contained 40GB of data, but occupied 79GB of disk space. Stunningly, Archiv 2 contained only 0.4GB of data, but occupied 151GB of disk space.

While adding sparse file support to QRecall 3.0, I've noticed that APFS can sometime over allocate space for a file and that is what seems to be happening here.

My working theory is that APFS is not correctly handing pre-allocation requests. As the repository.data file grows during a capture, QRecall periodically makes pre-allocation requests at the end of the file so if the disk suddenly runs out of space, QRecall has enough "head room" to write its wrap-up metadata and session records.

To test this theory, I've built a pre-release version of QRecall 2.1.16(1) that you can download and install. This hacked version doesn't perform any preallocation of the repository.data file. It won't fix the over-allocated space you have now, but if you compact the archive and perform new captures, those new captures won't cause it to over-allocate the file?assuming my theory is correct.

Give this version and try and please keep me posted.

I compacted the Archive 1 (run surprisingly fast). Qrecall size is back to Finder size.
I installed the special version of qrecall, opened the archive: Size in Qrecall is back to 80 something GB (strange)
Again compacting (this time it took as long as usual). Qrecall Size is back to finder size.
Now let's see what happens the next days ...

Johannes

James,
it seems your theory is correct. After 2 weeks both archives have kept their size. Everything works again as expected with the hacked version.
Thanks for the quick support.
Johannes

Johannes,

Thanks for the confirmation!

Look for a new release of QRecall that works around this issue.