QRecall Community Forum
  [Search] Search   [Recent Topics] Recent Topics   [Hottest Topics] Hottest Topics   [Top Downloads] Top Downloads   [Groups] Back to home page 
[Register] Register /  [Login] Login 

Archive size on APFS fills up disk in a strange way RSS feed
Forum Index » Problems and Bugs
Author Message
Johannes


Joined: Dec 10, 2010
Messages: 68
Offline
(Not sure whether this is related with this topic: http://forums.qrecall.com/posts/list/913.page)

Since changing two Volumes from hfs+ to apfs I have two archives that frequently fill up my hard disk for apparently no reason.
After some investigation here are my findings:

  • Both archives report significant lower size in Finder than within QRecall (about 40 GB vs 120 GB).

  • Free Space of the disk indicates that Finder is wrong (like in the other thread)

  • Compacting the archive brings the archive back to the small size given in Finder

  • Trying to automate the solution with a compact action (condition: ignore if archive size is <60GB) failed. The action is not triggered (Finder size is 40 GB, Qrecall reports 72 GB today).I would assume that the condition is looking at the Finder size.

  • this did not happen when the Volmes were hfs+ formatted. Source volume and archive volume are both apfs.


  • Both archives are triggered several times daily, have regular merge actions and mostly have small layer sizes.

    I'll send a diagnostic report in a few minutes.

    Thanks for looking onto the issue.
    James Bucanek


    Joined: Feb 14, 2007
    Messages: 1572
    Offline
    Johannes,

    You are the third user that has reported something like this. And I'm beginning to suspect it's a bug in APFS. It also makes no sense to me that compacting the archive would make any difference.

    I would be interested in getting some allocation information about the archive files by running the 'ls -lskn' Terminal command on the archive (particularly at a point in time when the archive size and Finder size disagree), like this:
    ls -lskn /Volumes/YourVolume/Path/To/Archive.quanta

    Secondly, I'd be interested to know if the disk repair tool in Disk Utility produces any anomalous output when you repair that volume.

    Finally, I'd be curious to know if there are any snapshots of that volume. You can find out by issuing the command:
    tmutil listlocalsnapshots /Volumes/YourVolume

    - QRecall Development -
    [Email]
    Johannes


    Joined: Dec 10, 2010
    Messages: 68
    Offline
    James,

    thanks for looking into this.

    Here is part 1 of the answers:

    Archiv 1:
    Finder size: 42,94 GB
    Qrecal size: 82,62 GB
    Output of ls -lskn:
    total 80155512
    
    60 -rw-r--r-- 1 501 0 59224 10 Jul 18:06 displayname.index
    15920 -rw-r--r-- 1 501 0 16301912 10 Jul 18:06 filename.index
    4 -rw-r--r-- 1 501 0 56 10 Jul 18:06 fill.index
    196612 -rw-r--r-- 1 501 0 201326648 10 Jul 18:06 hash.index
    576 -rw-r--r-- 1 501 0 588449 10 Jul 18:06 hash_adjunct.index
    1068 -rw-r--r-- 1 501 0 1088080 10 Jul 18:06 layer.index
    16388 -rw-r--r-- 1 501 0 16777220 10 Jul 18:06 negative.index
    4 -rw-r--r-- 1 501 0 1489 10 Jul 18:06 outline.index
    485932 -rw-r--r-- 1 501 0 30081072 10 Jul 18:06 package.index
    756 -rw-r--r-- 1 501 0 770428 10 Jul 18:06 package_adjunct.index
    76945184 -rw-r--r-- 1 501 0 40124391024 10 Jul 18:06 repository.data
    38792 -rw-r--r-- 1 501 0 39183976 10 Jul 18:06 repository_4k.checksum32
    1224892 -rw-r--r-- 1 501 0 1253888000 10 Jul 18:06 repository_p4w8k32m2.0.anvin_reed_sol
    2208 -rw-r--r-- 1 501 0 1224500 10 Jul 18:06 repository_p4w8k32m2.0_4k.checksum32
    1224892 -rw-r--r-- 1 501 0 1253888000 10 Jul 18:06 repository_p4w8k32m2.1.anvin_reed_sol
    2208 -rw-r--r-- 1 501 0 1224500 10 Jul 18:06 repository_p4w8k32m2.1_4k.checksum32
    4 -rw-r--r-- 1 501 0 122 10 Jul 18:06 sequence.index
    4 -rw-r--r-- 1 501 0 1814 5 Jul 09:33 settings.plist
    4 -rw-r--r-- 1 501 0 874 10 Jul 18:06 status.plist
    4 -rw-r--r-- 1 501 0 3872 30 Jun 2018 view.plist


    Archiv 2:
    Finder Size: 491,5 MB
    Qrecall Size: not available (opening the Archive presents an error as reported), probably 100 GB or more

    total 151814832
    
    4 -rw-r--r-- 1 501 20 240 8 Jul 17:00 displayname.index
    36 -rw-r--r-- 1 501 20 33144 8 Jul 17:00 filename.index
    4 -rw-r--r-- 1 501 20 520 8 Jul 17:00 fill.index
    24580 -rw-r--r-- 1 501 20 25165880 8 Jul 17:00 hash.index
    4 -rw-r--r-- 1 501 20 82 24 Mai 09:00 hash_adjunct.index
    4 -rw-r--r-- 1 501 20 3152 8 Jul 17:00 layer.index
    4 -rw-r--r-- 1 501 20 770 11 Jun 23:00 outline.index
    339172 -rw-r--r-- 1 501 20 1278000 8 Jul 17:00 package.index
    148 -rw-r--r-- 1 501 20 147532 8 Jul 17:00 package_adjunct.index
    151398188 -rw-r--r-- 1 501 20 412997368 8 Jul 18:00 repository.data
    200 -rw-r--r-- 1 501 20 201660 8 Jul 18:00 repository_8k.checksum32
    26224 -rw-r--r-- 1 501 20 25812992 8 Jul 18:00 repository_p8w8k16m2.0.anvin_reed_sol
    16 -rw-r--r-- 1 501 20 12604 8 Jul 18:00 repository_p8w8k16m2.0_8k.checksum32
    26224 -rw-r--r-- 1 501 20 25812992 8 Jul 18:00 repository_p8w8k16m2.1.anvin_reed_sol
    16 -rw-r--r-- 1 501 20 12604 8 Jul 18:00 repository_p8w8k16m2.1_8k.checksum32
    0 -rw-r--r-- 1 501 20 0 8 Jul 18:00 sequence.index
    4 -rw-r--r-- 1 501 20 2038 10 Jun 18:22 settings.plist
    4 -rw-r--r-- 1 501 20 1016 10 Jul 18:06 status.plist


    No snapshots on both Volumes.

    I'll try Disk repair later (have to save boot from an other account)

    Johannes
    James Bucanek


    Joined: Feb 14, 2007
    Messages: 1572
    Offline
    What appears to be happening is that the repository.data file?the file with all the important data?is taking up more physical (allocated) space than it actually contains.

    In the case of Archiv 1, the repository.data file contained 40GB of data, but occupied 79GB of disk space. Stunningly, Archiv 2 contained only 0.4GB of data, but occupied 151GB of disk space.

    While adding sparse file support to QRecall 3.0, I've noticed that APFS can sometime over allocate space for a file and that is what seems to be happening here.

    My working theory is that APFS is not correctly handing pre-allocation requests. As the repository.data file grows during a capture, QRecall periodically makes pre-allocation requests at the end of the file so if the disk suddenly runs out of space, QRecall has enough "head room" to write its wrap-up metadata and session records.

    To test this theory, I've built a pre-release version of QRecall 2.1.16(1) that you can download and install. This hacked version doesn't perform any preallocation of the repository.data file. It won't fix the over-allocated space you have now, but if you compact the archive and perform new captures, those new captures won't cause it to over-allocate the file?assuming my theory is correct.

    Give this version and try and please keep me posted.

    - QRecall Development -
    [Email]
    Johannes


    Joined: Dec 10, 2010
    Messages: 68
    Offline
    I compacted the Archive 1 (run surprisingly fast). Qrecall size is back to Finder size.
    I installed the special version of qrecall, opened the archive: Size in Qrecall is back to 80 something GB (strange)
    Again compacting (this time it took as long as usual). Qrecall Size is back to finder size.
    Now let's see what happens the next days ...

    Johannes











    Johannes


    Joined: Dec 10, 2010
    Messages: 68
    Offline
    James,
    it seems your theory is correct. After 2 weeks both archives have kept their size. Everything works again as expected with the hacked version.
    Thanks for the quick support.
    Johannes
    James Bucanek


    Joined: Feb 14, 2007
    Messages: 1572
    Offline
    Johannes,

    Thanks for the confirmation!

    Look for a new release of QRecall that works around this issue.

    - QRecall Development -
    [Email]
     
    Forum Index » Problems and Bugs
    Go to:   
    Mobile view
    Powered by JForum 2.8.2 © 2022 JForum Team • Maintained by Andowson Chang and Ulf Dittmer