There are actually three archive size limits.

QRecall's internal limit is now 6TB. It was 2TB—and still is if you happen to be running one of the (very old) 32-bit versions of QRecall/OS X.

The second limit is the maximum file size supported by the archive's volume. This varies from one format to another, but if there is a limit it's typically either 2TB or 4TB. For example, a USB volume factory formatted for MS-DOS (AKA FAT32) can't store files larger then 4TB.

Finally, QRecall tries to prevent the archive from growing until it fills up the entire volume.

If you're hitting the 2TB limit, you need to upgrade your OS. If you're hitting the 6TB limit, splitting your archive is a good solution (and will probably improve performance). If you're hitting the file size limit, consider reformatting the volume to a format the supports larger files or split your archive. If your volume is full, you probably need a second/larger volume.

There's no way to split an archive directly. (This has been a requested feature, but is still on the wish list.) However, it's pretty easy to do manually.

Decide on a division of content. In my personal system, I capture my iTunes folder (which is pretty huge) to a second archive, and everything else to my primary archive.

Duplicate your archive.

In one archive, delete everything you plan to capture in the second archive.

In the second archive, delete everything you'll be capturing in the first archive.

Compact both archives.

In the first (say the "everything else") archive's settings, exclude the item(s) you plan on capturing to the second archive.

Continue capturing everything to the first archive (the exclusion rule will omit the items you plan to capture to the second archive).

Create a new capture action to the capture the excluded item(s) to the second archive.

Make a copy of all of the maintenance actions for the first archive (merge, compact, repair, ...), and change the archive so those same actions are performed regularly on the second archive.

It's a good idea to suspend your scheduled actions while you're performing this kind of surgery.

Sorry, but no. QRecall captures and restores volumes.

Darryl,

QRecall will certainly reduce the data to its minimum amount. And don't forget to turn on compression too!

Creating a disk image of the archive won't provide many benefits and will actually make the archive a little larger.

---If, on the other hand, you want to create the absolute smallest possible file to upload consider creating your archive with compression off and then create a compressed disk image (disk image compression is more aggressive than QRecall's) Warning: that is going to take a long time and need double the space of the archive.---

The biggest problem I see is reliably uploading a single, gigantic, file to the server in one shot. 2TB of data, even if you have a 100Mb/sec internet link, it going to take a couple of days to upload. If Google Drive's software will handle partial transfers, failures, and restarts, etc., then use that. Otherwise, I'd look for software that can deal with interruptions or slicing up the archive (or the image of the archive) into smaller pieces and upload those.

Finally, if you go the disk image route, consider turning on a modest amount of data redundancy (say 1:8) when you create the archive. That much data over a WAN is at risk of dropping a few bits here and there. Downside, it will make the whole archive 12% bigger.

Ernest,

QRecall does not read or attempt to capture the EFI partition of a volume.

In macOS, the EFI partition really isn't used (for booting at least), and isn't modified in a way that would require it be preserved or restored. If you repartition the device using Apple tools, a new (empty) EFI partition will be created.

The problem with taking snapshots of an archive volume is there are index files in a QRecall archive that get completely rewritten every time a capture or merge is performed. The data is mostly unchanged, but the filesystem doesn't know that.

If there are snapshots, the copies of these index files will consume a fare amount of space. Add to that the changes being made to the other files, and it starts to add up. I'm not worried about Time Machine snapshots because macOS is smart enough to discard them if you start to run out of disk space. But it will cause a discrepancy between what you think should be the free space and the actual free space.

So if this is the discrepancy, it might be something you can just ignore.

Because of the issues I've encountered with APFS volumes getting corrupted, and since you mentioned that you have the space available to move the archives to a different volume, I'd suggest copying the archive to another volume, repartition and reformat the APFS volume, then move the archives back.

Also, you mentioned that "I take a system snapshot of the boot volume", but we're talking about the volume the archives are on, right? There shouldn't be any snapshots of 'zoo'. (It doesn't make any sense to make a snapshot of an archive volume, since archives are literally a collection of snapshots/layers.) If there are, that could be the problem?at least the root of the free space problem.

It's not relevant to this issue.

The thread in the link seems to be delving into the conundrums introduced by file cloning and snapshots in APFS, which lead to strange things like duplicating a large file but still having the same amount of free space. (update: thread also discusses the difference between the free space as reported by the Finder and the free space as reported by Disk Utility / df, that might be related to this issue, but I'm more concerned about du.)

QRecall 2.x does not take advantage of file cloning, sparse files, or snapshots (on the archive's volume), so these shouldn't be an issue. (QRecall 3.x will leverage file cloning and sparse files, something I'm working on right now.)

No, the problem is that du is reporting more data in a archive's package folder than the total of the files in that folder. Finder is reporting the correct size, and it agrees with QRecall and the total of the POSIX sizes reported for each file. So I'm still not sure what's going on. (I've used du to check the size of my test archives on APFS volumes and they all report the correct size.)

I will say that I think APFS is still a little green. In the past three months I've had two APFS volumes became un-repairable (one of those was my boot volume) that had to be erased and reformatted. Luckily, I had backups.

I'm mystified. If you add up the sizes of those files it's clearly close to 20GB, and certainly not 150GB.

Have you tried repairing the volume?

Mikael,

QRecall identifies items in an archive strictly by path.

So if you capture your Documents folder yesterday and it contained a folder (~/Documents/Folder) and today it contains a different folder (~/Documents/New Folder) QRecall will record that Folder was deleted and it will capture New Folder.

QRecall doesn't care how New Folder got there. You might have renamed Folder. You might have deleted Folder and copied New Folder from another volume. You might have created New Folder, added some files to it, and later deleted Folder. QRecall simply records the final result.

Now, if New Folder was just a renamed Folder, then QRecall will detect that all of the files and file data are duplicates of what was already captured, and no significant amount data will be added to the archive (because it's all duplicate data).

The only disadvantage is that Folder and New Folder have separate histories. So if you're trying to find an earlier version of a file in New Folder that was previously in Folder, you'll need to search both folders. (This is where the "Show Deleted Items" view option comes in handy.)

Thanks for sending a diagnostic report.

QRecall agrees with the Finder that your archive is approximately 20GB in size. So the mystery is why du thinks there's an extra 100GB of data in there.

The next step would be to examine the details of the archive package. In the terminal start with ls -lhan /Volumes/zoo/docm.quanta and look at what's inside the package. It should look something like this:

drwxr-xr-x@ 14 501  20   448B Apr 25 17:10 .
drwxrwxr-x   8 501  20   256B Apr 16 22:34 ..
-rw-r--r--   1 501  20   264B Apr 24 11:31 displayname.index
-rw-r--r--   1 501  20   1.6M Apr 24 11:31 filename.index
-rw-r--r--   1 501  20   354K Apr 24 11:31 fill.index
-rw-r--r--   1 501  20   3.0G Apr 24 12:29 hash.index
-rw-r--r--   1 501  20    57K Apr 24 11:15 layer.index
-rw-r--r--   1 501  20    16M Apr 25 08:52 negative.index
-rw-r--r--   1 501  20   722B Apr 24 11:15 outline.index
-rw-r--r--   1 501  20   456M Apr 25 00:20 package.index
-rw-r--r--   1 501  20   2.4T Apr 25 17:10 repository.data
-rw-r--r--   1 501  20   122B Apr 24 11:31 sequence.index
-rw-r--r--@  1 501  20   1.8K Apr 24 01:10 settings.plist
-rw-r--r--   1 501  20   1.0K Apr 25 17:10 status.plist

The largest file should be repository.data; that's where all of your data is. At a distant second is the hash.index, with the rest of the files being a tiny fraction of those sizes.

Darryl,

Yes, you can delete it. The next action will recreate the file.

To limit the size and scope of the QRecall log file, visit QRecall > Preferences… > General and adjust your log file rotation settings.

I generate some pretty huge log file during testing, so I keep the log file for a week and then retain 6 week's worth.

I would suggest asking yourself how long you think you'd need to keep log information and adjust your settings accordingly.

I would suggest opening the QRecall app, go to Preferences > Authorization, and see if the Capture and recall items using administrative privileges option is checked. If it is, then you're probably OK. (Most likely, the upgrade managed to successfully replace the privileged helper.)

If this isn't checked, try checking it. Then count to 10, quit QRecall, relaunch, and see if it's still checked. If it is, again everything is likely just fine.

If it isn't, or you get another message that you need to authorize, then we've got an issue. Try deleting the com.qrecall.hoist binary, restart, and try those steps again. Follow that up with a diagnostic report and we'll dig deeper.

Steven,

First of all, Full Disk Access does not (it turns out) solve the access issues with regards to Mojave's sandboxing. The limitation is that Full Disk Access only applies to user processes running in your login session; QRecall runs as root outside your login session, and thus can't be granted Full Disk Access.

The solution was to switch to using APFS snapshots. QRecall can now make a hidden snapshot of your APFS volume and capture that. This (a) creates much more stable captures and (b) gets around the sandboxing issues.

So if you're encountering sandbox restrictions then you either aren't running the latest QRecall or your home folder is not on an APFS volume.

Start by making sure your QRecall is up-to-date. If you've relocated your home folder to a different volume, covert it from HFS+ to APFS (you can do this in the Disk Utility app and it doesn't require erasing your volume, although you'll probably need to do this from a different user or boot from an alternate partition).

Finally, QRecall should not be repeatedly asking you for authorization. If it's reporting that it is not authorized and it should be, then there's something wrong with your installation. Send a diagnostic report and we'll see if we can figure out what.

But here's something to try first. For some reason, the /Library/PrivilegedHelperTools/com.qrecall.hoist get's "stuck" (for lack of a better description). It isn't the right version, but macOS doesn't prompt you to install a new one. Try deleting that file, restart, then launch QRecall again. With any luck, it should ask to authorize and install the proper helper. If it doesn't, go into settings and tell it to use authorization.

Mike,

"Wake" and "Power On" requests are registered and scheduled with the power management processor—the hardware that actually controls the power for the system. The power manager runs even when the system is shutdown. This is the same hardware used when you go to the energy saver system preferences, click "Schedule," and tell your laptop that you want it to start up every day at 08:00.

If you're interested, you can manipulate the power manager using the pmset tool in the Terminal. For example, to list the currently scheduled wake/startup events use pmset -g sched.

If you set the "Before scheduled time: Wake" option in the schedule, QRecall will schedule a "wake" event to occur a couple of minutes before the action is next scheduled to run.

If you don't, the action will run as soon as the system wakes up again.

As a rule, scheduled actions run as soon as they can. If an action was scheduled to run at 03:00 and the system was asleep or shutdown, that action will run as soon as the system wakes up or is booted. You can make an exception for the latter case by adding an "Ignore if shutdown" condition.