It's not relevant to this issue.

The thread in the link seems to be delving into the conundrums introduced by file cloning and snapshots in APFS, which lead to strange things like duplicating a large file but still having the same amount of free space. (update: thread also discusses the difference between the free space as reported by the Finder and the free space as reported by Disk Utility / df, that might be related to this issue, but I'm more concerned about du.)

QRecall 2.x does not take advantage of file cloning, sparse files, or snapshots (on the archive's volume), so these shouldn't be an issue. (QRecall 3.x will leverage file cloning and sparse files, something I'm working on right now.)

No, the problem is that du is reporting more data in a archive's package folder than the total of the files in that folder. Finder is reporting the correct size, and it agrees with QRecall and the total of the POSIX sizes reported for each file. So I'm still not sure what's going on. (I've used du to check the size of my test archives on APFS volumes and they all report the correct size.)

I will say that I think APFS is still a little green. In the past three months I've had two APFS volumes became un-repairable (one of those was my boot volume) that had to be erased and reformatted. Luckily, I had backups.

I'm mystified. If you add up the sizes of those files it's clearly close to 20GB, and certainly not 150GB.

Have you tried repairing the volume?

Mikael,

QRecall identifies items in an archive strictly by path.

So if you capture your Documents folder yesterday and it contained a folder (~/Documents/Folder) and today it contains a different folder (~/Documents/New Folder) QRecall will record that Folder was deleted and it will capture New Folder.

QRecall doesn't care how New Folder got there. You might have renamed Folder. You might have deleted Folder and copied New Folder from another volume. You might have created New Folder, added some files to it, and later deleted Folder. QRecall simply records the final result.

Now, if New Folder was just a renamed Folder, then QRecall will detect that all of the files and file data are duplicates of what was already captured, and no significant amount data will be added to the archive (because it's all duplicate data).

The only disadvantage is that Folder and New Folder have separate histories. So if you're trying to find an earlier version of a file in New Folder that was previously in Folder, you'll need to search both folders. (This is where the "Show Deleted Items" view option comes in handy.)

Thanks for sending a diagnostic report.

QRecall agrees with the Finder that your archive is approximately 20GB in size. So the mystery is why du thinks there's an extra 100GB of data in there.

The next step would be to examine the details of the archive package. In the terminal start with ls -lhan /Volumes/zoo/docm.quanta and look at what's inside the package. It should look something like this:

drwxr-xr-x@ 14 501  20   448B Apr 25 17:10 .
drwxrwxr-x   8 501  20   256B Apr 16 22:34 ..
-rw-r--r--   1 501  20   264B Apr 24 11:31 displayname.index
-rw-r--r--   1 501  20   1.6M Apr 24 11:31 filename.index
-rw-r--r--   1 501  20   354K Apr 24 11:31 fill.index
-rw-r--r--   1 501  20   3.0G Apr 24 12:29 hash.index
-rw-r--r--   1 501  20    57K Apr 24 11:15 layer.index
-rw-r--r--   1 501  20    16M Apr 25 08:52 negative.index
-rw-r--r--   1 501  20   722B Apr 24 11:15 outline.index
-rw-r--r--   1 501  20   456M Apr 25 00:20 package.index
-rw-r--r--   1 501  20   2.4T Apr 25 17:10 repository.data
-rw-r--r--   1 501  20   122B Apr 24 11:31 sequence.index
-rw-r--r--@  1 501  20   1.8K Apr 24 01:10 settings.plist
-rw-r--r--   1 501  20   1.0K Apr 25 17:10 status.plist

The largest file should be repository.data; that's where all of your data is. At a distant second is the hash.index, with the rest of the files being a tiny fraction of those sizes.

Darryl,

Yes, you can delete it. The next action will recreate the file.

To limit the size and scope of the QRecall log file, visit QRecall > Preferences… > General and adjust your log file rotation settings.

I generate some pretty huge log file during testing, so I keep the log file for a week and then retain 6 week's worth.

I would suggest asking yourself how long you think you'd need to keep log information and adjust your settings accordingly.

I would suggest opening the QRecall app, go to Preferences > Authorization, and see if the Capture and recall items using administrative privileges option is checked. If it is, then you're probably OK. (Most likely, the upgrade managed to successfully replace the privileged helper.)

If this isn't checked, try checking it. Then count to 10, quit QRecall, relaunch, and see if it's still checked. If it is, again everything is likely just fine.

If it isn't, or you get another message that you need to authorize, then we've got an issue. Try deleting the com.qrecall.hoist binary, restart, and try those steps again. Follow that up with a diagnostic report and we'll dig deeper.

Steven,

First of all, Full Disk Access does not (it turns out) solve the access issues with regards to Mojave's sandboxing. The limitation is that Full Disk Access only applies to user processes running in your login session; QRecall runs as root outside your login session, and thus can't be granted Full Disk Access.

The solution was to switch to using APFS snapshots. QRecall can now make a hidden snapshot of your APFS volume and capture that. This (a) creates much more stable captures and (b) gets around the sandboxing issues.

So if you're encountering sandbox restrictions then you either aren't running the latest QRecall or your home folder is not on an APFS volume.

Start by making sure your QRecall is up-to-date. If you've relocated your home folder to a different volume, covert it from HFS+ to APFS (you can do this in the Disk Utility app and it doesn't require erasing your volume, although you'll probably need to do this from a different user or boot from an alternate partition).

Finally, QRecall should not be repeatedly asking you for authorization. If it's reporting that it is not authorized and it should be, then there's something wrong with your installation. Send a diagnostic report and we'll see if we can figure out what.

But here's something to try first. For some reason, the /Library/PrivilegedHelperTools/com.qrecall.hoist get's "stuck" (for lack of a better description). It isn't the right version, but macOS doesn't prompt you to install a new one. Try deleting that file, restart, then launch QRecall again. With any luck, it should ask to authorize and install the proper helper. If it doesn't, go into settings and tell it to use authorization.

Mike,

"Wake" and "Power On" requests are registered and scheduled with the power management processor—the hardware that actually controls the power for the system. The power manager runs even when the system is shutdown. This is the same hardware used when you go to the energy saver system preferences, click "Schedule," and tell your laptop that you want it to start up every day at 08:00.

If you're interested, you can manipulate the power manager using the pmset tool in the Terminal. For example, to list the currently scheduled wake/startup events use pmset -g sched.

If you set the "Before scheduled time: Wake" option in the schedule, QRecall will schedule a "wake" event to occur a couple of minutes before the action is next scheduled to run.

If you don't, the action will run as soon as the system wakes up again.

As a rule, scheduled actions run as soon as they can. If an action was scheduled to run at 03:00 and the system was asleep or shutdown, that action will run as soon as the system wakes up or is booted. You can make an exception for the latter case by adding an "Ignore if shutdown" condition.

In the modern macOS, laptops no longer have an explicit idle sleep time (that you can set). It is now "always" and "automatic", with a few exceptions.

The power saver system preferences settings is now divided into two panes: Battery and Power Adaptor.

In the battery section you have the ability to set the idle sleep time for the display. The shorter this time the more power you'll save, and the sooner your display will sleep. The entire computer sleeps immediately, or a short time, after this.

In the power adaptor pane you can also set a sleep time for the display. The system will also go to sleep soon afterward, unless you've set the time to "Never" or checked the "Prevent computer from sleeping automatically when the display is off" option.

So when you're not using your system, the display will dim and then power down after the time you've set, and the entire system will go to sleep soon afterwards. There are, of course, a number of exceptions to this.

Software (like QRecall) can request that the system not go to sleep. QRecall actions do this when they start running (or an action is about to start). This is what prevents the system from going to sleep while a QRecall action is working. Once the action has finished, it signals to macOS that it no longer needs to keep the system awake.

Another exception is the lid. Closing the lid of a laptop will cause the system to go to sleep. It doesn't matter what your settings are or what software is running, the system will stop. Normally this isn't a problem for QRecall, but it can be depending on your hardware (external volumes sometimes have problems reconnecting after power is restored and cause read/write errors immediately after waking up).

When you add a "Before scheduled time: Wake" to the schedule, QRecall tells the power manager to wake your system (assuming it's asleep) a minute or two before the action is scheduled to run. If the lid is closed, there's not enough battery, or the system is shutdown, this request it ignored.

The "After running actions: Sleep" option sends a request to the system that, after the QRecall action has finished running (and there are no other actions ready to start), the system should go to sleep at its earliest convenience. It is not a command. It does not force the system to go to sleep. If you're using the system, or if other software is requesting that it not go to sleep, this request is ignored. So if you want your (idle) system to go back to sleep soon after a QRecall action has finished, set this schedule option. Note that even without this option, the system will eventually go back to sleep on its own, it just might take longer depending on your display sleep time setting.

Finally, "Power Nap" is a feature added in macOS 10.8 that, on certain laptops, will occasionally "wake" from sleep to enter a very-low power mode that performs limited background tasks and then goes back to sleep. These include checking your mail, updating your calendar, performing Time Machine backups, downloading software updates, and so on. The system isn't useable and only Apple designated tasks are suppose to be running. But in real life, we've experienced problem with Power Nap that it allows some background QRecall tasks to run/start, but will often leave external drives powered down so those tasks might fail.

If you need or like Power Nap, go ahead and turn it on. Your mail will be waiting for you when you wake the system up again. On the other hand, if you have QRecall actions that start and fail during power naps, consider turning it off.

I hope that answers everything!

Oops, I got distracted and forgot to answer the other half of the question.

The compact is incremental, because (most of) what it's doing is simply relocating records in the file so all of the records are physically adjacent to one another (no empty/unused space between those records). If you stop the compact you leave some records adjacent and others not. When you start it again, it will simply resume moving the non-adjacent records.

Empty space is created following actions that remove old data from the archive, most notably a merge.

A capture action only adds new records, so a capture won't create empty space or delete records.

So a capture won't have any material impact on the next compact action that runs because it never creates empty space or non-adjacent records.

Steven J Gold wrote:... but what would happen if I attempted to take another backup (recapture) before the Compact is restarted & completes?

Just start it and find out.

Spoiler: the second action will simply wait for the compact to finish.

QRecall uses several layers of file access coordination so that no two actions are modifying the archive at the same time. (It's possible for multiple actions to read the archive at the same time; for example, you can run a recall action simultaneously with a verify or while the archive open in a browser.)

This works no matter how you run the actions. The scheduler will hold conflicting actions that are ready to run, but even if you start an action from the command line or another (networked) computer, the actions should handshake with one another so that only one is modifying the archive at a time.

Tim,

If the reindex was successful, then your archive is fine. Also, a reindex and a verify require about the same amount of time/effort which lends to the theory that this is a semi-random event.

When I get reports that a drive is un-mounting in the middle of long action (like verify) but not for other actions, I also look to the possibility that it is un-mounting at other times and there's simply no process to notice.

If you're handy in the Terminal, you might occasionally use something like fgrep 'mounted volume' ~/Library/Logs/QRecall/QRecall.log* | fgrep Seagate to look for times when your Seagate drive(s) unmount at unexpected times. (The log you uploaded only covered two days, which isn't enough time to catch something like this.)

If you find these scattered around, I suspect your drive has been spontaneously unmounting all along but only causes grief when it intersects a long running action like a verify or repair.

Tim,

I'm sorry to hear you're having problems.

I suspect the problem is in your hard drive (hardware). Less likely, it might possibly be a bug in the some of the special software you're running—although, honestly, I can't image how.

QRecall definitely does not issue unmount requests to volumes that are being verified or repaired. And even if it did, it wouldn't matter; software can only request that a volume be unmounted. The filesystem will refuse that request if there are still open files—which is obviously the case here.

The only way for a volume to get unmounted while you have open files is a spontaneous loss of connection with the device, and this is usually a hardware issue. (Full disclosure: there have been a few OS bugs that caused this too—like the infamous "FireWire sleep" bug—but right now I'm not aware of any).

We'd be happy to help you look for clues in your logs, but we'll need a lot more context. Start by sending a diagnostic report (in the QRecall app, choose Help > Send Report). Then locate your QRecall.log file, compress it, and then use a file transfer service like We Transfer to send the compressed file to support@qrecall.com.

Greetings Mike,

Each action is stored as a document in the ~/Library/Preferences/QRecall/Actions folder. Each document has a name that should make it clear what that action does.

You can move those documents, en mass, somewhere else and they will cease to be actions in QRecall. (When I'm doing testing I just move them all to a folder named Actions (disabled).)

QRecall does not expect the documents in this folder to suddenly appear or disappear, so you'll need to let QRecall know that the contents of the Actions folder has changed. Creating a new action or deleting an existing action is usually sufficient. Alternatively you can quit both the QRecall application and the QRecallScheduler process (you can do that in Activity Monitor). If that's too much hassle, a system restart will get it done.

Let us know if you have any other questions!