The fix for this problem is in QRecall version 2.2.12.

Available at a "Check for updates..." menu near you.

QRecall will attempt to auto-mount, and eject, the volume containing the archive. The capture assumes items to be captured are already reachable, and will simply ignore any that aren't.

There are two (simple) strategies for capturing intermittently available items. One is to create a captured action triggered by a "Capture Item Volume Connects" event schedule. This will trigger the capture action to start as soon as the volume containing the item to capture is mounted.

Another alternative is to schedule the action to run at regular intervals, but use a schedule condition to "Hold if no capture items". That will queue the action to start, but suspend it until the item(s) appear to be online.

Both of these assume that the items are occasionally mounted by you or some other process. (If they're never mounted, it's assumed there are no changes to capture.)

If you have volumes that never get mounted, you'll need to do some work. The simplest would be to write a prolog script that mounted the items to capture, then attach that script to the capture action. When the capture action starts, it will run the script, which should mount the item, and then let the capture proceed.

Chris Caouette wrote:Part 1: What I am wondering is I'd love to have an Archive called "Archived Projects" but those projects can come from different drives. Rather than place them onto another drive first I'd prefer, if possible, to place them into a QRecall archive under a single folder.

QRecall can capture any arbitrary set of items (files, folder, entire volumes) in a single capture action. The files and folders don't need to be on the same volume.

You can also just do them manually if you don't need to automatically capture changes. When you're ready to archive a project, just drag it into the archive's browser window. Or use the Services extension right in the finder: Control/Right+click Services > Capture to QRecall Archive.

If you want them to be neatly organized in a single folder of the archive, that will take a little fiddling because the archive is organized around where the original item actually was. But you could create a folder just for archiving projects, and when you ready to archive one just put a copy in there and capture it. Once captured, you can discard that copy of the project.

Part 2: I have been keeping duplicate backups (backup of a backup). If I use Carbon Copy Cloner to keep copies of QRecall archives in another location, when the QRecall archive changes, will it just be an entirely one large new file that gets copied, or would CCC potentially only see the changes? Not sure you could make an Archive of an Archive. The example here is I have an archive called "Projects" on a networked drive. If I want to make an overnight update of that on another drive is it going to copy the entire file or just parts?

It will copy the entire archive (or at least most of it).

QRecall 3.0 is adding intelligent and efficient archives of archives, which avoids those problem and is specifically designed to play nice with file synchronization. Look for a beta next month.

My suspicion is that actions running as the user are updating the status just fine. But the capture action?that runs as root?is not.

I'll dig into this and see what the problem is. I'll post again here when there's a fix.

Steve,

Thanks for sending a diagnostic report.

Your "Status Window" command is greyed out because you don't have any archive status files in your ~/Library/Preferences/QRecall/Status folder. These status files are (or should be) automatically updated by capture, compact, verify, and various repair actions.

But it's obvious you've been running capture actions, yet this folder is still empty. I (obviously) suspect a bug. I'm just not sure where, as this mechanism has been working for many years.

When a capture ends, it writes a summary of the archive's status to a small "status.plist" file inside the archive package and writes a copy inside QRecall's Preferences folder. So let's start by teasing out what, and what isn't, happening when it should.

Let me first ask you locate a recently captured archive in the Finder, right/control-click on it, and choose the "Show Package Contents" command. Inside the package see if you have a "status.plist" and let me know what the last modified date and time are.

Secondly, run a verify on any of your archives. Afterwards, see if the status window becomes enabled.

Finally, send another diagnostic report after you've done those things. That should give me enough information to figure out what the problem is.

Thanks!

Steven,

In general, you've got it in the right order: capture, merge, compact, verify.

Merge has nothing to do until something is captured and time has passed.
Compact has nothing to do until something is merged (or manually deleted).
Verify always verifies everything, but will be slightly faster after a merge and compact because there's less data to verify.

Now, if your goal is for your computer to spend less time maintaining the archive, there are some techniques for running the big actions less often.

Using less aggressive actions

Compact is optimized so that if there's nothing to remove (post merge), then the command will finish immediately. Similarly, if the compact determines that only a small amount of space would be recovered, it also stop immediately and reevaluates again the next time it's run.

So the capture and compact command can be scheduled to run every time the volume connects, and they won't waste hardly any time.

Merge, like compact, will first determine if there's anything to do and, if not, will stop almost as soon as it was started. So you can schedule the merge to run repeatedly, but if you configure it so the first tier to merge is broad (say with week tiers), then there will only be layers to merge once a week. If the first tier to merge is month tiers, then it will kick in once a month. And, subsequently, the compact won't have anything to do until something is merged.

Using schedule conditions

Finally, you probably don't need your verify to run daily (unless you're paranoid). Once or twice a month should be sufficient. To accomplish that, you'll need to do a little schedule hacking.

Change your the verify action's schedule to something less often (say the 2nd and 4th Friday of the month). Now the verify will start at a specific time. But what if your archive isn't connected? If not, the action will simply fail with an "archive not found" error.

The trick is to add a "hold if no archive" condition to the action's schedule. This tells the action to start, but if the archive isn't online it will simply pause until it is. In your activity monitor you'll see "Verify My Archive ... waiting for archive". When you do finally connect it, the waiting action will immediately resume, perform its action, and go away until the next interval.

You can also use this technique on the other actions too, as an alternative.

Bonus tip: if the entire drive is dedicated to QRecall archives, you might consider adding a "cancel if free space ..." condition to your compact action's schedule, which will prevent the compact from running at all until the free space of your drive drops below a particular threshold. This usually prevents the capture action from doing anything for months, and then?boom!?it performs one massive (highly efficient) compact, and it's done for another season.

Let use all know if that helps.

No silly questions...

Archives are automatically added to the status window whenever they are captured or verified.

The "Forget?" command is for archives that no longer exist, or you are not actively using, and you don't want to be constantly reminded of the fact that you haven't captured any new items to them in an overly long time. (There are also options to silence the capture and verify alarms, without forgetting the archive entirely.)

If you've accidentally forgotten an achieve you are still using, it will eventually reappear on its own.

Nico Dudek wrote:If I would restore the archive again AFTER doing all my hardware upgrade work again on the external Blade SSD would this be incremental
or would QR do it from scratch?

QRecall also performs incremental/smart restores.

When restoring a group of files, QRecall compares each existing file to the one in the archive. If they appear to be identical, it skips the file and moves on to the next.

And even if it decides to overwrite the file with the version in the archive, it actually starts by reading the file and comparing it, byte-by-byte, with the version in the archive (rather than blinding overwriting the same data). This is because reading is generally faster than writing and it avoids overwriting SSD sectors when it doesn't have to. (Flash memory blocks have a limited number of write cycles, so every write shortens its lifespan.)

So your second restore should be pretty quick and won't age you new SSD.

Moving forward...

You'll eventually want to switch to capturing your new SSD volume once your migration is finished. This might happen automatically once your SSD is your startup volume. If it doesn't, edit the capture item in the action.

The first post-migration capture will create a new volume in your archive.

Since this volume is essentially the same set of files that was on your original Fusion Drive, you'll want to Combine the two volumes so it appears in your archive as a single volume with a single history. (See Help > QRecall Help > Guide > Advanced > Combine Items)

Enjoy your new SSD!

Nico Dudek wrote:Does this mean I have to redo the restore on my blade SSD again?

No, the restore is fine.

What I was simply pointing out is that ownership and permissions are a requirement for startup volumes and home folders.

QRecall knows this and thoughtfully restored ownership and permissions before it started the restore.

So if you go check, you'll see that ownership and permissions has been set again on that volume. (Note that the macOS installer will also force ownership and permissions back on...)

I think I solved the problem: There was no full disk access checked for "com.qrecall.hoist"

Security will definitely make restores harder.

Just FYI, the "Ignore ownership and permission" option must be turned off when restoring items on a startup volume.

And if you look at your log, you'll see that QRecall set that option back when it started the restore:

2021-01-26 09:28:40.549 Setting "uses ownership and permission" on volume Catalina-1TB-BladeSSD

Sorry to hear you're having problems.

The log you sent would indicate that nothing was recalled, which is very strange. Particularly since you weren't booted from the Catalina-1TB-BladeSSD volume at the time. (It's not unusual to encounter permissions issues when recalling to the booted system volume, but it doesn't interfere with recalling the rest of your files.)

Here's my suggestion:

(1) Delete the Catalina-1TB-BladeSSD and Catalina-1TB-BladeSSD-Data volumes.
(2) Create and format an empty volume (make sure it's formatted APFS)
(3) Use the Restore Volume To... command to restore your Macintosh HD to the new, empty, volume.
(4) If that's successful, then run the Catalina installer and install macOS on top of the files you've restored.

If it's not successful (for the most part), then something else is very wrong. Please send a diagnostic report (choose Help > Send Report in the QRecall application). The report will include your recent log records as well as some information about your system and configuration.

Steven Haver wrote:Hello, I am trying to keep some large files out of my archive. My question is: is there a way that I can exclude files in a specific path (folder and sub folders) above a certain file size?

Unfortunately, the short answer is no. File exclusions (which are already pretty complex), at the end of the day, simply include or exclude files. There's no "exclusion math" that can be made to make discrete decisions for individual items.

Unfortunately, that is proving not to be an ideal solution either because when the files are not all together it increases the chances of me making a stupid human error and deleting the wrong file etc. So, ideally, I would be able to leave all the project files together in the same folder but exclude the large files from being captured to the archive. Any ideas? Thanks!

I have two suggestions.

My first suggestion would be to keep the arrangement you have now (leaving the huge files in a TEMP or DO NO BACKUP folder), and your working project files in individual project folders.

If one of those huge files really belongs to a project, create an alias or symbolic link in that project folder to the (correct) huge file in the TEMP folder. For the most part, it will be as if that huge file is actually in your project folder (you can see it, preview it, open it, drag it into applications, ...) but the actual file will be in the TEMP folder, where it will not get captured.

My second, and slightly-more-technical, suggestion is to use an exclusion pattern to exclude all folders below a particular folder level with a specific name (like DO NOT BACKUP).

In this scheme you'd keep all of your files (both large and small) inside a top-level Projects folder, organized into project subfolders, however you like. Then, wherever you need one, create a DO NOT BACKUP sub folder and move some very large files into that sub-folder. For example:

Projects/
    News Project/
        Exclusive Video Footage.mv4
        Graphics.psd
        Voice Over.mp3
        DO NOT BACKUP/
            Hours of Surveillance.avi
    Holiday Project/
        Jingle Bells.mp3
        DO NOT BACKUP/
            Unedited Santa Clause Interview.mkv
    ...

Now select your top-level Projects folder in the Finder. From the menu, choose Finder > Services > QRecall Capture Preferences...

Locate the Exclude Patterns in the capture preferences. Click the plus button and choose Add glob Pattern. Edit the pattern so it looks like this:

(empty) // DO NOT BACKUP [∞]

Leave the first field empty (no path), the pattern set to the exact name of the folders to exclude, and make sure the infinity symbol option is checked. This exclusion patten will, starting at the level of your Projects folder, exclude any item (file or folder) that is named, exactly, NO NOT BACKUP. It doesn't matter how many DO NOT BACKUP folders you create, or how deeply they're nested in other folders, they all get excluded.

Alternatively you could use a filename convention, rather then creating specially named folders. To do that, you'd use a glob pattern like this:

(empty) // EXCLUDE* [∞]

That glob pattern will match any file or folder that begins with the word EXCLUDE. (If you want this to be case sensitive, you'll have to use an expert pattern.) So the file "EXCLUDE Hours of Surveillance.avi" will get ignored, but the file "Exclusive Video Footage.m4v" will get captured. Again, with the infinity symbol checked, it doesn't matter where in the Projects folder hierarchy these items appear, they all get excluded.

Let me know if either of these is a workable solution.

There was a security issue that corrupted the installation disk image QRecall 2.2.9.

Try downloading the latest release, QRecall 2.2.10.

Nico Dudek wrote:1 Hardware swap
2 Installing Catalina new from an earlier created USB Stick/External SSD (with Apple Catalina installer and Qrecall installed) onto the new 1TB new internal iMac disc (in my case it will be a SSD)
3 From the same USB Stick/External SSD I open Qrecall and open the Archive from the external 2TB HFS+ formatted drive and recover the files to the new replaced 1TB new internal iMac disc

I will also note that you can, if you like, flip steps 2 & 3. It's also permissible to format a brand new APFS volume, restore all of your files to it from the archive, and then install the operating system on top of that (this is actually the technique I use). The install will figure out that macOS has never been installed on this volume, split it, leaving all the files you previously restored on the "data" half of the system/data volume pair.

When I freshly install Catalina on my swapped iMac system HD it is blank without all my programs, that I had installed before. My question: are programs part of the Data so that they are in the found "data" half

Here's the only rule you need to remember: Everything that you, the system, or any other user, writes to your running system volumes resides on the "data" half of the system/data volume pair.

The whole point of having a "system" volume is that no process is allowed to write to it. Not even processes with super-user privileges. Not even the kernel. Once the macOS installer is finished, the "system" volume is frozen in amber until the next system update.

So if you installed something, it's on the "data" half of your startup volume pair. And it's that "data" volume that QRecall captures.

At run time, the operating system stitches the two together so it appears to be a single volume. This involves a lot of smoke and mirrors, and can seem pretty crazy at times?at least from the perspective of someone working at the filesystem level ... so almost no one.

As an example, in your Catalina /Applications folder you have many applications. Some were installed by Apple and some you installed yourself. If you use the low-level filesystem to ask what volume your copy of /Applications/Mail.app is on, it will tell you that package resides at /System/Applications/Mail.app. But if you ask it where /Applications/Photoshop.app resides, it will tell you that package is located at /System/Volumes/Data/Applications/Photoshop.app. So neither app is actually located at /Applications. Even stranger, you have a single folder (/Application) with items that reside on two different filesystems. Like I said, it's pretty crazy.

But the bottom line is that Mail is stored on the immutable "system" volume and cannot be modified/hacked, while Photoshop is on the "data" volume that you can alter and gets captured by QRecall.

Nico Dudek wrote:For this scenario
I need a backup strategy.

This is an excellent place to start.

Can I backup the FD completely on an external HFS+ disk?

With QRecall, yes!

QRecall chops up your files and stores that information in (essentially) a database. You're not actually storing the files or directories on the archive's volume, so the capabilities of the filesystem where the archive is written makes no difference. You can store APFS file on an HFS+ volume, HFS+ files on a Windows (FAT32) volume, ...

I guess not cause Catalina system requires an AFPS volume and AFPS on spinning HDs is endlessly slow.

Again, it doesn't matter. Also, APFS is much faster now. (In its early days, APFS was developed exclusively for SSDs and was not optimized for HDs, but that's in the past.) And going forwards, QRecall 3 will take advantage of APFS features for faster, and more reliable, archive updates.

So maybe one possibility is to backup only the data/document folder onto an external HFS+ disk and in case of a a drive failure
do a catalina system install from another SSD with APFS where I installed the system backup with all programs etc.

This is exactly what QRecall does now.

Starting in Catalina, your startup volume is actually two volumes. The system files are sequested on an immutable volume; a volume that cannot be modified, even by the kernel. The only[*] way to reinstall your operating system is to use the Apple macOS installer. When you capture your startup volume, QRecall find's its "data" volume (the secondary volume that you can write to) and captures everything there. The immutable system files are ignored.

To perform a whole-volume restore of a startup volume, you begin by formatting the volume and reinstalling the macOS (either via recovery mode or from an external startup drive -- I recommend creating a bootable USB drive for this purpose). Once installed, you can then use QRecall to restore the volume; QRecall will figure out that this is a startup volume, find the "data" half, and restore all of your files. Alternatively, you can then boot from the new system volume, reinstall QRecall, then use that to restore your files, but I prefer the former technique.

So here's my suggested preparation. Using either a handy external drive (which can also be the one your archive is stored on), or a bootable USB stick you keep around for emergencies:

Format the drive/stick APFS

Install a copy of Catalina

Reboot from the dirve/stick.

Download and install QRecall

Download and keep copy of the Catalina installer

When disaster strikes:

If replacing your drive, do the hardware swap.

Boot from the external drive/stick you created earlier.

Format your new startup drive and install macOS using Apple's installer.

Launch QRecall, open your archive, and restore the captured volume to the new one.

Reboot your new system and get back to work.

Does that make sense for you?

[*] Technically, you could restore the system files by booting from another volume and turning all of the safeties off, but I think that's a terrible idea and QRecall no longer supports this (without some hacking).