No silly questions...

Archives are automatically added to the status window whenever they are captured or verified.

The "Forget?" command is for archives that no longer exist, or you are not actively using, and you don't want to be constantly reminded of the fact that you haven't captured any new items to them in an overly long time. (There are also options to silence the capture and verify alarms, without forgetting the archive entirely.)

If you've accidentally forgotten an achieve you are still using, it will eventually reappear on its own.

Nico Dudek wrote:If I would restore the archive again AFTER doing all my hardware upgrade work again on the external Blade SSD would this be incremental
or would QR do it from scratch?

QRecall also performs incremental/smart restores.

When restoring a group of files, QRecall compares each existing file to the one in the archive. If they appear to be identical, it skips the file and moves on to the next.

And even if it decides to overwrite the file with the version in the archive, it actually starts by reading the file and comparing it, byte-by-byte, with the version in the archive (rather than blinding overwriting the same data). This is because reading is generally faster than writing and it avoids overwriting SSD sectors when it doesn't have to. (Flash memory blocks have a limited number of write cycles, so every write shortens its lifespan.)

So your second restore should be pretty quick and won't age you new SSD.

Moving forward...

You'll eventually want to switch to capturing your new SSD volume once your migration is finished. This might happen automatically once your SSD is your startup volume. If it doesn't, edit the capture item in the action.

The first post-migration capture will create a new volume in your archive.

Since this volume is essentially the same set of files that was on your original Fusion Drive, you'll want to Combine the two volumes so it appears in your archive as a single volume with a single history. (See Help > QRecall Help > Guide > Advanced > Combine Items)

Enjoy your new SSD!

Nico Dudek wrote:Does this mean I have to redo the restore on my blade SSD again?

No, the restore is fine.

What I was simply pointing out is that ownership and permissions are a requirement for startup volumes and home folders.

QRecall knows this and thoughtfully restored ownership and permissions before it started the restore.

So if you go check, you'll see that ownership and permissions has been set again on that volume. (Note that the macOS installer will also force ownership and permissions back on...)

I think I solved the problem: There was no full disk access checked for "com.qrecall.hoist"

Security will definitely make restores harder.

Just FYI, the "Ignore ownership and permission" option must be turned off when restoring items on a startup volume.

And if you look at your log, you'll see that QRecall set that option back when it started the restore:

2021-01-26 09:28:40.549 Setting "uses ownership and permission" on volume Catalina-1TB-BladeSSD

Sorry to hear you're having problems.

The log you sent would indicate that nothing was recalled, which is very strange. Particularly since you weren't booted from the Catalina-1TB-BladeSSD volume at the time. (It's not unusual to encounter permissions issues when recalling to the booted system volume, but it doesn't interfere with recalling the rest of your files.)

Here's my suggestion:

(1) Delete the Catalina-1TB-BladeSSD and Catalina-1TB-BladeSSD-Data volumes.
(2) Create and format an empty volume (make sure it's formatted APFS)
(3) Use the Restore Volume To... command to restore your Macintosh HD to the new, empty, volume.
(4) If that's successful, then run the Catalina installer and install macOS on top of the files you've restored.

If it's not successful (for the most part), then something else is very wrong. Please send a diagnostic report (choose Help > Send Report in the QRecall application). The report will include your recent log records as well as some information about your system and configuration.

Steven Haver wrote:Hello, I am trying to keep some large files out of my archive. My question is: is there a way that I can exclude files in a specific path (folder and sub folders) above a certain file size?

Unfortunately, the short answer is no. File exclusions (which are already pretty complex), at the end of the day, simply include or exclude files. There's no "exclusion math" that can be made to make discrete decisions for individual items.

Unfortunately, that is proving not to be an ideal solution either because when the files are not all together it increases the chances of me making a stupid human error and deleting the wrong file etc. So, ideally, I would be able to leave all the project files together in the same folder but exclude the large files from being captured to the archive. Any ideas? Thanks!

I have two suggestions.

My first suggestion would be to keep the arrangement you have now (leaving the huge files in a TEMP or DO NO BACKUP folder), and your working project files in individual project folders.

If one of those huge files really belongs to a project, create an alias or symbolic link in that project folder to the (correct) huge file in the TEMP folder. For the most part, it will be as if that huge file is actually in your project folder (you can see it, preview it, open it, drag it into applications, ...) but the actual file will be in the TEMP folder, where it will not get captured.

My second, and slightly-more-technical, suggestion is to use an exclusion pattern to exclude all folders below a particular folder level with a specific name (like DO NOT BACKUP).

In this scheme you'd keep all of your files (both large and small) inside a top-level Projects folder, organized into project subfolders, however you like. Then, wherever you need one, create a DO NOT BACKUP sub folder and move some very large files into that sub-folder. For example:

Projects/
    News Project/
        Exclusive Video Footage.mv4
        Graphics.psd
        Voice Over.mp3
        DO NOT BACKUP/
            Hours of Surveillance.avi
    Holiday Project/
        Jingle Bells.mp3
        DO NOT BACKUP/
            Unedited Santa Clause Interview.mkv
    ...

Now select your top-level Projects folder in the Finder. From the menu, choose Finder > Services > QRecall Capture Preferences...

Locate the Exclude Patterns in the capture preferences. Click the plus button and choose Add glob Pattern. Edit the pattern so it looks like this:

(empty) // DO NOT BACKUP [∞]

Leave the first field empty (no path), the pattern set to the exact name of the folders to exclude, and make sure the infinity symbol option is checked. This exclusion patten will, starting at the level of your Projects folder, exclude any item (file or folder) that is named, exactly, NO NOT BACKUP. It doesn't matter how many DO NOT BACKUP folders you create, or how deeply they're nested in other folders, they all get excluded.

Alternatively you could use a filename convention, rather then creating specially named folders. To do that, you'd use a glob pattern like this:

(empty) // EXCLUDE* [∞]

That glob pattern will match any file or folder that begins with the word EXCLUDE. (If you want this to be case sensitive, you'll have to use an expert pattern.) So the file "EXCLUDE Hours of Surveillance.avi" will get ignored, but the file "Exclusive Video Footage.m4v" will get captured. Again, with the infinity symbol checked, it doesn't matter where in the Projects folder hierarchy these items appear, they all get excluded.

Let me know if either of these is a workable solution.

There was a security issue that corrupted the installation disk image QRecall 2.2.9.

Try downloading the latest release, QRecall 2.2.10.

Nico Dudek wrote:1 Hardware swap
2 Installing Catalina new from an earlier created USB Stick/External SSD (with Apple Catalina installer and Qrecall installed) onto the new 1TB new internal iMac disc (in my case it will be a SSD)
3 From the same USB Stick/External SSD I open Qrecall and open the Archive from the external 2TB HFS+ formatted drive and recover the files to the new replaced 1TB new internal iMac disc

I will also note that you can, if you like, flip steps 2 & 3. It's also permissible to format a brand new APFS volume, restore all of your files to it from the archive, and then install the operating system on top of that (this is actually the technique I use). The install will figure out that macOS has never been installed on this volume, split it, leaving all the files you previously restored on the "data" half of the system/data volume pair.

When I freshly install Catalina on my swapped iMac system HD it is blank without all my programs, that I had installed before. My question: are programs part of the Data so that they are in the found "data" half

Here's the only rule you need to remember: Everything that you, the system, or any other user, writes to your running system volumes resides on the "data" half of the system/data volume pair.

The whole point of having a "system" volume is that no process is allowed to write to it. Not even processes with super-user privileges. Not even the kernel. Once the macOS installer is finished, the "system" volume is frozen in amber until the next system update.

So if you installed something, it's on the "data" half of your startup volume pair. And it's that "data" volume that QRecall captures.

At run time, the operating system stitches the two together so it appears to be a single volume. This involves a lot of smoke and mirrors, and can seem pretty crazy at times?at least from the perspective of someone working at the filesystem level ... so almost no one.

As an example, in your Catalina /Applications folder you have many applications. Some were installed by Apple and some you installed yourself. If you use the low-level filesystem to ask what volume your copy of /Applications/Mail.app is on, it will tell you that package resides at /System/Applications/Mail.app. But if you ask it where /Applications/Photoshop.app resides, it will tell you that package is located at /System/Volumes/Data/Applications/Photoshop.app. So neither app is actually located at /Applications. Even stranger, you have a single folder (/Application) with items that reside on two different filesystems. Like I said, it's pretty crazy.

But the bottom line is that Mail is stored on the immutable "system" volume and cannot be modified/hacked, while Photoshop is on the "data" volume that you can alter and gets captured by QRecall.

Nico Dudek wrote:For this scenario
I need a backup strategy.

This is an excellent place to start.

Can I backup the FD completely on an external HFS+ disk?

With QRecall, yes!

QRecall chops up your files and stores that information in (essentially) a database. You're not actually storing the files or directories on the archive's volume, so the capabilities of the filesystem where the archive is written makes no difference. You can store APFS file on an HFS+ volume, HFS+ files on a Windows (FAT32) volume, ...

I guess not cause Catalina system requires an AFPS volume and AFPS on spinning HDs is endlessly slow.

Again, it doesn't matter. Also, APFS is much faster now. (In its early days, APFS was developed exclusively for SSDs and was not optimized for HDs, but that's in the past.) And going forwards, QRecall 3 will take advantage of APFS features for faster, and more reliable, archive updates.

So maybe one possibility is to backup only the data/document folder onto an external HFS+ disk and in case of a a drive failure
do a catalina system install from another SSD with APFS where I installed the system backup with all programs etc.

This is exactly what QRecall does now.

Starting in Catalina, your startup volume is actually two volumes. The system files are sequested on an immutable volume; a volume that cannot be modified, even by the kernel. The only[*] way to reinstall your operating system is to use the Apple macOS installer. When you capture your startup volume, QRecall find's its "data" volume (the secondary volume that you can write to) and captures everything there. The immutable system files are ignored.

To perform a whole-volume restore of a startup volume, you begin by formatting the volume and reinstalling the macOS (either via recovery mode or from an external startup drive -- I recommend creating a bootable USB drive for this purpose). Once installed, you can then use QRecall to restore the volume; QRecall will figure out that this is a startup volume, find the "data" half, and restore all of your files. Alternatively, you can then boot from the new system volume, reinstall QRecall, then use that to restore your files, but I prefer the former technique.

So here's my suggested preparation. Using either a handy external drive (which can also be the one your archive is stored on), or a bootable USB stick you keep around for emergencies:

Format the drive/stick APFS

Install a copy of Catalina

Reboot from the dirve/stick.

Download and install QRecall

Download and keep copy of the Catalina installer

When disaster strikes:

If replacing your drive, do the hardware swap.

Boot from the external drive/stick you created earlier.

Format your new startup drive and install macOS using Apple's installer.

Launch QRecall, open your archive, and restore the captured volume to the new one.

Reboot your new system and get back to work.

Does that make sense for you?

[*] Technically, you could restore the system files by booting from another volume and turning all of the safeties off, but I think that's a terrible idea and QRecall no longer supports this (without some hacking).

That's a great idea ... which is why it's already in the works.

However, we can't "track the clone status of files", which is why I haven't done this already.

Basically what we want to do is implement the same logic for cloned file that we do for hard links. The ideal solution is to detect when a file is a cloned copy of another file being captured. And when those two files are restored, restore the first file and then clone it to restore the second file.

The problem is that (unlike hard links), the filesystem provides no information whatsoever about which files are clones of other files. And even if it did, cloned file can later be modified wherein it becomes a "partial" clone of that other file?with some blocks sharing storage with the original file and other blocks being independent of the original. So the two files could still be different and must be treated as unique files.

What we need to do is to build a "cloned file recognition" engine that can detect when a second file is a full or partial clone of another file. But to do that thoroughly requires a massive amount of memory and processing time, since we'd literally have to compare the data allocation of every file with every other file.

So the idea on the workbench right now is to (a) limit cloned file recognition to relatively large (multi-megabyte) files and (b) make it probabilistic, so it matches cloned files imperfectly, but is likely to recognize very large cloned files. This would, hopefully, be a reasonable trade off between capture speed and catching a few large cloned files. But I haven't proven it will work yet.

David,

So there was a problem. It wasn't, technically, specific to Big Sur, but Big Sur is what drove QRecall over the edge.

The layers.index file stores a compact summary of the directory structure for all of the archive's layers. There was an assumption that a folder wouldn't have more than few hundred subfolders. If a folder has more than 65,000 subfolders, the file would get corrupted, and that's what was happening.

The metadata daemon in Big Sur creates temporary folders inside /private/var/folders (the standard UNIX location for temporary and cached data). But unlike Catalina, the Big Sur version creates a lot of subfolders. And I mean, hundreds of thousands of subfolders?all in a single folder. (It's clearly taking advantage of APFS's hashed directory structure, but it was unexpected behavior.)

So if you happened to capture a volume while the metadata daemon was working particularly hard, there was a possibility that you'd capture a folder with too many subfolders.

The latest version of QRecall (2.2.9), expands the subfolder limit to 4 billion.

Update at your leisure.

Rose,

There are two parts to this. You first download a disk image file, which should appear wherever your downloaded files normally go, typically your Downloads folder.

If you can't open the disk image file, then I suggest trashing it and trying again.

Once you've opened the disk image file, you need to copy the QRecall app to your Applications folder. Optionally, you can open the Install QRecall utility application, which will do that for you.

Once in your Application folder, open the QRecall archive and it will install itself.

If you can't open either the QRecall application or the Install QRecall utility, you might have your GateKeeper security settings to restrictive. In your System Preferences, under Security, go to the General tab and make sure Allow apps downloaded from: is set to App Store and identified developers.

If you have anti-virus software installed, make sure it hasn't blocked or sequestered the app.

QRecall 2.2 has been tested under Big Sur and appears to be generally functional. We're performed hundreds of captures here at QRecall central.

There may be some odd issues with the layers.index file (see the earlier thread), which we're looking into. But for now it appears that repairing the archive will resolve the issue (and this issue does not involve any data loss).

As mentioned in the 2.2.8 release notes, there are a few minor cosmetic issues, most of which are being dealt with in QRecall 3.0 (which has not been released yet).

Update: Probably not a coincidence.

So when I said "other users" ... that was me.

I haven't run into this error on any of the hundreds of captures performed by my regular machines running Big Sur.

But ... I just ran into the same error while debugging some new QRecall 3.0 code. So there's probably a Big Sur related glitch lurking somewhere.

It might be something to do with Big Sur. Or it might just be a coincidence.

The next step is to see if it happens again, or to other users.