Hello,

I recently upgraded my QRecall (version 2) to version 3.

After first (new) backup (of my user-folder) I tested restoring some files.
And I found: my emails are missing in the backup.
I checked "old" backups from v2, and there I can see all my local mails.

I was using "standard" seetings of the new archive.

I added attached screenshop where both "invisible" and "excluded" files are shown.
Everything under my "V9" folder (which contains all my mailboxes) is excluded
Se attached screenshot.

I also uncheced some of the "standard" excludes (see attachement), so my exclude settings should not be the reason for the missing mails...

After first (new) backup (of my user-folder) I tested restoring some files.
And I found: my emails are missing in the backup.

That is correct.

In QRecall 3.0.7 and later, if you check either the "Exclude: Cache" or the "Exclude: Items Excluded by Time Machine" option, the contents of any ~/Library/Mail/V* folder will be excluded.

This is because most of the world has migrated to IMAP, and in IMAP the contents of your mail folder are simply cached copies of what's on the server(s). Additionally, recent changes to the Mail program have made these folders immensely complicated and time consuming to capture.

If you have POP or local mailboxes that must be captured, and you do not want to turn off both of the exclude options above (which I would not recommend), then the most straightforward solution is to create a second archive just for mail. Set it up so that it does not exclude "Caches" or "Time Machine" items, and then schedule a capture to capture just the specific POP or local mailboxes you need to preserve. (Again, avoiding any IMAP accounts/mailboxes because capturing these is largely a waste of time.)

Hello,

Thanks for the reply, this explains the behavior

However:
I think it is a very bad idea to "hide" emails under the "cache" entry, because it is not obvious that emails belong to "Cache"!

Also: I completely disagree on your "IMAP" statement:
As you wrote, "real" IMAP mailboxes act like a cache,
but: typically you have lots of local mailboxes (which are not "IMAP caches"), these need to be backed up reliably.
Providers do not allow unlimited mail storages, so local mailboxes and archives are essential.

In my case, I can use only 500MB online space for my IMAP mails, so I have to store nearly everything locally (and of course I need a reliable backup).

In case I would have not checked what QRECALL puts into the backup, I would have simple trusted my backups, and in emergency case I would have lost all my emails!
This risk should be minimized by any backup software!!

For me, this means: I have to stay with QRECALL 2, because I do not see a simple way to backup my mails (I do not want to include all cache files, and I want to have everything in one archive)

JoeMFox wrote:I think it is a very bad idea to "hide" emails under the "cache" entry, because it is not obvious that emails belong to "Cache"!

In general, I agree with you on all points. The current mail exclusion filter is basically a hack because I couldn't really come up with a "real" solution.

As I mentioned, you could simply create a second archive for just your mail.

I could also easily add an advance settings that would disable the new ad-hoc Mail filter, restoring QRecall 3 to the behavior of QRecall 2. If that would be of interest to you, let me know here or write to support.

What I'd really like to find is a straighforward way to mark all of the data storage for IMAP mailboxes as cache and capture everything else. Maybe a shell script? Just thinking out loud.

Hello,

why not just add another checkbox liek "ignore emails (use only for IMAP mails)" ?

And: as you mentioned, it would be good to know, what the "Cache" checbox is doing (so any user should be able to recreate the behavior by own setting).
In general, for every checkbox which affects more than just a simple file or folder, every user should be able to fully understand what this checkbox is doing.

Especially for backup functionality this is very important, as (in case the backup is required due to some problem), the user has to rely on the backup content!
So the documentation / manual should clearly describe it!
Missing files in a backup are a worst-case scenario!

Thanks a lot for all your support!

JoeMFox wrote:why not just add another checkbox liek "ignore emails (use only for IMAP mails)" ?

I have a lot of boring reasons why I don't want to do that (just yet).

And: as you mentioned, it would be good to know, what the "Cache" checbox is doing (so any user should be able to recreate the behavior by own setting).
In general, for every checkbox which affects more than just a simple file or folder, every user should be able to fully understand what this checkbox is doing.

This answer used to be simple, but as macOS has matured it's become progressively more complicated. Right now, the short answer is there are a set of items excluded when you turn on the "Exclude caches" option:

/private/var/folders
/System/Library/Caches
/Library/Caches
~/Library/Caches
~/Library/Containers/*/Data/tmp/*
~/Library/Containers/*/Data/Library/Caches/*
~/Library/Mail/V*/*
Where ~ is every user's home folder defined in the system.

So for now, I've implemented a new advanced setting (QRExcludeCasheExcludesMail) that turns that last filter rule (Mail/V*) on or off. You can download this pre-release version of QRecall to try it out.

To capture the Mail folder with "Exclude Caches" or "Exclude Items Excluded by Time Machine" turned on, do this:

1) Go to the QRecall > Settings > Advanced
2) Edit the "Excluding Cache or Time Machine excludes Mail" setting. Click the setting until it is un-checked.
3) Save the new setting.

The next capture that captures that folder should recapture everything. But there is some weirdness when changing the ad-hoc filters between capturers because QRecall relies on the filesystem change history to determine which folders needs to be examined, and the change history doesn't reflect that fact the the filters changed. So to be absolutely sure, do this:

4) Open the capture action that captures your home folder
5) Set the "Deep Scan" option
6) Save the action (File > Save)
7) Run the action

Once the action is running, you can turn the "Deep Scan" option off and save the action again. The "deep scan" run will cause the capture to ignore the filesystem change history and iterate through every folder in the captured item.

Hello,

basically the solution you are proposing is absolutely fine,

however: it is not working (or maybe I am doing something wrong):
the advanced setting "QRExcludeCasheExcludesMail" (as you described) does not exist; instead there is a setting "Excluding Cache or TM excludes Mail"; I guess, this is the correct setting.

I have set this "Excluding Cache or TM excludes Mail" to FALSE, and then I have created a "NEW" backup of my home-folder
in the backup, all emails are still excluded (interestingly the mail attachments are part of the backup).
I have done several tries:
- in the settings I have excluded caches
- I also have tried "Items exclued by TimeMachine" (in this case caches are grayed out)
- I have captured my home folder a send time
- I have used "direct capture"
- I have also tried a capture action (with and without "deep scan", but this should not make a difference for a new backup)
all the tests above give lots of combinations, but for every combination, my email mailboxes are still excluded (everything under "~/Library/Mail/V9/" is excluded)

in case I am doing something wrong, please let me know.
Thanks a lot!

JoeMFox wrote:the advanced setting "QRExcludeCasheExcludesMail" (as you described) does not exist; instead there is a setting "Excluding Cache or TM excludes Mail"; I guess, this is the correct setting.

The defaults setting key is QRExcludeCasheExcludesMail—that's the value that gets set in ~/Library/Preferences/com.qrecall.helper. The "Excluding Cache or TM excludes Mail" is its human friendly description in the advanced settings panel. When you open the value for editing it will show you the equivalent defaults command.

There is, however, a long standing bug in macOS wherein a defaults setting set by one app for a different app isn't immediately visible to the second app. The easiest way to correct this is to restart the system and check the setting again. If it's still set, try the capture again.

I have done several tries:
- I also have tried "Items exclued by TimeMachine" (in this case caches are grayed out)

Time Machine exclude cashes, so one implies the other.

The easiest way to test if the setting is working is to use the command line:

qrecall capture /Path/To/Archive --deep ~/Library/Mail

I tested it here and it's working, so my suggestion is to just restart and try again.

Hello,

Thanks a lot, I am really impressed by the service you are providing!
Yes, the restart solved it, so it is working as expected.

One side comment:
in the older version, when creating a new archive, the "Data redundancy" by default is set to "1:8" (which I think is a good choice); but the current version, by default, redundancy is switched off.
This is not an issue for me, as I have experience with ECC memories for nearly 20 years. And (based on my experience), for me redundancy is a key feature of QRecall!
But users having no experience with Error correction, they may leave it at the default....

BTW: when I find some time I might test the redundancy (by "destroying" some data using a hex-editor)

Again: Thanks a lot for your great support and also for this great product!

JoeMFox wrote:Thanks a lot, I am really impressed by the service you are providing!

Yes, the restart solved it, so it is working as expected.

That's a relief to hear!

in the older version, when creating a new archive, the "Data redundancy" by default is set to "1:8" (which I think is a good choice); but the current version, by default, redundancy is switched off.

I was encountering too many customers who had redundancy turned on for archives stored on RAID or cloud drives, which themselves provide their own error correction, which is—dare I say it?—redundant. So I made this particular feature "opt-in".

BTW: when I find some time I might test the redundancy (by "destroying" some data using a hex-editor)

Have fun! That's the way we test it here. (I even wrote a custom tool that scrambles random bits of a file so I could automate the tests.)