Excellent question.

And the answer is a, resounding, "absolutely!"

In fact, you can use almost any mix of filesystems.

Unlike backup solutions that simply copy files, QRecall stores the information about your captured files in a database. So it doesn't matter what the capabilities of the destination filesystem are, as long as it can hold an archive.

This makes QRecall unique in its ability to capture items from a filesystem that supports user permissions, hard links, extended attributes, access control lists, and compression, and faithfully preserve them in an archive on a volume that doesn't support any of those features.

David Ramsey wrote:I did try closing the "QRecall Monitor" application to prevent this from happening; however, closing it is impossible: that is, you can close it, but it just pops right back up.

In 2.1 the monitor app now appears in the dock and, more or less, acts like a regular app.

If you don't want it in the dock you can hide it (See QRecall > Preferences > Monitor, "Show in dock" setting).

However, if you hide the monitor in the dock you'll also disable the feature where QRecall will prevent your system from restarting or shutting down until all of your running QRecall actions have finished.

If you want to restore the pre-2.1 behavior, go to QRecall > Preferences > Monitor, turn off "Show in dock" and turn "Show in menu bar" back on.

It's possible the Monitor apps is just a monitor.

It is.

However, if it's involved in actually starting scheduled actions, it sure would be nice to close it and have it stay closed.

The QRecallScheduler process starts scheduled actions.

David Ramsey wrote:It seems to me that scheduled tasks should never interrupt a currently running task.

It won't. QRecall will only allow a single action to modify an archive at a time. And it doesn't matter how the action was started (manually, scheduled, from the command line, etc.)

A capture action will wait until all processes that are currently using the archive have closed it before it begins. Having the archive open in a browser window counts as one process (and is why you got that dialog). The second recall process counts as a second, and the capture action would have waited until both had finished before starting.

I can't think of anything for you to test, but I do have a question. Are you capturing the entire volume/folder, or specific items inside of the "Sound room" folder?

Johannes,

It looks like you're doing everything right, so I'm not sure why it's not working for you.

I just set up a similar test, capturing a folder on an external volume, excluding all of the *.dmg files, and it worked as advertised.

So here are my suggestions:

Make sure the path is correct

If you're typing in the path, it's easy to mess it up. Instead, select the path portion of the pattern, clear its contents, and then locate the folder in the Finder and drag that into the path field (the insertion cursor must be active for this to work). That will alway paste in the exact path to the folder.

Make sure the extension is correct

Select one of your .mp3 files, choose Get Info in the Finder, then look at the Name & Extension section. Make sure the extension is correct and it's really the extension of the file. (Some files have what look like extensions, but their extension is hidden in the Finder and the "extension" you see is actually part of its name.)

Verify where you are creating the pattern

You didn't mention whether you were setting this pattern in the Archive's settings or in the Capture Preferences of an item. When using the capture preferences, absolute paths are not allowed and if you create a pattern with an absolute path, it will be ignored. So you have the choice of adding the pattern to the archive's settings or using a relative path in a pattern attached to a folder:

Archive Settings: path=/Volumes/WorkingMedia/Sound room
Capture prefs for /Volumes/WorkingMedia/Sound room: path=(empty)
Capture prefs for /Volumes/WorkingMedia: path=Sound room

Finally, you also have the option of creating a .qrecall_exclude_patterns.txt file in the "Sound room" folder. It would consist of a single, simple, repeating, glob pattern:

//*.mp3/+

Let us know if any of that helps.

Paul,

The problem is Mojave. Even when you grant the helper app Full Disk Access, sometimes the operating system still denies access to protected items.

Note: first make sure you've granted the QRecallHelper app Full Disk Access. Between QRecall versions 2.1.4 and 2.1.5 the helper has changed from an executable file to an executable bundle. If you've granted the file Full Disk Access, remove it and replace it with the app. The Capture Assistant will walk you through the steps.

It seems to be a timing issue. If the helper immediately attempts to access a protected item, it will typically get blocked. But if it first reads a bunch of other files and then tries to access the same item, the request is granted.

Consequently, initial and infrequent captures tend to work just fine, while frequent ones will often fail. The reason is that when only a few items have changed since the last capture, QRecall jumps right to those items. And if one of them is protected, it access is denied.

As a workaround, I've found that setting the "Deep Scan" option of the capture action helps. It significantly increases the capture time (because QRecall checks every directory), but it's enough busy work to skirt around the bug and capture all of the files.

Naturally, I've brought this to Apple's attention and have had some short conversations about it. But at the moment, I'm waiting on Apple for a solution.

A 1.6TB archive is pretty big for being hosted on a NAS. The problem isn't so much the size of the archive, as much the number of records you've accumulated and the nature of networked volume transactions. It's also not a memory issue.

The early part of the repair (checking archive data) is highly optimized to read large chunks of the archive, sequentially. This can be very efficient even on a networked volume, and is usually constrained only by the network's speed.

Some of the later phases, like reindexing the layers, is highly transactional. QRecall is going back to each file and directory record and reconstructing the tree of items captured in each layer. It reads these (small) records one at a time.

Most filesystems are pretty smart about this, and the operating system will both cache data (so that reads of multiple, small, records come from RAM) and read ahead (so that it's likely the next read request can be satisfied from RAM). Networked filesystems tend to be transactional, which means that every tiny read request gets packaged up as a request, sent to the server, the server performs that request, and returns the results. Wash, rinse, repeat.

This means that the latency of requests for networked storage is much higher than for local storage. You can probably see this in the Activity Monitor. The total network throughput will be rather modest, but you'll see that the number of network requests per second is in the thousands.

You have a some 26,000,000 file and directory records to read and process. If you're processing only 1,000 records per second, that's at least 6 hours just to read them all. If your latency is higher, you might only be processing a few hundred a second, which could mean 18 or 36 hours. In contrast, a local drive would typically be able to read 2,000 to 3,000 records per second.

I'm working on future optimizations so that QRecall can group multiple record reads into a single request, which I hope will speed up networked volumes. But for now, it relies on the native filesystem drives for whatever optimization it can get.

In conclusion, if the repair is once-in-a-blue-moon occurrence, you might just live with the performance for now. Another option might be to split up the archive. I have a Mac mini that I capture the media (iTunes) folder to one archive and a second archive captures everything else (by capturing the startup volume but excluding the iTunes folder). These archives are stored on an Airport time capsule, and it keeps the size of each archive down to a manageable size.

Paul,

QRecall is, for the most part, compatible with Mojave. There are still issues with Full Disk Access, which we're currently working with Apple to resolve, but you've taken the correct step of adding the ~/Library/Application Support/QRecall/QRecallHelper program to the Full Disk Access group.

There are some minor bugs you might encounter. Mojave makes some subtle changes in how timers work and you might experience issues like the QRecall scheduler sometimes won't actually schedule anything for about 15 minutes after you first boot the system?oddities like that.

Mojave doesn't really change how the archives are accessed, so if you're getting "waiting for archive" status messages then maybe something else is going on. You might start by sending us a diagnostic report (Help > Send Report) and take a look at the troubleshooting topic in Help > QRecall Help > Guide > Troubleshooting > Problems > Can't Open Archive. (Note that opening an archive to modify it and opening an archive to read/browse it are different modes, and one can still work while the other is stuck.) Often just waiting 10 or 15 minutes, or a restart, will break these kinds of deadlocks.

The big outstanding issue is that Mojave's Full Disk Access doesn't act consistently. Even if you grant the QRecallHelper program Full Disk Access, some future captures might still be unable to read restricted items. Keep an eye out for an update that addresses this.

Jeff,

I agree that the assistant isn't that flexible, and is really only meant to cover very generalized solutions.

But you're on the right track: use the assistant the get a feel for what actions you need and how they are scheduled. After that, edit the daylights out of them.

The maintenance actions (merge, verify, ...) apply to everything that's in the archive, so you only need one of those. You'll probably want to fiddle with their settings, schedule, and conditions.

It's not uncommon to have multiple capture actions; either actions that capture different things or in response to different events. For example, I have three capture actions on a main development system: capture my home folder every 2 hours (ignoring changes to mail and iTunes), capture the entire volume once a day, capture important development directories 1 minute after they change.

Also note that you can attach a executable script to start or finish of any action. So if you need a special script to mount a volume (assuming it's not auto-mountable) you can write a script that will do that before the action runs. You can also have the action wait until a volume is mounted. There are lots of possibilities.

QRecall (or at least the current incarnation) only works with mountable volumes. So I you say "over the internet" I assume you're speaking of a remotely mounted volume on a server or NAS device.

Compression, and QRecall's unique data analysis, can significantly reduce the amount of data your backups consume. This, in turn, can significantly reduce the amount of data you have to transmit over a network connection. (I have one customer who says recalling from a compressed archive is faster than doing a straight copy of the same files from his networked volume.)

Trying to split your archives into different types probably won't help and will ultimately introduce more overhead (due to the need of updating multiple archive documents). QRecall is really smart about compression and de-duplication, and it works best when everything is in the same archive. The only major advantage would be to split very large archives into smaller ones so that maintenance is manageable (see last answer).

In modern (multi-core) computer systems, compression adds only a modest amount of CPU and memory overhead. It's measurable, but if you can benefit from the compression it's certainly worth turning on.

Merging over a slow network connection isn't going to be particularly slow. The verify, compact, and repair actions will, on the other hand, be slow due to the requirement of reading every byte in the archive. Either schedule these tasks to run infrequently (say, once a month), or (if possible) try to perform them from a system that has a fast(er) connection to the data.

If you're asking how that would look, that would be a merge action set up to create 14 day layers, and nothing else.

Attached is a screenshot.

Jeff,

You might want to start with the Capture Assistant and have it implement a capture strategy, which you can then customize.

You'll most likely want to customize your capture actions. You can capture every hour, if you like, or you can add additional capture actions. Examples: I capture my home folder every hour. I also have a capture action on my server that captures my Git repositories 2 minutes after any change is detected.

The rest of the actions (merge, compact, verify) are maintenance tasks that only need to be done once a week, often less. That's why I suggest you start with the capture assistant; it will create all of the regular maintenance actions with reasonable schedules, which you can then adjust to meet your needs.

Jeff Lambert wrote:Hi, I've had a couple of warning because I had a scheduled Backup and my mac wasn't running at the time. Now I want to get rid of those warning and not see a orange circle in my menu bar QRecall icon. Is it possible to reset the logs?

If you're talking about the warning indicators (yellow or red signal) in the archive status window, you can silence these warnings in that same window.

(1) Open the Archive Status window from either the QRecall Monitor dock icon or menu bar item.
(2) Expand the status pane of the archive with the warning
(3) Click the action (gear) button
shortcut for 2 & 3: control/right-click anywhere in the archive's status pane
(4) Choose one of the Silence options

You can have the monitor ignore a warning forever, or only until the action that would remedy the warning has had a chance to run again.

On another issus, It seems that I can't backup my Home Folder anymore. I upgraded to Mojave on Monday, and since then, I get an error message saying can't backup Home Folder, and the backup takes 0 sec. I've tried reinstalling QRecall, gave it full disk access, but still it's not working. I now have deleted that action that backup to my main Archive which is for the full disk.

You do need to grant QRecall Full Disk Access, but it's not the file you think it is.

Open your <home folder>/Library/Application Support/QRecall folder. Inside that you'll find an executable file named "QRecallHelper". This is the binary that actually performs the capture/recall, and is the one that needs full disk access. Drag that file into the Full Disk Access category of the security preferences panel and it should start working.

Note: We're working with Apple to resolve a number of issues still outstanding with Mojave and Full Disk Access, so this isn't a done deal just yet.

I thought this would be more stable than TimeMachine but I guess with TM I just didn't know when it was not working and now I know when QRecall isn't working

I occasionally get complains about how much QRecall complains when things aren't working perfectly. But if your data is important, you need to know this stuff!

Jeff,

Glad you found the Actions window. Each archive window also has an Actions sidebar (on the right) where you can see just the actions for that archive.

Jeff,

Excellent question.

The Stop at 8PM condition means that if the action is still running when 8PM comes around, the action is stopped, just as if you clicked the stop button at 8PM.

The schedule condition you probably want is "Ignore between...". Set up something like "Ignore between 8PM and 9AM", and the interval schedule will run every hour starting at 9AM through 7PM.