Ralph Strauch wrote:What's my best course forward at this point? I assume that the archive is as big as it is because it now contains both unencrypted and encrypted data.

Ralph,

Thanks for sending a diagnostic report. From the logs, I can see that your encryption action is failing because the macOS encryption services are crashing. Some systems?and I don't yet have a sense for which ones?have a great deal of difficulty executing multiple encryption/description operations simultaneously. Sometimes the operations fail (which QRecall can deal with), but for some users the operations crash.

So what's happening is that your encryption action is crashing. For safety, Qrecall first makes an encrypted copy of the archive data. So if there is a problem, you won't be left with a half-encrypted archive. That intermediate copy is what's using up your disk space.

First, let's fix the disk space problem

Control/Right+click the archive in the Finder and choose Show Package Contents

Trash any files that begin with the name repository_scribble

Close the window and empty the trash

Now you should be able to start a new encryption task. But to keep that from crashing again, you'll want to limit the number of concurrent encryption operations QRecall will perform.

Make sure you've upgraded to QRecall 2.0.7

Go to QRecall > Preferences > Advanced

Find the Concurrent Encryption Limit setting and set it to 1 or 2

Now try the encrypt command again.

You can try adjusting the concurrent limit to a higher value, just realize that if it's too high some QRecall actions will randomly crash. One of the best ways to test a higher setting is using a verify command; verify performs a lot of concurrent encryption operations and there won't be any data loss if it crashes.

Alexandre,

I suspect something else is going on.

QRecall can certainly be configured to "just work," but in QRecall that's a choice.

It doesn't make sense to me that you can keep 6 months of Time Machine data on a volume, but QRecall runs out of space in a couple of weeks. QRecall is much more space efficient than Time Machine. You should be able to keep anywhere from 2 to 4 times as much history using QRecall.

For example, my main development system has a 960GB SSD. I capture my home directory every 3 hours and the entire volume daily. There's a rolling merge that keeps every capture for the past three days, consolidates daily captures for 30 days, then consolidates those into weekly layers for 26 weeks, and then finally merges those into monthly layers going back 18 months.

This means I have captured data, at various levels of granularity, stretching back over 2 years. The archive size is 1.2TB (60% of a 2TB volume). I have never, once, had to "manually" manage anything about this archive.

I'd be happy to help you find out what's going on. There are a number of possibilities. QRecall might be capturing a lot more data than Time Machine backs up, something else is using the space on your archive volume, or you need a different set of merge and compact actions to automatically manage its size.

Taking the last one first, I would recommend deleting all of the actions you have for this archive and use the Capture Assistant (in the Help menu) to create a fresh set of actions. For you, I suggest choosing the "Keep 5 days, then less frequently for 1 month" and "Yes, discard the oldest items to make more room" options in the assistant. When the assistant is done, I invite you to review the rolling merge action and adjust the time scale/granularity to your liking. You might also want to create an additional capture action that captures your home folder every few hours during the day. This combination will most closely emulate Time Machine's default schedule.

One of the important differences between QRecall and Time Machine is that when you run out of disk space, QRecall stops and lets you know. Time Machine just starts deleting things, without warning, until it has enough room.

Mike M wrote:Or ask them to pretend they are teaching it to me.

There's nothing that focuses one's learning like having to teach it.

Let me know if you need to teach me anything else.

Mike,

I can appreciate that some of these concepts give you headaches. Filesystems are hard.

I'll address some of you specific questions, but for the most part you can think of layers as "checkpoints" or "snapshots" or whatever concept you find easiest to deal with.

In the case where you have a single capture action, so that you capture the exact same set of files each time, and nothing else, most of these concepts can be equated. Specifically:

Here is how that is different from a layer. Your documentations describes layers as containing (1) files, (2) deltas from previous layers, and/or (3) some indication that a file has been deleted.

Start by ignoring the implementation details. How QRecall represents files and folders in a layer is really immaterial. Conceptually, each layer contains a complete copy of every item captured. This is your "checkpoint."

(In reality, a layer doesn't contain anything but references to unique database records, which contain the data and metadata of those items; the reason it's done that way is because it allows QRecall to store all of this data in the minimal amount of space possible.)

Your layers also have numbers rather than time tags--at least that's how you always describe it in the documentation.

Layer numbers are simply convenient labels that make referring to them in the interface, on the command line, or in actions, simple and easy to understand. Every layer has a date, the date it was captured. This date appear in the layer pane of the archive browser.

It's also useful to know that each item has a capture date (which you can see in the inspector panel). This is the exact moment in time that specific item was captured.

So what about the idea of checkpoints? Well, let's say there's a checkpoint at time T. I can then know, simply, that enough information is in the archive to recreate the state of all captured items at time T.

That's a layer. If an item exists in a layer, then that item can be recalled by rewinding the archive back to that layer.

This is easily visualized using the browser timelines. Select a captured item in the browser, and QRecall will draw its timeline back through the layers where that item was captured, recaptured, or simply existed in. Conceptually, if a timeline intersects a layer, that item "exists" in that layer.

I don't have to concern myself with how that data is represented. It might be deltas, it might need to refer to prior layers, it might look at other markers in the layer... but I don't need to know any of that stuff.

Now you're getting the idea.

Now let's consider the concept of "merging layers." This gives me a headache. I think it's actually a quite involved algorithm, and your examples have lots of parts in their diagrams.

What if, instead, we say that QRecall "deletes checkpoints"? No longer is there the concept of "merging." Instead, I can think about a "checkpoint deletion" as removing information from the archive ... and it's easy to think about that. The information lost, simply, is the state of the file system at the deleted checkpoints.

I don't have to care whether this is done via merging layers or any other algorithm. I also don't get a headache trying to think about the behavior of the system.

Again, if you limit the example to a single, uniform, capture action that captures the same set of files every time, then these concepts are equivalent. And if that makes you're life easier, then use that.

Most of the rest of your description is accurate. Basically, if an item exists in a layer then you can recall that item at some future date. Rolling merges eliminate intermediate layers/checkpoints so that only the last captured version in any particular timespan is retained. Whether you imagine layers being merged or checkpoints being deleted, the results are the same.

Now let me stop here and ask, is this actually an accurate way of thinking about it?

It is, as long as your layers remain simple. But layers can get complicated.

Consider capturing your whole startup volume at 3:00, then repeatedly capturing just your home folder every hour during the day. At the end of the day you merge all of those layers together. What do you have?

You have an interesting mixture of items captured at 3:00 (your applications) and newer items captured as late as 23:00 (in your Documents folder). That's because merging is just that; you can't think of it as "deleting" all of the earlier layers, because there's data in the very first layer that's not superseded by subsequent layers.

Now take another example of two volumes, or even two separate computer systems. Volume "A" is captured to layer 1. Later, volume "B" gets captured to layer 2. You then merge those two layers. What information was deleted?

The answer is nothing. The new layer contains a complete copy of both volumes "A" and "B" because none of the items in those layer sets intersect.

Sorry if that makes you're head hurt.

Alexandre,

Scheduled actions are the way you "set and forget" in QRecall. The actions described in the earlier post will manage your archive's overall size automatically, every day. Add a routine verify action and maybe an automated repair action, and you're good to go.

If you want help setting up a comprehensive set of actions, use the QRecall Assistant (under the Help menu). It will ask you a series of questions and create a set of actions that implement your answers. You can then review the actions the assistant created, edit them to refine your solution, or throw them away and start over.

The notable difference is that Time Machine has one management algorithm which you have no control over. QRecall gives you broad discretion on how you manage your archive. And that management can be completely automated, manual, or some combination you choose.

Alexandre Takacs wrote:Is it possible to set a fixed limit for the archive size - say 1TB - and let QRecall manage it so that it doesn't grow past that ?

No such option currently exists, for a variety of reasons, which I'll touch on in a moment.

If you really want do have a "hard" limit on how much your archive grows I suggest placing the archive in its own drive partition of the desired size. QRecall detects when the disk is nearly full and will abort captures and other actions that would fail if they ran out of disk space.

If you do this, you'll probably want to adopt the actions created by the capture assistant when you choose "keep items as long as possible" and "use all available space":

Merge that first three (0 through 2) layers whenever the free space on the volume is less than 10%, run daily.

Compact the archive whenever the free space on the volume is less than 10%, run daily, after the merge.

These two actions mimic Time Machine's logic. They monitor the free space on the volume and start discarding the oldest items in the archive whenever the volume starts to get too full. The rest of the time, they do nothing.

A requested feature, which is on the wish list for the next major version of QRecall, is to add a new "archive size more than X" schedule condition. That would allows actions to be scheduled based directly on the archive's size, rather than indirectly based on the free space available on its volume. If a feature like that would satisfy your needs, let me know.

It would be technically very easy to add an option that strictly limits the size of an archive. But doing so just sets up QRecall for failure (rather than success). The logic that prevents the archive from getting too big when the disk space is low works by aborting capture actions, leaving items un-captured. In this situation, QRecall is choosing to protect the integrity of the archive over capturing all of the new items. But this is still a capture failure.

If I implemented a user-configurable archive size, it just creates a situation where QRecall arbitratily stops working, and that seems like a bad idea.

Mike,

There are two reasons an item will exist in an archive: it was captured in a layer or it has been preserved as a deleted item.

tl;dr: captured items in older layers are preserved until they are merged with more recent layers. The "Keep deleted items..." option will ensure that deleted items are kept in the archive for a minimum period of time.

Here's the long-winded explanation...

Items in layers

The first should be obvious. Each capture creates a "layer" in the archive that represent a snapshot of all of the items at that moment in time. Let's take this example:

Create a document on Monday. Preform a capture

Delete the document on Tuesday. Perform a capture.

Your archive now has two layers. The first layer contains the document, the second layer does not. As long as both layers exist, you can always rewind the archive back to the first layer (using the layer shade) and recall the document as it was captured on Monday.

The document in the first layer will exist in the archive until that layer is merged with a subsequent layer. Merging the two layers combines them and keeps only the most recent state. Since the document didn't exist in the second layer, the document will not exist in the merged layer.

To answer your first question, merging layers is the natural way in which old documents, or old versions of documents, get deleted from the archive. As layers are merged, older items are discarded, and their space in the archive reclaimed. You can automate this using a merge action. How often you merge layers, and the granularity of those merged layers, determined how long captured items will remain in the archive.

Keep Deleted Items

Merging layers can sometimes discard items in a seemingly haphazard fashion, particularly for short-lived items. For example, let's say that you capture every day, periodically merge all of the layers captured during the week into a single layer (using a rolling merge action), and you keep the past 52 such layers in your archive.

A document that was created on Friday and deleted the following Monday will persist in the archive for at least a year. (The document existed at the end of the week, so it will exist in the merged layers for that week, and you keep 52 such layers, so the document will be kept for at least 52 week.) On the other hand, a document created on Monday and deleted on Friday won't exist in the archive for more than a couple of weeks. (When the layers for that week get merged, the document is purged because it didn't exist at the end of the week.)

To combat this capricious behavior, the archive has a "Keep deleted items for at least:" setting (see Archive > Settings....). If you set this period then a merge action will preserve the last version of a deleted item that would normally be discarded, if that item existed within the period you set.

Going back to the Monday-Friday example, if your "Keep" setting was set to 2 months, the file created on Monday and deleted on Friday would be preserved when the layers for that week were merged. The file exists as a special "deleted" item in the layer. Use the View > Show Deleted Items command to see and recall deleted items.

Deleted items are purged naturally as part of a subsequent merge or by a compact action. The compact action seeks out deleted items that are now past their "keep" date (two months, in the previous example) and removes them, without merging any layers.

To sort-of answer your second question, use the "Keep deleted items..." setting to enforce a grace period in which you can still recall the last captured version of a deleted item.

Other Reasons

The special "Delete Item(s)" command will delete any arbitrary item (file, folder, volume) from all layers of an archive, as if that item had never existed. This is mostly for cases where you have a very large file occupying space in the archive and you have no interest in keeping it, but don't want to sacrifice any other items or layers.

For completeness, I should mention that all of the file deletion logic only applies to captured items. If you create a file, immediately delete it, and then perform a capture, the file will never be captured and its fate is outside QRecall's purview.

Jeffery,

Send a diagnostic report (QRecall > Help > Send Report...) and we'll investigate further.

Jeffrey Fort wrote:I have an "encryption key password" and a "recovery key."

An "encryption key" is the cryptographic key used to encrypt, and decrypt, the data in your archive. It is stored in a "key file" in your home directory.

An "encryption key password" is a way of protecting that key file from unwanted agents by encrypting the file with a password.

If you've encrypted your key file with a password, QRecall will need you to supply that password every time it opens the archive. You can enter it manually when browsing the archive. For actions to run automatically, it will require that you store the password on your keychain.

When I try to restore I get a notice that I need a password.

That's a tough one. If you get this dialog when you open the archive, it's probably asking for the encryption key password (see above). Or it might be asking for your recovery key passphrase (see below). But if it's telling you that it needs to perform privileged operations, then it's asking for your administration account password. To avoid that in the future, go to QRecall > Preferences > Authorization and pre-authorize QRecall to use administrative privileges.

A "recovery key" is a backup of your key file stored in the archive itself, and protected with a passphrase. This is independent of your encryption key password (if any). It's basically a protected backup of your encryption key file and is only needed if you've lost your key file. (Without your encryption key file, your archive is unusable.)

For example, if you lose your startup volume and need to restore from scratch, you would start by installing a fresh copy of macOS. But that fresh copy of macOS doesn't have your encryption key file, so QRecall can't open up your archive and restore your hard drive.

That's where the "recovery key" comes into play. When you open the archive, QRecall will prompt you for the recovery key passphrase. Enter it, and it will restore the encryption key file from the secure backup copy stored in the archive. Once the encryption key file has been recovered, QRecall can then open the archive and retrieve your files.

For a explanation of how all of this works, see QRecall > QRecall Help > Guide > Advanced > Encryption. The section "Do not lose your encryption key!" is highly recommended reading.

On the face of it, that should have worked (assuming that Drobo02 was mounted and writable). The only thing that comes to mind is there might be something odd about duplicating/binding file descriptors on the Drobo02 when running as root, in single user mode, or from the recovery partition.

It's a bit late now, but you could try:

.../qrecall restore osx.temp ... --logfile stderr ... 2>>/Volumes/Drobo02/log/qrecall.logfile

That would let the shell create the file descriptor and then pass that the helper, rather than having the helper create its own.

Jeffrey Fort wrote:Just purchased. First Capture is underway.

Congratulations!

How do I make sure the Capture, to a new external HD, is encrypted?

Short answer: You encrypt an archive by installing an encryption key. Open the archive and choose Archive > Encryption > Install Encryption Key.

Long answer: See Help > QRecall Help > Guide > Advanced > Encryption.

Also, do I schedule period Captures or does in just happen automatically?

You'll want to schedule captures, merges, and other periodic maintenance. This is done by creating actions.

The easiest way to get started is to use the Capture Assistent. Choose Help > Capture Assistent and tell it that you want to create a backup strategy. Choose the archive you've already created and answer the rest of the questions.

The assistant will create a set of actions to automate QRecall. Review these actions to understand what's been set up. Later you can edit them to refine them, delete them if you don't like them, or create new ones that better fit your needs. If you really don't like the actions that were created, just delete them all and run the assistent again.

Norbert Karls wrote:I'm aware that after booting into internet recovery using ??R I can use a shell to dissolve the fusion drive, format my SSD as HFS+, mount my backup box and use QRecall's command line utility conveniently waiting there with

qrecall restore osx.longterm.quanta ':OSX' --tovolume /Volumes/SSD

to pull everything from the archive to the newly created volume without having to install a temporary OS first.

While that's technically true, it's a lot of work. If you have an external, bootable, drive it's much easier to install macOS & QRecall there first, then use that to reformat, reconfigure, partition, install, and restore your new internal drive(s). I keep a 16Gb thumb drive around just for that purpose.

You may have purposefully glossed over some of the command-line details details, but if you didn't be aware that running the qrecall tool from a recovery partition is a little more complicated than that. You can refer to the Help > Guide > Troubleshooting > Restore OS X section for all of the gory details. In fact, I'd suggest having this page loaded on a tablet or smart phone before you begin.

But how would I go about restoring everything except that one big folder? Is there a hidden switch or option to achieve this? The man page doesn't say so, but I hope that's just because it tries not to drown people in text.

Sorry, there is no such option. Your original idea of first duplicating the archive an then deleting the one big document folder will work. If you still had your original drive, I would have suggested taking a final capture, deleting the one massive folder, then taking one more capture. You could have then restored your SSD from the last layer, then rolled back one layer to recall your documents folder to some other location.

And just for there record, the whole purpose of a man page is to drown users in text.

That seems an awful lot of i/o for a fairly intuitive transaction request, though.

You say "intuitive," but I say "this is there first time anyone has asked for this."

I'll add it to the wish list. It might not be that difficult to implement. There's already a to-do item to add an exclusion mechanism to the restore command. This is to solve an unusual problem where you've captured the volume that contains the archive itself. During the restore, QRecall will attempt to delete the archive (since the archive always excludes itself from any capture). It's actually the opposite problem (not deleting an item that wasn't captured), but the mechanics of not recalling an item that was captures would be, more or less, the same.

Mike,

You want to "shade" a layer or two. What you're seeing in the browser is the composite results?the most recent version?of all of the files captured in those four layers.

To play time machine, you shade the most recent layers. The browser then presents the view of your files before those layers were captured, allowing you to recall earlier versions of those items.

To move the bottom shade, the one used to hide the most recent layers, either click and drag the bottom shade or hold down the Control key and press the Up (or Down) Arrow.

See QRecall Help > Guide Layers > Layer Shades (for how to manipulate the layer shades) and Understanding Layers for an overview of the layers concept.

Let us know if you have more questions.

Mike,

Yes, that's one way of doing it. There are three popular methods for handing a source volume that isn't always connected:

Add a Hold if No Capture Items condition (the one you suggested)

This condition will allow an action to start at a scheduled time, but then hold the action until the source volume is connected. Once connected, it will start immediately.

Add an Ignore if No Capture Items condition

Similar to the hold version, this condition just skips the action if the source volume isn't mounted. When the scheduled time arrives, the volume will be captured if it's connected, and ignored if not.

Use a Capture Items Volume Connects schedule

This event schedule starts the action whenever the source volume connects. Use this schedule for volumes that are connected very infrequently or unpredicatably. The ignore and repeat property can control how frequently this action runs.

See QRecall > Help > Guide > Automation > Schedules for more details.

And almost before this page reloads, Sierra is here.

QRecall 2.0.4 has been released. Update at will.

And with it the "macOS" era returns?for those of you old enough to remember the first MacOS era.