Mike M wrote:I have a 2TB WD Passport drive. This is only a few months old. I recently purchased DriveDx in order to read the SMART data, which shows one condition of concern, which is that there are 3 bad sectors waiting to be remapped. DriveDx pops up a little message saying that a drive with even one bad section *waiting* to be remapped (I think the "waiting" concept if part of the issue) is 16 times more likely to fail within a year.

I can't speak to the "16 times more likely" prophecy; that seems high to me (because, as others have pointed out, all high density drives have bad sectors), but I'll assume DriveDx has the stats to back that up.

Regardless, no drive is perfect?which is why we make backups in the first place.

QRecall 2.0 has a data redundancy feature. When creating the archive, or from the archive settings, choose the level of data redundancy you want QRecall to add to the archive. The best settings (from a performance standpoint) are 1:16, 1:8, and 1:4. In your situation I'd recommend 1:16, or 1:8 if you're really cautious. At the 1:8 level, QRecall will write one redundancy byte for every 8 bytes of archive data, increasing the size of your archive by about 12.5%. The upside is that QRecall can then recover from the loss of one or two blocks of data within any span of 16 consecutive blocks. So QRecall can still read your archive, even if you lost 2 or 3 sectors on the drive.

Be aware that data redundancy can't protect against wholesale loss of the entire drive, or even several whole tracks. But even in those dire circumstances, its recovery feature can extract all readable data to a second hard drive.

This 2 TB drive would be handy for holding two QRecall archives (one for the whole drive, one for my home directory),

There are really very few advantages to doing this. You'll use a lot more disk space and cause QRecall to work twice as hard.

The most efficient arrangement is to store your entire startup volume (which will include your home folder) in one archive. If you want to capture your home or Documents folder to your archive more frequently than your system and applications, create two capture actions with different schedules, both capturing to the same archive.

Think of archives like hard drive partitions: it's almost always more efficient and simpler to manage one big partition than a bunch of small ones.

Mike M wrote:I really liked QRecall during my trial period and I purchased a key.

Welcome aboard!

Thus far I have just been backing up my user directory, so I'm not backing up any apps or the OS or anything else. I know that the term "cloning" applies to making a copy of entire drives. But I am not sure that is what I want.

QRecall works quit differently than almost any other backup solution you'll fine. The term "cloning" does apply at all.

QRecall always capture the changes?and just the changes?made to your files. Conceptually, it preserves every version of every item it captures. Its de-duplication logic means that those copies are stored in the least amount of space possible.

What I would like to do is back up all my files on a regular schedule. Sometimes I make use of backups to look at old versions in the case that I accidentally delete or modify a file. The other situation is a total failure of my main hard drive, in which case I would like to set up the entire drive or partition just the way it used to be. Is it possible to handle both of these use cases in a single type of backup operation?

QRecall can, absolutely, handle all of these cases. To capture your operating system, create a capture action that captures your entire startup volume. You can have just one big capture action, or split it up into multiple actions (say one for your home folder and another for your entire volume), so they can be scheduled to run at different times.

If you only have capture actions, QRecall will preserve every version of every file you tell it to capture. But unless you have an unlimited amount of disk space on your archive volume, you'll eventually want to discard some of those older copies to make room for new ones. That's where the merge and compact actions come into play.

These are all explained in the documentation, but the easiest way to get started is to use the Capture Assistant (under Help). Tell it you want to create a backup strategy and then answer the questions. When it's done, take a look at the actions the assistant creates (in the Actions window) and then begin tweaking and modifying those actions to get what you want. If you want to start over, just delete all of your actions and run the capture assistant again.

As for restoring your entire system in case of a catastrophe, there are several sections of the help dedicated to this very subject. Short version: do yourself a big favor and create a minimal startup system (on an external hard disk?the one with your QRecall archive is perfect for this?or a USB drive) ahead of time. When disaster strikes, all you have to do is boot from your external volume, restore what you need, and be back in business within minutes.

If you have any further questions post them here or write to support@qrecall.com.

Gary K. Griffey wrote:So, I take it that QRecall cannot spread its work out over multiple CPU cores?

QRecall is heavily multi-threaded. You'll see that some actions, like verify, can keep 6 or more CPU cores running at 100%.

But the de-duplciation process is, inherently, a sequential process. Consider just two blocks. If you check to see if they are both unique simultaneously, they can both be considered unique (no similar blocks found in the archive). But those blocks could be duplicates of each other. Furthermore, if both blocks are unique, they have to be added to the cache before checking the next block. But two processes can't modify a structure like a cache simultaneously, and you can't use the cache until the add operation is complete. So while there are a few operations that could be performed in parallel, there are so many intermediate steps that have to occur sequentially, that it's actually a waste of time trying to turn it into a multi-tasking problem.

Now if you happen to have encryption, compression, or data redundancy turned on, those processes are independent of the de-duplication logic and are all performed on separate threads.

Gary,

Your I/O isn't maxed out, but I bet the CPU for the QRecallHelper process is running at 100% (or more).

If you want performance, you have to avoid physical I/O as much as possible. I/O is slow. Memory and the CPU are fast.

So QRecall uses massive in-memory maps, tables, indexes, and caches to first determine if a data block is unique or not, before any reading or writing of the archive begins. In fact, most (around 80%) of the memory used by QRecall during a capture are these lookup tables and caches.

When your archive is relatively small, QRecall can determine if a block is unique 99.9% of the time, without having to read any data directly from the archive. This process is typically 200 to 300 times faster than the time it takes to write a single new block to the archive, which is why shifted-quanta detection can test 30,000 variations of a single block so quickly.

Gary K. Griffey wrote:In reading the details concerning the benefits/drawbacks of using shifted quanta detection in your help file, it would seem to me that during the initial capture of a file to a newly created archive, shifted quanta would not be relevant at all. It would seem, at least from my understanding, that shifted quanta detection would only be relevant to the conversation during subsequent captures of the same file.

It's true that it would be "most relevant" when applied to a previously captured file, but the mechanics of shift-quanta detection are applied to every block of data no matter what files have been captured. Specifically, data de-duplication is performed at the data block level; it has no knowledge of files. It simply compares each newly captured block of data against the corpus of previously captured blocks.

This, however, does not appear to be the case. I created a new archive...and began to capture a single large virtual disk file that is being housed on a network share...this file is roughly 150 GB. At first, I set the shifted quanta in the archive to its maximum setting. After allowing the initial capture to run for nearly 60 hours...it was only 50% complete.

60 hours? Wow, you are really committed to this experiment.

Here, in a nutshell, is how QRecall performs data de-duplication.

First, let's consider how de-duplication work when shifted-quanta detection is turned off:

Files are divided into 32K blocks. When you capture a file, the first 32K of data (the bytes at offset 0 through 32,767) are read. QRecall then determines if there's an identical block of data already in the archive. If there is, then this is a duplicate block and it does not need to be added. If not, the new block is unique and is added to the archive.

The next 32K of data (the bytes at offset 32,768 through 65,535) are read and the process repeats.

The work to capture data increases exponentially as the size of the archive increases, but through various data tricks and optimizations, it goes pretty fast because each 32K block of data only needs a single up/down vote on whether it's unique or not.

Now let's look at what happens when you turn shifted-quanta detection all the way to the maximum.

The process starts out the same, the first 32K (bytes 0 through 32,767) are read and checked for uniqueness. If the block is a duplicate, then it is not added to the archive, and QRecall skips immediately to the next block.

If the block is unique, however, the process changes. When shifted-quanta detection is on, QRecall then shifts 1 byte and considers the 32K block of data at offset 1 (bytes 1 through 32,76. If that block also turns out to be unique, it shifts one more byte and sees if the block formed by bytes 2 through 32,769 is unique, and so on:

Are bytes 0 through 32,767 contained in the archive?
No?
Are bytes 1 through 32,768 contained in the archive?
No?
Are bytes 2 through 32,769 contained in the archive?
No?
Are bytes 3 through 32,770 contained in the archive?
No?
(repeat 32,764 more times)

If any of those blocks turn out to a be a duplicate, then QRecall found a "shifted duplicate". The fragment before the shifted duplicate is captured as a separate block and the whole thing repeats, starting with the next block immediately after the shifted duplicate block.

However, it's also likely that all 32,768 tests will be false. If so, then QRecall knows that the block is unique and it doesn't overlap any shifted duplicate block. The block is added to the archive and the process starts over with the next block in the file.

While shifted-quanta detection is pretty ruthless about finding duplicate data, you can also see that if the data is all unique the process takes 30,000 times longer[1] to come to that conclusion. That's why a 1 hour capture with no shifted-quanta detection takes 120 times longer when you max it out.

That explains "no" shifted-quanta detection and "full" shifted-quanta detection. The settings in-between simply toggle between the two modes based on how effective they are.

When shifted-quanta detection is set to "low" or "medium," the capture starts out performing full shifted-quanta detection. But after a few dozen blocks, if this work doesn't find any shifted duplicates, shifted-quanta detection is temporary turned off and QRecall switches to straight full-block de-duplication. This is controlled by a CPU budget. If shifted-quanta detection doesn't find any shifted duplicate blocks, the budget runs out and it stops performing shifted-quanta detection. If shifted-quanta detection does find shifted duplicates, the budget is increased, so it spends more time searching for duplicates.

The basic idea is that shifted-quanta detection is most productive when there are large amounts of shifted, duplicate, data. So QRecall doesn't have to test every single block to discover one. If shifted-quanta detection is producing results, QRecall does it more. If it's turning out to be a huge waste of effort, it does it less.

Let me know if that adequately explains things.


[1] Full shifted-quanta detection doesn't, literally, take 30,000 times longer than de-duplicating a single block because of a few mathematical tricks and other optimizations. For example, QRecall uses so-called "rolling" checksums to create a "fingerprint" of the data in a block. The work required to calculate the first checksum, for bytes 0 through 32,767, requires about 64,000 shift and add operations. But the checksum for bytes 1 through 32,768 can then be calculated with just three more operations?essentially "removing" the value of the first byte and "adding" in the value of the next byte. Thus, the checksums for all of those intermediate blocks of data can be calculated very quickly, the total work not being that much more than calculating the checksum of a second block. Of course, that's just the checksum; each block still has to be tested against those blocks already captured, but it does save a lot of time.

Mike,

Thanks for checking out QRecall.

You can pause a running action, but that simply suspends the process. The archive files, and the volume that it's on, will remain open and connected.

recommended method

If a capture has started and you need to disconnect the archive volume or shutdown/restart, simply stop the capture by clicking the (X) button in the monitor. Unless your archive is huge, the capture will wind up and close the archive within a few seconds. (Or just shutdown/restart. The normal OS X shutdown will send a stop signal to the capture action, it's the same thing.)

All captures are incremental; the next capture will begin where the last one left off.

This will result in an "incomplete" layer in your archive. This is just a caution that the capture which created that layer didn't get a chance to finish, so when recalling from that layer be aware that not all items were captured. The next capture will fill in the blanks. (If you want to be extra neat, merge the incomplete layer with the one that follows to create a complete layer?as if the incomplete capture had never happened.)

Most long-running actions, notably capture and compact, are incremental. It's always safe to stop them when you need to and start them again later. There's even a shortcut in the monitor: right/control+click, or click-and-hold, the (X) stop button in the action's progress pane. The Stop and Reschedule command will stop the running action and immediately reschedule it to start again at a later time.

A few actions, like merge and combine, are not incremental and can't be interrupted. But these actions are typically quick and generally don't take more than a few minutes to finish.

the not recommended method

QRecall is designed to tolerate crashes, network failures, power outages, and dog/cat/ferret/foot induced volume disconnects. If you have a running capture, you could just pull the plug on your external volume and run. The next action will detect that the archive was left in an unfinished state and rewind it back to where it was, undoing all of the work of the interrupted action. This works about 99% of the time. In the rare conditions where QRecall can't automatically recover, you'll need to repair the archive before continuing.

While this isn't the recommended method for stopping a capture, it should underscore that you don't have to treat archives with kid gloves. The archive structure is carefully designed to survive all kinds of disasters and data loss, which the repair command will recover from.

David,

It's likely that the cause is one of two reasons.

First, both Time Machine and QRecall exclude certain items. Both have an internal list of items that are never backed up along with a mechanism to arbitrarily mark items that should be excluded. For Time Machine, this "do not backup" attribute is a service of the operating system. Applications can, for example, mark files or folder that should not be backed up. Typically this is because they contain superfluous or transient data (like a video edit's render files).

Time Machine will always exclude items marked as "do not backup" by the operating system. QRecall will exclude them too, if you have the "Exclude items excluded by Time Machine" option set in the archive's settings. When this option is set, QRecall uses the same OS attribute that Time Machine does to exclude items (in addition to any other QRecall exclusion settings you might have set).

Since these two VM files don't appear anywhere in QRecall, I'm assuming that this is your issue. The solution/test (for QRecall) is to turn off the "Exclude items excluded by Time Machine" setting. QRecall will stop using the operating system's exclusion attribute to make capture decisions.

The down side of this change is that the operating system does mark a lot of things that really don't need to be backed up, and now QRecall will ignore that advice and capture them anyway. If this archive only captures your virtual machine files and nothing else, then turning off this setting won't capture too many superfluous files. If this archive also captures the rest of your system, then I would suggest splitting your archive into two:

- An archive with the "excluded by Time Machine" setting on that captures your entire system, but excludes your VM files.
- A second archive with "excluded by Time Machine" setting turned off that captures just your VM files.

This has other advantage too, such as the ability to trigger a re-capture of your VM files when you quit your VM software, or prevent QRecall from trying to capture your VM files while your VM software is running (which will undoubtedly result in an incomplete capture of your VM state).

The second possible issue is how OS X tracks filesystem changes. Both QRecall and Time Machine depend on the filesystem change history to quickly determine what portions of the filesystem have been modified. There are, however, well-known limitations in how the filesystem change history works that makes it blind to certain kinds of changes, and there are a few VM software packages out there are are notorious for making just those kinds of changes. The result is that neither Time Machine nor QRecall see the changes in your VM.

QRecall combats this deficiency with two features. First, it has a "Filesystem Change History Trust" setting (in the Advanced settings pane). Periodically, it will ignore the filesystem change history and exhaustively scan the entire folder hierarchy looking for modified items. The default interval is once a week. You can edit this interval, or set the "Deep Scan" option in the capture action. The later will force QRecall to perform an exhaustive search for changes every time that capture is run. The capture option was specifically designed to let you create a separate capture action for just your VM files that always ignores filesystem change history, while your other captures will continue to take advantage of it.

But since you say that your VM files are not captured at all, I suspect that the filesystem change history issue isn't your problem (although you might discover that it's a problem once you start capturing your VM files).

John,

You're not alone.

This has been on the to-do list for awhile, it's just a matter of coming up with a cogent UI and an efficient algorithm for performing the pattern matching during the capture process.

As a hack, it's sort of possible to do this now using the qrecall command line tool. You can use the tool to set the capture preferences for a file, and you can use the shell script to perform the pattern matching. Thus, it's possible to execute a script like this to manually mark all *.mov files your Document/Projects folder so that they are never captured:

find ~/Documents/Projects -iname '*.mov' -print0 | xargs -0 echo qrecall captureprefs exclude

Of course, this isn't dynamic. Every time you create new *.mov files you'd have to re-run this script to mark the new ones, but it's better than nothing.

I should also note that many modern applications will correctly mark intermediate files (rendering cache files and so on) using OS X's "do not back up" setting. If you set the archive's "Exclude items excluded by Time Machine" option, QRecall will honor those hints and automatically exclude those items.

My working UI design is to allow you to attach a note to an item (file or folder). That note would appear in the inspector window when you selected that captured file(s), and a summary of all notes would appear in the inspector panel when you selected a layer.

This feature might not be that far away, although the design of the UI has changed a couple of times. Right now I'm favoring an implementation that lets you attach short textual notes to items the same way you set the capture preferences.

So, for example, you could make a major change to a file and then attach a capture note. The new version, and the note, would both be added to the archive when the file was captured.

Ah, that's actually a distinction I wasn't considering?the difference between having an encryption key installed and ensuring that all of the data in the archive has been encrypted.

Thanks for the clarification.

I can't see what utility that would have, but I'll put it on the wish list.

Feel free to enlighten me.

Steven J Gold wrote:Remember RAM Doubler from Connectix back in the 90's?

Wow, does that bring back memories. It also reminds me of Stacker (from my bad MS-DOS days).

Connectix's patents on memory compression expired, and recently Apple used stuff from them to implement Memory Compression in what, Yosemite?, because it was faster to compress memory than to swap it to disk.

Startup drive I/O is pretty fast these days, so I suspect this is more to conserve disk space than for performance. Most boot drives use eSATA that can write 100MB/s, and 300MB/s is no longer uncommon. The newer SSDs can move data as fast as 500MB/s. Even with a fast 6-core CPU, it would be tough to compress 300MB of data in less than a second.

I actually found that the fastest way to move a large file (40+ GB) file from a USB-2 connected disk to my laptop is to Restore it from a QRecall archive to the target disk rather than do a straight copy. I assume this is because the archive is compressed and thereby takes fewer I/O operations to "read" the file than to do a Finder copy from the external disk?

(I would never have guessed I'd use QRecall as a "faster than Finder file copier" )

That's very cool, and makes perfect sense.

Steven J Gold wrote:BTW, does the existing quanta need to be decompressed for the comparison, or does the comparison operate on the compressed data?
I guess I'm asking if the de-duplication process is slower if the archive is compressed?

Short answer: yes, compression adds overhead, which means it's probably going to be slower.

Uncompressed file data is used to search for a duplicate block in the archive. If found, the quanta is decompressed and compared with the file data. If no match is found, the file data is compressed and written to the archive. This is potentially faster than first compressing the file data, because decompression is always faster than compression. In other words, QRecall avoids compressing a block until it has to.

Having said that, if you have really slow archive access (USB, slow network, ...) and a relatively fast (multi-core) computer, it's possible that using compression can actually speed up actions, if the amount of time the computer takes to decompress the data is less than the amount of additional time it would have taken to read an uncompressed record from the archive.

Welcome to the club. I upgrade my workhorse Mac Pro to and SSD last year and can't imagine life without it.

Steven J Gold wrote:... I expected QRecall to capture it as a new volume but since its contents are almost completely identical to the prior replaced volume, I expected the Capture to find 99% of the data already in the archive (it turned out to be 98.69%) and complete very rapidly.

So I was surprised when it took over 4 hours to capture 167.7GB since it actually only needed to write 1.53GB.

QRecall wrote 1.53GB of data, and read 335.4GB of data. Remember that de-duplication requires that every block of every source file be looked up in a gigantic database of captured quanta. Once found, the archive record containing the captured quanta is read and compared, byte-for-byte, with the data block in your file to ensure they are identical.

So even if the files you're capturing are 100% duplicates of what's in the archive, it still means QRecall has to read all of that data twice (once from the files and again from the archive).

Most of the capture speed improvements come from anticipating the data being captured or determining that a file is already captured and not reading it all. Both of those optimizations only happen when items are recaptured; they never happen during the initial capture.

Most surprising was the variance in speed it reported. Sometimes it reported "1.63GB per second", but sometimes only "7.28 *MB* per second" -- that's quite a magnitude variance(!). The average rate was 687 MB/min. I'm curious why it sometimes dipped into the single MB/sec digits.

QRecall has a lot of moving parts. It's really hard to tell what's going on from one moment to the next. Sometimes the capture needs to pause while directories are pre-scanned, hash tables are updated, record number indexes are pruned, or a glut of empty records are being erased. The bottom line is, unless you perform a sample of the QRecallHelper process while it appears to be stuck, I can't tell you exactly what it was doing (not that it's usually that interesting anyway).