Suppose I'm archiving my entire home folder daily. This includes a number of files (mail files, for instance) which will change daily, but for which shifted quanta detection would be useful. (Many mailboxes will change only in the last few messages added or deleted.) This also includes a large iTunes Music Library. I daily add and delete a number of podcasts, but most of the library is songs, which don't change except for the tag data (play count, last play date, etc.) Again, I think shifted quanta detection would be useful here.

The problem is that if the home folder is large (10 to 20 GB), and I crank shifted quanta detection in the archive settings all the way to the "slower smaller" end of the scale, it'll take hours, if not longer, to do the initial capture. Backing up an entire hard drive (several hundred GB) would be even worse.

As it happens, however, I believe that I have very few duplicate (or even partial duplicate) files in my home folder. If that is correct, then on the initial capture, wouldn't QRecall be mostly wasting its time looking for shifted quanta?

Assuming I have adequate space for the archive, would it not be better, for the inital home folder (or entire hard drive) capture, to move all the sliders to the "faster larger" end of the scale? This would allow the initial capture to proceed as quickly as possible while not wasting time looking for shifted quanta that probably don't exist. Then, AFTER the initial capture, move the shifted quata detection slider (and possibly the compression sliders as well) to the "slower smaller" end of the scale. That way, subsequent recaptures, where the liklihood of shifted quanta is great, would be optimized. Yes, the recaptures would run much slower, but there would likely be far fewer files that need to be recaptured, as compared to the initial capture. We're probably talking only a few minutes per day at that point.

Is this a good scheme, or am I missing something?

Also, regarding the two compression sliders, suppose capture compression is set to "faster larger", and compact compression is set to "slower smaller". Would a capture, followed by a compact, produce the same results as if I had set the capture compression to "slower smaller", and then not performed a compact?

-- Bruce

Bruce Giles wrote:Is this a good scheme, or am I missing something?

No, you didn't miss anything. It's a good scheme, but I'm working on a better one.

I current have experimental code that automatically throttles the shifted quanta detection up or down based on past performance and available resources. QRecall will randomly check blocks looking for shifted data. If it finds some, it will increase the amount of time it spends looking for more shifted data. If the data being captured doesn't contain any shifted data, it will throttle the shifted quanta detection back for better capture performance.

If this proves successful, I will replace the shifted quanta archive setting with a simple on/off setting, or possibly eliminate the setting altogether.

Also, regarding the two compression sliders, suppose capture compression is set to "faster larger", and compact compression is set to "slower smaller". Would a capture, followed by a compact, produce the same results as if I had set the capture compression to "slower smaller", and then not performed a compact?

Correct. The capture compression level is applied during the capture. If it is the same as the compact compression level, then the compact will not recompress the data added by the capture.

My advice: Unless you are running out of disk space, leave all compress settings off (faster larger). Compression can dramatically reduce the size of an archive, but it comes at a huge cost in performance. If you have free disk space, use that up before using compression.

James Bucanek wrote:I current have experimental code that automatically throttles the shifted quanta detection up or down based on past performance and available resources. QRecall will randomly check blocks looking for shifted data. If it finds some, it will increase the amount of time it spends looking for more shifted data. If the data being captured doesn't contain any shifted data, it will throttle the shifted quanta detection back for better capture performance.

If this proves successful, I will replace the shifted quanta archive setting with a simple on/off setting, or possibly eliminate the setting altogether.

That sounds pretty cool, if it works. But suppose I have a 15 GB iTunes Music Library. I capture it, add album covers to a dozen songs, and then recapture it. Would the experimental code likely catch this and just backup the changed quanta on those 12 (out of 3000 or so) songs? If not, then I might prefer a manual setting option for those times when I know the changes were "minor".

My advice: Unless you are running out of disk space, leave all compress settings off (faster larger). Compression can dramatically reduce the size of an archive, but it comes at a huge cost in performance. If you have free disk space, use that up before using compression.

OK, but once the archive has been compressed, is there still a performance hit? Suppose I capture something with capture compression turned off and compact compression turned on, and then schedule a compact for later when I'm going to be away at work, so I don't really care long it takes to do the compression. Will the performance of subsequent captures be affected, assuming that capture compression is still turned off?

-- Bruce

Bruce Giles wrote:That sounds pretty cool, if it works. But suppose I have a 15 GB iTunes Music Library. I capture it, add album covers to a dozen songs, and then recapture it. Would the experimental code likely catch this and just backup the changed quanta on those 12 (out of 3000 or so) songs?

It would do that now, even if you turned shifted-quanta detection off. Duplicate quanta and shifted quanta detection are different. Duplicate detection works 100% of the time.

Shifted quanta detection is the special case where data is "shifted" relative to its previous location in a file (typically by reading it into memory, then writing it back out at a new location). iTunes and iPhoto don't typically shift any significant amount of data when one changes the metadata of a song or picture. This metadata is either stored separately, or tacked on to the end of the file. Either way, the bulk of the MP3, AAC, or JPEG data isn't changed or moved.

OK, but once the archive has been compressed, is there still a performance hit?

Yes, big time; which is why I don't recommend its casual use.

Once data in an archive is compressed, every operation that needs that data again requires QRecall to decompress it. During a capture, everything about a file has to be decompressed before it can be compared to the existing file. This is probably the biggest performance penalty. Browsing a compressed archive also requires decompression of every chunk of compressed file information. The same is true for merge and verify actions.