Messages posted by: James Bucanek

I purposefully disabled resursive expansion (holding down the Option key while clicking an enclosure triangle) in the browser. The problem is that a single folder could potentially expand to thousands, or even hunderds of thousands, of items. QRecall *might* eventually come back after 10 minutes, if it doesn't simply blow up the operating system's outline framework (which is not designed to display that many items at once).

However, I'll look into revisiting this. I might limit the expansion to two or three levels, or maybe stop expanding subfolders once two or three hundred items are revealed. This might make you repeat the process to display everything, but it would certainly be less work than it is now.

Bruce Giles wrote:Option 1: Using a second Mac, ...

Yes, that would work.

Bruce Giles wrote:Option 2: I reinstall the OS on the original computer. Once it's reinstalled, I reinstall QRecall and then connect the backup drive and do a restore back to the original computer.

That will work too. QRecall can perform a "live restore" of the currently booted operating system (assuming you have a little free disk space). The operating system, naturally, can become confused if you try to use it while it's having its brains operated on.

What you'll want to do is:
1) Reinstall the OS. Don't bother upgrading, you're about to replace everything anyway.
2) Run QRecall, open the backup archive, select the volume to restore, hold down the Option key and choose the Archive > Restore To... command.
3) Do absolutely nothing with the OS or any applications while the restore is in progress.
4) When it's done, immediately restart.

The potential problem I see is that I'm trying to Restore the OS (among other things) on top of an already running OS. I'm guessing there are files that are open and in use that QRecall can't overwrite.

QRecall works around this problem by going to the BSD file APIs. These allow QRecall to overwrite (or delete) a file that currently in use. The original process uses the original file until it is closed. It seems counter-intuitive, but it works.

Is there a better way than either of these two options?

Create a bootable recovery drive.

If you're external drive is the kind you can boot from, install a minimal operating system and a copy of QRecall. If anything happens to your main volume, boot from the external and restore what you need.

1) Use your OS installation CD/DVD to install Mac OS X on the external drive. When you perform the install, customize the installation by unchecking all optional packages: Your recovery OS doesn't need extra fonts, or languages, or printer drivers. When the installation is complete, you can further trim the OS by discarding superfluous applications, like Chess. I typically throw away everything except Disk Utility and Terminal.

2) Copy QRecall to the backup drive

You now have a completely reliable emergency boot drive. Since the OS is stable (is not a copy of your working OS), there's no chance that any mishap or failed upgrade that could make your primary OS unbootable will affect your emergency system. You can boot, reformat, perform diagnostics, and restore with impunity.

Good eyes.

This won't make it into the next release, but should be included in the one after that.

This appears to be a race condition bug. Just out of curiosity, are you running QRecall on a single or multi-processor Mac?

I think I've resolved the problem. If you want to experiment, I've uploaded an inter-release build: http://www.qrecall.com/release/QRecall_1.0.0b42.i.dmg. Warning: This has a bunch of other untested changes, so try it at your own risk.

To install the new version: (1) download the disk image. (2) Trash (but don't delete) your currently installed version of QRecall. (3) Copy the new version into its place. (4) Launch the new version and reauthorize it. (5) Delete the old version.

Bruce Giles wrote:I re-launched QRecall and tried to open the same archive again, and this time it opened without any problems. I did not get any warnings about needing to reindex or repair the archive, so presumably it was unaffected by the crash.

When browsing, QRecall opens the archive in read-only mode. So browsing can't alter or damage the archive in any way, no matter what happens to the application.

(Well, that's not entirely true. You can change the archive settings while browsing, but the worst that would happen is the archive would lose its custom settings.)

Bruce Giles wrote:I now have an archive with a number of layers marked as "Unknown Zero KB -Damaged-", except for the latest layer, which is good. I'm currently merging all the layers, primarily just to see what happens. If I do a verify after that, and it checks out, then it should be safe to continue using, right?

Yes. If the last layer merged is complete, the merged layer will be complete.

Although if I decided to toss this one out and do a fresh capture, it should work this time, since I've apparently fixed all the folders that were vulnerable to the launch services bug.

That's the theory. I'd be very interested to know if you find otherwise.

Bruce Giles wrote:That sounds pretty cool, if it works. But suppose I have a 15 GB iTunes Music Library. I capture it, add album covers to a dozen songs, and then recapture it. Would the experimental code likely catch this and just backup the changed quanta on those 12 (out of 3000 or so) songs?

It would do that now, even if you turned shifted-quanta detection off. Duplicate quanta and shifted quanta detection are different. Duplicate detection works 100% of the time.

Shifted quanta detection is the special case where data is "shifted" relative to its previous location in a file (typically by reading it into memory, then writing it back out at a new location). iTunes and iPhoto don't typically shift any significant amount of data when one changes the metadata of a song or picture. This metadata is either stored separately, or tacked on to the end of the file. Either way, the bulk of the MP3, AAC, or JPEG data isn't changed or moved.

OK, but once the archive has been compressed, is there still a performance hit?

Yes, big time; which is why I don't recommend its casual use.

Once data in an archive is compressed, every operation that needs that data again requires QRecall to decompress it. During a capture, everything about a file has to be decompressed before it can be compared to the existing file. This is probably the biggest performance penalty. Browsing a compressed archive also requires decompression of every chunk of compressed file information. The same is true for merge and verify actions.

Bruce Giles wrote:Do I understand correctly that this only happens with folders containing applications?

To be honest, I'm not sure. I've only encountered this problems with applications, but I suspect that it might be related to bundles/packages in general. Again, I haven't been able to get a definitive reason from Apple as to why this is happening so I can't say for sure.

Bruce Giles wrote:Is this a good scheme, or am I missing something?

No, you didn't miss anything. It's a good scheme, but I'm working on a better one.

I current have experimental code that automatically throttles the shifted quanta detection up or down based on past performance and available resources. QRecall will randomly check blocks looking for shifted data. If it finds some, it will increase the amount of time it spends looking for more shifted data. If the data being captured doesn't contain any shifted data, it will throttle the shifted quanta detection back for better capture performance.

If this proves successful, I will replace the shifted quanta archive setting with a simple on/off setting, or possibly eliminate the setting altogether.

Also, regarding the two compression sliders, suppose capture compression is set to "faster larger", and compact compression is set to "slower smaller". Would a capture, followed by a compact, produce the same results as if I had set the capture compression to "slower smaller", and then not performed a compact?

Correct. The capture compression level is applied during the capture. If it is the same as the compact compression level, then the compact will not recompress the data added by the capture.

My advice: Unless you are running out of disk space, leave all compress settings off (faster larger). Compression can dramatically reduce the size of an archive, but it comes at a huge cost in performance. If you have free disk space, use that up before using compression.

Bruce Giles wrote:I've noticed that when QRecall displays files and folders captured in the archive window, it seems to be able to go only one layer deep when reporting folder sizes. That is, a captured folder that contains only files will have a folder size displayed, but a folder that contains other files AND FOLDERS, will display only "--" for the size.

Is this correct? Any way to get QRecall to report the correct size on all folders?

It's a bug (sort of). I plan to replace this with a mechanism that pre-calculates the size of folders during the capture so that the correct size of folders and packages are always displayed.

Right now, QRecall has to calculate the size by recursively adding up the size of all enclosed items. But that information is only read in when the contents of subfolders are displayed in the browser. If you expand every nested subfolder, then cause the top-level folder to redisplay (by scrolling or collapsing/expanding it), the correct size is displayed.

Bruce Giles wrote:When I try to archive my entire home folder, however, which is about 16 GB in size, it proceeds only so far (varies between about 2 GB and 6 GB) before I get a message that QRecall has lost connection with the process. Checking the logs shows that QRecallHelper has crashed.

This problem is due to a bug in Mac OS X 10.4. I am working with Apple to develop a workaround, but there isn't one yet.

The problem is with Apple's Launch Services system. This is the part of the OS that identifies objects on a file system. Given a file or folder, it will tell you what kind of item it is, what its localized display name is, what applications will open it, what icon it should have, and so on.

QRecall uses this facility to capture the icon of every file. However, if one calls Launch Services to obtain the icon of a bundle and that bundle is in a folder which has never been opened in the Finder, occasionally that call will crash:

The solution is annoying, but simple: Open the folder containing the offending item in the Finder. When the Finder displays an item for the first time it does something to update the local .DS_Store file and the Launch Services database which permanently fixes the problem for that item.

For example, when I install a fresh OS I find I must open up the Applications and Utilities folder once to display all of the applications it contains. Once I do that, QRecall can capture the /Applications folder without any problem.

If you find it difficult to locate which item(s) is having problems, create a test archive and manually capture folders one at a time until you find the one that crashes.

As for why your QRecall application is crashing, I'll have to investigate that. The crash appears to be associated with a window. Does it crash when you open a specific kind of window (like the log or actions window) or when you open a specific archive?

Al Pawlowski wrote:Is there a way to mark a file so that a merge would not delete it, even if deleted in layers being merged? If not could some kind of marking, say a file name additive like [ ] around the nam, be put into your "to do" list?

Is your primary concern that you want to make sure you keep the last copy of each file in the archive?

Trying to merge two layers (where a file exists in the earlier layer but doesn't exist in the latest layer) but keeping the deleted file doesn't fit the model used by QRecall. A layer is a snapshot of the what files do, and do not, exist at a single point in time. The only way to keep the older copy of the file would be to maintain a second layer -- which is exactly what you already had before the merge. So the short answer to this problem is don't merge.

I've considered this and similar problems in the past and have been toying with a number of possible solutions. But I haven't come up with any yet that are both simple and keep the layer information consistent.

Here's my suggestion: If you have a group of files that you want to keep the last version each day, use two archives.

Create one archive for your active documents that you capture on a daily or hourly basis and a second for everything else. For the first, have a rolling merge keep daily layers for that archive going back as far as you want (30 day, 6 months, ...). Unless you're working on large multimedia files, keeping a year or more of daily backups of your documents shouldn't take up too much space. Because of QRecall's efficiency, a layer with no changes takes up virtually no storage.

Use a second archive to store everything else on the volume. That can be captured daily or weekly, and you probably only need to keep a few daily layers before merging older layers into weekly or monthly layers.

This will let you restore your OS or applications folder with a fairly recent version should something happen to it, while being able to recall that last version of the day of any document during the past year.

I realize that I could have a copy of old "not being used" files where I was not likely to mess with them, but that is not as easy as keeping all my related files together. If I did have such a copy, it would be nice if recall had a marker to mark files you do not want captured that are in a directory that you do want captured.

When creating a capture action, you can add whatever files or folders you want to the filter section. Those items will be ignored (treated as if they did not exist) during the capture. This lets you exclude whatever you want from regularly scheduled captures.

By the way, great program concept and I am doing a capture right now to a FAT32 networked attached (NSLU) USB drive that seems to be working fine.

QRecall should be able to use an archive on virtually any file system that correctly supports file sizes larger 2GB.

Disclaimer: I've never tested QRecall on an SMB mounted volume and have no idea if it will even work. QRecall was designed to work with HFS+ volumes, although I'm open to getting it work with other filesystems as time permits.

With that out of the way, I'd be very interested in getting a copy of your log file (~/Library/Logs/QRecall/QRecall.log) that covers the time both for successful and unsuccessful captures. You can send them directly to james@qrecall.com.

I can imaging all kinds of things that might be going wrong. Looking at the logs will help me narrow down the possibilities.

One item on my to do list is to add a drawer to both action document windows and the main Actions window that would show a list of upcoming run times. The drawer in the action document would show the schedule just for that action, while the Actions window drawer would show a complete list of all upcoming run times.

Try this:

Open the actions windows in QRecall (Windows > Actions, or Command+1). Click on the Next column header. This will sort the actions table based their next scheduled execution time, essentially giving you a schedule of when actions will start.

That's typically the view I leave my Actions window in. Maybe I should make it the default.

Gary,

F25s are fun to own, even if you just use it to get the mail.

A rolling merge is close to what you want, but has one tiny hiccup; Any layers created today are considered part of the "recent" tier (even if most recent is set to 0) and are never merged.

To get what you're asking for, create a simple Merge action instead of a Rolling Merge action. Set the merge criteria to

From: 0 layers after first layer
Through: 0 layers before last layer

That action will merge every layer in the archive into a single layer. Schedule it run immediate after your daily capture.

I'm glad you enjoy the app.