Wayne Fruhwald wrote:There are times when I would like to merge two layers preserving those files (e.g. only remove the older version of files that exist in both layers).

Wayne,

I'm working on a new "Keep Last Version" feature that will preserve the last version of a deleted file (limited to some user-defined amount of time). I think that will give you what you're looking for.

David Kaminsky wrote:I'd like to have a "merge trial" function which would tell me how much space will be saved by doing a particular merge.

David, that's an interesting suggestion. I've also had many requests for statistics about how much duplicate data is contained in each layer, and in the archive as a whole.

I've put off implementing any of these because the calculations required are rather expensive. It requires building a map of all the data references in the archive, then isolating those used by one layer but no others. The "Searching for free space" phase of the capture action that occurs following a merge is (essentially) the same calculation. As you've probably already experienced, that can be a very time consuming process—often requiring 10 minutes or more to complete.

Nevertheless, I've had enough requests for this kind of information that I'm considering implementing some form of free/duplicate space calculator in the browser. With multi-core CPUs and multi-GB RAM becoming the norm, maybe it will actually turn out to be a usable feature.

Charles,

I'm sending you a special version of the QRecallKickStart deamon. This is the system component that causes launchd to start your scheduler process as soon as you boot (rather than waiting for you to log in the first time). Hopefully, it should shed some light on what's going on—or not going on.

Bruce wrote:I think your software would be my favourite backup-solution if it would encrypt the backuped data.

Bruce,

This is an often requested feature and is currently planned for the next major release. In the mean time, several users are backing up to archives they created on encrypted disk images.

Chris Caouette wrote:Maybe having a skip free space search can become preference in a future update?

It's already on the to-do list.

Chris Caouette wrote:Why is my scheduled backup spending about 20 minutes searching for unused space? Should I reindex?

Chris, reindexing won't help. When QRecall captures new items, it tries to identify and reuse any wasted space in the archive. This obsolete space is created when you merge layers. Merging may result in records of files, directories, or individual blocks of data that the archive no longer needs to keep.

Following a merge, the next capture or compact action begins by cross-referencing every record used in the archive in order to determine isn't being used. If the archive is particularly large, complex, or access to the archive is slow (say, via a network), then this can take some time.

If the amount of time spent locating free space is becoming excessive, here are two solutions:

Merge less often. A merge is what causes records to be freed and triggers the search for free space during the next capture or compact action. If you haven't just merged, neither of those actions have anything to find. Set the schedule of your merge actions to run, say, once a week.

Turn off the the free space search for capture actions. Turn off the free space sweep by issuing the following command in the terminal (see Advanced QRecall Settings). Make sure you schedule a compact to occur at least once a week. The capture will skip looking for empty space, even after a merge, but the compact still perform its normal maintenance.

defaults write com.qrecall.client QRCaptureFreeSpaceSweep -boolean false

The second solution is particularly well suited for situations where you capture from a remote computer (say a laptop) to an archive on a desktop via a network. The laptop can capture quickly, and the desktop can schedule local compacts to keep the archive tidy and efficient.

Charles Watts-Jones wrote:FYI there were no com.qrecall files in the /Library/LaunchDaemons, /Library/LaunchAgents, and ~/Library/LaunchAgents folders. While the /Library/Application Support/QRecall and the ~/Library/Application Support/QRecall folders existed, they were empty.

That's the way it should be. When you use Shift+Option+Command+Q to Quit and Uninstall QRecall, it should remove all of those components. I just wasn't taking any chances.

Well, it appears from your log that everything was reinstalled correctly but your scheduler still isn't getting started when you start up. Why I can't say. Launch your Console application and look in your console and system logs for anything from the QRecallKicker (just search for "qrecall"). If you find any messages, send them to me. If you don't, I'll send you a special version of the QRecallKicker daemon that logs extra detail to the console log.

The QRecallKicker daemon is a system daemon that (should) poke the launchd service into starting your scheduler when your computer boots. My guess is that either it's not running, or launchd isn't starting your scheduler when it gets poked.

Two quick questions: Do you have your system configured to auto-log in, or boot to the login screen? And is your /Users folder on a different partition than your main /System folder?

Duly noted. I've already added it to the wish list.

All good suggestions!

Wayne Fruhwald wrote:At times I need to search for a file that is no longer on my hard drive (I may have deleted it days, weeks or months ago). Is there a way to search for that file across all layers?

Yes, but it requires waiting for the next version of QRecall. I have an "x-ray" view on the drawing board that let deleted items from earlier layers show through later layers.

In the mean time, you can set the search options to "Only show matching files", enter you search string, then start moving the layer shade backwards (Command-up arrow). The lost file should appear pretty quickly.

Wayne Fruhwald wrote:I would guess that you currently have a layer index that has an entry for each file in that layer.

Programmers always find it amusing with other people guess how their code works.

Assuming that to be true, it would seem that "Delete Item" for just one layer would only entail removing the appropriate file entry from that layer rather than removing it from all layers.

I must be missing something here. Would you explain why removing a file entry from one layer would be "difficult to implement."?

OK, if you want to get technical, here's the problem. Each layer is essentially a directory structure that represents a complete picture of the captured items at the time they were captured. As QRecall examines the items being captured, it tries to find duplicate data and directory information already in the archive and creates links to it (not really links, but references since the archive is really a database not a directory structure). If a block of data is the same, it links to it. If a file hasn't changed, the layer simply refers to the file record in an older layer. If a directory hasn't change, it stores a reference to that entire directory. So each layer inherits everything that hasn't changed from previous layers.

Deleting an arbitrary file record from a layer would corrupt the data structure if any later layers referred to that record. Likewise, deleting a file from an arbitrary layer changes the "snapshot" of it's enclosing folder. This means that subsequent references to that enclosing folder in later layers are now invalid and might have to be rectified. This in turn would alter any reference to that folder, and its enclosing folder, and so on, and so on.

This problem is completely avoided by deleting the same item from every layer. This "vertical" deletion guarantees that there can't be any dangling references between layers, because all references to the item are deleted from every layer. The only housekeeping that must be done is to remove any direct references between the item and its enclosing folder in each layer, to and account for the size change in those folders. This is pretty straight forward and can be done very quickly.

So now you know.

Wayne Fruhwald wrote:There are times I would like to selectively delete an item from just one or more layers in the archive.

That's a tough one. I'll put it on the wish list, but that would be somewhat difficult to implement.

For example, I may have a large VM and to save space, I would like to delete older versions of the VM while preserving newer ones.

My knee jerk response is to say "then merge the layer with a newer one." That's what merging does. But trying to merge some items in a layer but not others creates anomalies in the archive's structure.

Wayne Fruhwald wrote:Hi James.

Hello, Wayne.

I would like to have the ability to have columns for the information contained in the "Captured n items, n KB (n% duplicate)" section of the log file.

For example, I would like to add columns for: Captured, Written, Duplicate, Files, Folders and Rate.

Unfortunately, those statistics are per-capture, not per-layer. When you merge two layers, there's no reasonable way to recalculate them.

What is sounds like you're interested in is a kind of "capture log" for an archive. I'll put that on the wish list.

It would also be nice to have a comments field to describe the reason for the capture (e.g. "Before applying Mac OS X 10.5.8 update"). This would also be a column field.

A generalized annotation feature is planned for an upcoming version.

Thanks for the suggestions and the feedback.

James

Charles,

I did find some inconsistencies in your log records. For example, at one point you changed from running the scheduler only when you're logged in to always. In the log, I can see that the "agent" scheduler is shutdown and uninstalled. Then the "daemon" scheduler is installed, but it never starts. I think this is why QRecall spontaniously asked if you wanted to install it. That makes sense, but why it didn't start is a mystery.

Anyway, I did find a recent occurrence where your scheduler was configured to run when you're logged out and did, indeed, continue to run through several log-out/log-in cycles. So that's good.

But some things still seem confused. I'm going to suggest that you (almost) completely uninstall QRecall and set it up again. This will require two restarts, so do it when it's convenient:

1. In QRecall > Preferences > Authorization, uncheck the option "Start and run actions while logged out."
2. Hold down the Option and Shift keys and choose the QRecall > Quit and Uninstall command.
3. Trash any files beginning with com.qrecall from the /Library/LaunchDaemons, /Library/LaunchAgents, and ~/Library/LaunchAgents folders. (Do not attempt to empty the trash yet.)
4. Trash the /Library/Application Support/QRecall and the ~/Library/Application Support/QRecall folders (if they exist).
5. Restart your computer. (Now you can empty your trash.)
6. Launch QRecall.
7. In Preferences, pre-authorize QRecall to use Administrative privileges.
8. Check the option "Start and run actions while logged out."
9. Restart your computer.

After all of that, log out and back in at least twice (there's no rush) and then send me another diagnostics report.

Charles Watts-Jones wrote:I tried this (deleting the preference to run when not logged) and all went well. Activity Monitor confirmed that the Scheduler was running and QRecall ran a back-up as soon as the external drive mounted. QRecall then surprised me by asking if I wanted it to install the Scheduler (which was running at the time).

This seems completely wrong. If the scheduler is configured to run only while you're logged in, then QRecall should never ask to install it. It only does this when the scheduler is set to run when you're logged out and you haven't pre-authorized QRecall to use administrative privileges. In this one circumstance, QRecall needs your administrative privileges in order to install the scheduler—the problem being is that is shouldn't be asking in the first place.

Something strange is going on. It almost sounds like QRecall's preferences don't agree with the installation. What I need to do is sit down with the log records you've sent me and see if I can determine where the processes is going wrong. I may also send you some simple shell commands to execute, to verify that we don't have a permissions or file ownership problem, and that the scheduler is getting installed correctly.