Gabriele Caniglia wrote:- how do I exclude files from capture whose name is/contains/starts_with 'something'?
Say I want to exclude ".DS_Store" files or "Thumbs.db"?

Filtering items by name or other criteria is an oft requested feature that hasn't been added yet—but it's on the to-do list.

- how do I add invisibile files and folders to the filters list? I have workarounded this by opening a Disk Broswer window in BBEdit, then dragging and dropping the invisible items onto the filters section. Is there a better way to accomplish this?

That's an excellent solution. I'll add a request to see invisible files in the picker dialog to the wish list.

- how can I schedule multiple captures at specific times of the day? Say I want to capture at 9:30am and 4pm from Mon to Fri. I can only see an "Interval" option or a "Daily" option...

Christian Roth answered this one already. You can define as many actions as you want for a single archive, each with their own schedule and criteria. For example, I capture my development folder every 20 minutes, my home folder every 2 hours, and my entire startup drive once a day.

Chris Caouette wrote:any eta for an update?

A few more hours.

I've been struggling with a persistent performance problem with the package and names index. They perform just fine on small archives, but their processing overhead grew exponentially relative to the size of an archive.

For months I tried to find a feasible solution, but eventually gave up on the code. In late June I rewrote the entire package and names indexing code from scratch. I've been testing it for the past couple of months, fixing bugs, and refining performance.

The fruits of that labor have been running here for about a three weeks now. If the rest of the testing goes well, I'll push out a release towards the end of this week.

This has been a huge roadblock to adding new features, which I hope to get back to very soon.

Charles Watts-Jones wrote:I seem to recall that QRecall should alert the user to errors through a QRecall Activity Monitor message which is displayed until dismissed by the user. This does not seem to be so on my machine and I wonder if it's a problem with my installation or something more general.

In general, QRecall tries to be as annoying as possible when an error occurs.

However, there are some limitations which might be affecting you.

The warning indicator appears in the QRecall Activity window when an action fails to finish properly. If the activity monitor process isn't running, most likely because you're logged out at the time, then there's nothing to catch and display the warning. Also, if you use spaces the warning may be appearing in another space.

Is there a way to ensure that QRecall advises errors automatically? If not, might I add it to the wish list?

QRecall also works with Growl, which has many notification options—but again only while you're logged in.

I'll add a wish list item to be notified of problems that occur while you are logged out.

Steven Arnold wrote:To followup on this issue, I have now had an archive on a regular disk -- not a DMG, encrypted or otherwise -- since June 18th. That's about three weeks. Normally the problem would have come up by now, so I would conclude that QRecall has a problem with large archives on DMGs. Note that the problem could be an artifact of encryption, or it could just be an issue with DMGs.

Thanks for following up on this.

If you need some help writing a script to do this, I can probably knock something out in Ruby pretty fast to create the original files to be backed up, and then add a few files at random locations.

Thanks for the offer, but I've got plenty of QRecall test cases already set up. I'll put this on my list of things to test and see if I can't get it to happen here.

Steve Mayer wrote:Where are these stored so that I don't have to set these up again?

Actions are stored in the Actions folder inside ~/Library/Preferences/QRecall.

If you just copy those action files back into the Actions folder, the scheduler won't "see" the change until the next time it restarts or you make a change to the schedule. The easiest way to accomplish the latter is to create any new action (it doesn't have to do anything), save it, and then delete it. This will cause the scheduler to rescan all of the action files and update its schedule.

Bruce Giles wrote:The virtual machine file is currently about 60 GB in size, and is actually a package. The virtual disk itself is broken up into numerous chunks, each no larger than 2 GB in size. (That's one of the options for how Fusion segments and stores its virtual disk.)

This is a popular trend in large disk images, primarily so that file-oriented backup and synchronization applications don't have to copy the entire image every time.

So, I'm wondering how QRecall deals with this.

QRecall deals with this like any other set of files. It recaptures any of segment files that have changed since the last capture, analyzing each one for changed data.

Even though QRecall is backing up only the parts of the virtual disk files that change, it still takes significant time to back them up, and they're continuing to change during the backup. If I ever have to use QRecall to restore the virtual machine file from a backup, is it (the VM file) even likely to work? Or do I end up with a virtual disk inside the VM file in which the various chunks are no longer consistent with each other, because they were backed up at different times during the archive session?

The fundamental problem here is that the files are changing during the capture. The fact that there are multiple files is immaterial; the problem is essentially the same whether there were one file or a hundred.

It doesn't matter what backup/copy/synchronization utility you use, the inescapable problem is that the contents of the file are an incomplete representation of the current state of the application. This is also true for database files, dynamically modified document files, and so on. But it's particularly true for disk images, since the directory structure of the disk will more than likely be inconsistent with what's buffered in memory.

The same problem exists when you capture the OS X operating system. It's possible because most of the files that are constantly being modified aren't that significant. These include log, cache, scratch and temporary files along with memory mapped files. Failing to capture all of the data that should be in those files doesn't prevent your system from restarting.

I suspect that the same would be true for the VMWare disk image. Most of the files (the OS, applications, documents, ...) would be stable and are dependably captured. The one transient data structure that's most likely to give you problems is the directory structure itself. After recalling an older version of the VMWare disk image, I would immediately repair the volume before proceeding. If you have the disk space, I think it would be a good experiment to try this a couple of times using various captured images.

Would I be better off to not back up the VM file until the virtual machine is not running?

That's the only real solution—and that's independent of what backup solution you're using.

Chris Caouette wrote:My three macs have been running QRecall happily for several months now but I started wondering when we might see an update with any new features etc.

Chris,

Thanks for posting. My original plan was to add some of the many planned features to QRecall this spring and summer.

Unfortunately (for QRecall) I was sidetracked by two book deals. I'm currently writing Learning Objective-C for Java Developers to be published by Apress, and Professional Xcode 3 for Wiley Publishing. These two projects are currently consuming 98% of my available time, forcing QRecall development to the back burner for the moment.

I do have a dot release planned to fix a number of minor bugs and provide some performance improvements, pending the resolution of two very hard-to-reproduce bugs.

Once the books are behind me, I'm planning on full-time QRecall development—and I've got about a billion new features I want to implement.

In the mean time, please stay tuned. Let me know if you're having any problems, and don't hesitate to send or open a discussion on new features.

David Kaminsky wrote:If I start the backup again will the incomplete files (if QRecall doesn't finish the file it was copying when cancelled) be recopied?

Yes. QRecall always captures every file and folder that differs from what's in the archive. So the next capture will capture everything the incomplete layer missed, along with anything else that might have changed since the incomplete layer was captured.

Do I need to merge the last 2 layers?

You can, if you like. Merging any earlier layer with a complete layer leaves a complete layer. I prefer to merge incomplete layers with complete ones to avoid any "confusion" in the future. But if you choose to keep the incomplete layer for some reason, it won't affect how QRecall behaves.

I assume the archive files are treated as just files and can be copied themselves and put back in place if needed.

The archive itself is a package (a directory of files that acts as a single file in the Finder). It can be copied, moved, renamed, synchronized, or backed up like any other document.

Thanks for your assistance.

My pleasure.

Steven,

Steven Arnold wrote:Here's a theory: maybe iTunes changes something about one of the files being backed up during the backup process. There are two cases of potential importance, I think. One is that the file is changed after QRecall backs it up. This might cause QRecall to look at the file and die because it sees it's different.

QRecall does not compare the data it captures with the original files in order to verify it's validity. It's not a definitive test, for exactly the reason you mentioned. Instead, QRecall reads the original file and adds additional checksum and data consistency information to the archive. This allows QRecall to determine if the file data in the archive is OK, even after the original has changed or been deleted.

The second, probably more serious case, is where the actual file itself changes partway through the process of QRecall backing it up.

Another very valid concern, but that's not your problem. The error you got was an I/O error trying to finalize the archive. This happens long after all of the files have been captured, so it can't have anything to do with the source data files.

QRecall tries to deal with mutating data files in three ways:
- QRecall reads files in large chunks, very quickly. For most files (<12MB) this gets a "snapshot" of the file as quickly as possible, so any future modifications don't influence what's captured.
- If the file being captured is replaced, QRecall will stop capturing the file and start over. This gets logged as minutia.
- If the file is extended or truncated during the capture, QRecall will record that as a warning in the log. You'll want to recapture the file.

These are, admittedly, stop-gap measures that try to make as accurate of a capture as possible. They are all part of the larger issue of trying to copy files that are open and being modified, for which there is no complete solution. This is just a consequence of the way the operating system works and there are no backup solutions available that can completely eliminate the possibility of capturing a partially modified file.

If you have groups of files that are being modified regularly, and you're running OS X 10.5 or later, you might consider adding a "pick-up" capture that immediately follows your regular capture. Starting with Leopard, QRecall will very quickly recapture any files that were modified after the first capture started.

Steven,

Thanks for sending a diagnostic report.

I've looked at your log files, and you're encountering a consistent problem. Your archive is occationally encountering an I/O error when QRecall tries to close the archive. This happened on 4-30, and again 5-14:

2009-04-30 07:19:38.672 -0400 Failure Problem closing archive

2009-04-30 07:19:38.672 -0400 Details could not set file size

2009-04-30 07:19:38.672 -0400 #debug# IO exception

2009-04-30 07:19:38.672 -0400 Details Path: /Volumes/encrypted_backups 2/music_and_video.quanta/hash.index

2009-04-30 07:19:38.672 -0400 #debug# OSErr: -36

2009-04-30 07:19:38.672 -0400 Details Pos: 3221225528

2009-05-14 07:19:49.005 -0400 Failure Problem closing archive

2009-05-14 07:19:49.005 -0400 Details could not set file size

2009-05-14 07:19:49.005 -0400 #debug# IO exception

2009-05-14 07:19:49.005 -0400 Details Path: /Volumes/encrypted_backups 2/music_and_video.quanta/hash.index

2009-05-14 07:19:49.005 -0400 #debug# OSErr: -36

2009-05-14 07:19:49.005 -0400 Details Pos: 3221225528

Once an error like this occurs, the archive is considered suspect until it can be reindexed or repaired. Until that happens, QRecall will refuse to use it.

The problem (i.e. "curse") of QRecall is that it's absolutely fastidious about data integrity and it checks its work afterwards. The "invalid header length" errors you encounter afterwards is QRecall's way of saying that the file size that expected to find doesn't agree with the actual file size. This makes sense, since both errors occurred when trying to set the file size before closing the archive.

QRecall attempts to reindex the archive, but this inevitably fails, and it then has to rebuild the archive. I am not sure what the difference is, but both processes take about equally long -- and a very long time.

All the important data in an archive is stored in a single file. Most of the remaining files in an archive are "index" files that provide rapid access to archive data. If the data file is undamaged, a reindex reconstructs the index files by scanning the entire data file. A reindex assumes the single data file is completely valid; any problems or inconsistencies will cause it to fail.

A repair is very similar to a reindex, except that it make no assumptions about the information in the data file. It too scans the entire data file and rebuilds the indexes, but automatically "fixes" any problems that it finds in the data file.

So as long as you don't have any problems with your master data file, a reindex will fix your archive. However, there's no way of knowing that in advance. If reindexing your archive takes a very long time, just repair it instead. If the data file is OK, repair and reindex are essentially identical. If the data file isn't OK, the repair will fix it.

Now, back to the real question of why this is happening. Alas, I don't know. An I/O error (-36) is the generic "something when horribly wrong and I can't perform this file operation" error. It really doesn't say what the problem is or what could be done about it.

Two I/O errors performing the same operation at the same point in processing is unlikely to be a coincidence. "Real" I/O errors are typically random events caused by hardware glitches or by accidentally pulling out a FireWire cable.

I've never seen this problem in testing, so I suspect it's something specific to your environment. You say that the archive is on an encrypted volume? Are your other archives also encrypted? Are the as large? Have/could you try moving it to a non-encrypted volume to see if you have the same problem?

Steven,

Thanks for posting. I'm glad that you've found QRecall useful, despite the occasional problem.

I have two questions. First, what exactly is the problem that's causing your archiving to require repair? The answer should be in your log files, which I'd like to look at. You can send a diagnostic report or just attach/e-mail your recent log files (~/Library/Logs/QRecall). The failures that would require a repair are typically data corruption problems.

Which leads me to my second question. Is this archive on the same volume and/or physical drive as your other archives?

Ralph Strauch wrote:I tried to stop the backup by clicking on the x in Qrecallmonitor, with no response. ... While it would have been nicer if the interrupted backup had responded to my attempts to quit from Qrecall Monitor, I found this behavior overall acceptable and a big improvement over the way Qrecall used to handle such interruptions the last time I had one a year or so ago.

Ralph,

Thanks for the feedback. Technically, there's not much that QRecall can do in a situation like this. Manually killing the QRecallHelper process, as you did, is exactly the course of action I would have suggested.

Unplugging a hard drive may cause the thread that's trying to write to the volume to seize. The other threads in the process weren't affected, which is why the action appeared to continue running and responded to requests to quit. However, it can't actually quit until the stuck thread resumed—which I suspect it never would.

I agree that the auto-recovery feature added to the latest version is great improvement over having to repair the archive. I know it's saved me many a time.

Christian Roth wrote:I think the Instruments (an app included in Apple Dev Tools) file I linked to in my last post should contain already one, if you want to have a look at it right away.

I downloaded your instruments file and took a look. Sure enough, nearly 100% of the first thread's time is spend in NSHashInsertIfAbsent, which is the function that's called to add one entry to the name index hash table.

I'll just wait for the new release, since the archive in question is my actual working archive which I do not want to lose/corrupt.

I'll keep you posted. Hopefully, I'll have some time soon to squash that lingering bug and get a maintenance release out.

Christian,

Standard disclaimer: There's a lot that happens when an archive is being closed. How long will depend on how many new quanta were added to the archive, how many unused data blocks (aka "holes") are in the archive, the speed of your data connection with the archive, and so on. It could take 10 seconds or 10 minutes.

However, I sheepishly admit that you're probably encountering a serious problem with the names index that's present in the current release.

As the names index grows in size—and it sounds like you're is larger than most—the amount of time spent reading, updating, and writing the names table becomes enormous. I've seen QRecall sit and spin for 40 minutes just sorting the names index.

For the most part I've fixed this already. The new names index routines are anywhere from 10 to 60 times faster than the current release version. Unfortunately, there's another bug in the names index that I really need to get fixed before I release a new version, and I just haven't had the time to do that yet.

If you want to verify this, you can use the 'sample' tool to sample the QRecallHelper process when it gets stuck closing the archive. You'll have to run it as root (i.e. sudo sample ...) if QRecall is pre-authorized to use administrative privileges. Send me the sample file and I can tell if it's the same problem.

If this is causing you a great deal of grief, you're welcome to try—at your own risk—the unreleased QRecall 1.1.1 that I've given to a few users experiencing other problems, also fixed in this version.

Hansen Teidee wrote:the capture-process keeps on breaking up. dont kwow by what its caused.

Hannes, thanks for posting your log files.

What I see are bunch of I/O and data integrity errors. This usually indicates one of three problems:

1) The volume the archive is has a corrupted directory structure.
2) The hard drive is losing data.
3) You have RAM or data transfer problems.

The first problem is easy to check and fix. Use Disk Utility to repair the volume. If problems are found, and fixed, then repair your QRecall archive and go back to taking backups. That's probably all it was.

The second and third problems are more difficult to isolate. I'm not aware of any utilities that will fill an entire drive with test data and then verify it. I've considered creating one myself. This is what you really need to detect problem 2.

There are a number of utilities that will perform RAM tests. TechTool Deluxe that is included with most AppleCare plans should be sufficient. The Integrity utility by Intech will do a good job of detecting data transfer or firmware problems between your computer and drive.