QRecall seems temporarily stuck

Here's a problem that I've noticed -- occasionally -- for quite some time. I've been running the betas of QRecall for months, so I can't say if this was happing in the non-beta versions or not.

Sometimes, while an archive operation is executing, I'll notice in the activity window that a certain file is currently being archived. I'll go away and do something else for a while, and I come back 20 or 30 minutes later, and the activity window is still "stuck" on the same file. When it happens, I'm never sure if QRecall is still actually working on the same file, or if it's just that the activity window isn't updating the information for some reason.

I've never seen it happen on the same file more than once, and I can't see any pattern to explain when it does happen. Today it happened while archiving Adobe Reader in my Applications folder. I've also seen it get "stuck" for many minutes on a small data file (less than 10K). Eventually, QRecall always goes on, and there's no indication of any problem in the log window.

When this happens, is it because QRecall is doing some kind of time-consuming housekeeping in the middle of an archive? If not, do you have any idea what *is* happening?

-- Bruce

Bruce Giles wrote:When this happens, is it because QRecall is doing some kind of time-consuming housekeeping in the middle of an archive?

More than likely, this is the case. QRecall maintains an entire menagerie of caches, indexes, and lookup tables. Almost all of which require periodic maintenance. They need to be occasionally collated, sorted, written to disk, and so on. And, as you guessed, quite often this needs to be done at unpredictable times, usually in the middle of capturing some item.

Your system's ratio of RAM to archive size will impact the frequency of this kind of thing a lot. If you have a lot of physical RAM, QRecall will allocate really big caches for many of these tables and indexes. That reduces the frequency that they need to be flushed, sorted, or read again from disk.

If not, do you have any idea what *is* happening?

Sometimes QRecall is just looking for the next thing to capture. QRecall displays the item that it's capturing, but once it's done it goes looking for the next thing to capture. Until it finds the next item, the status line continues to display the previous item. In the future I might change this so that a prolonged scan for the next item is a little more transparent.

James Bucanek wrote:Your system's ratio of RAM to archive size will impact the frequency of this kind of thing a lot. If you have a lot of physical RAM, QRecall will allocate really big caches for many of these tables and indexes. That reduces the frequency that they need to be flushed, sorted, or read again from disk.

I'll bet that's the problem. This is a MBPro with 4 GB RAM, but I'm nearly always (as I was today) running VMWare Fusion and Windows 7 in a 2 GB partition, so that leaves only 2 GB for everything else, including QRecall. I probably don't need a full 2 GB for the things I'm doing in Win 7, so I'll see if I can cut that back a bit and give QRecall some more room when it needs it. And I knew QRecall did occasional housekeeping before and after archives, but I wasn't sure if it also did that *during* archives. Now I know. Thanks! You might consider adding some kind of notification in the activity window just so we have a better idea of what's going on when it appears to stall.

Sometimes QRecall is just looking for the next thing to capture. QRecall displays the item that it's capturing, but once it's done it goes looking for the next thing to capture. Until it finds the next item, the status line continues to display the previous item. In the future I might change this so that a prolonged scan for the next item is a little more transparent.

For the times I've noticed it, that's probably not what's happening. After it finally started moving again today, I watched it go through the rest of my Applications folder, and the rest of my hard drive, pretty quickly.

-- Bruce

If you're really curious, and have nothing better to do, you can open a terminal window and use the iosnoop tool to see what QRecall is up to:

sudo iosnoop

My guess is that when it's stuck, it's writing to the hash_scribble.index file. A bunch of tiny writes to this file indicate that RAM has filled up with newly created (or previously cached) record entries that now must to be written to the disk before any new records can be added. This is probably the most time consuming, and I/O intensive, periodic chore that QRecall has to do. If the archive is on a network volume, this could be even worse (performance wise).

If you're running a virtual machine at the same time, there could also be some good old VM thrashing going on. QRecall bases its table and cache sizes on the physical RAM installed in the machine. There's no rational way to determine what the RAM demands of other processes are, so two or more RAM hungry tasks (which would definately include QRecall) could be fighting for resources.