i've just run my first backup of my startup drive to my NAS. the backup was 47.5 GB in size. the progress bar is still on the screen and the little spinny-roundy thing still spinning round, and beneath the progress bar is the status message, "Closing archive." this has been the state of things for five or six (or more) hours now. so my questions are, what is "closing" an archive, and how long should i expect it to take? do i wait still more hours, or is it time to force-quit QRecall?

W. Darson wrote:ow. so my questions are, what is "closing" an archive

Closing an archive encompasses a number of tasks. Mostly it involves updating the archive's various indexes.

and how long should i expect it to take?

That's a hard one to answer. The short answer is "it takes at long as it takes."

In your instance, I suspect that QRecall is preoccupied updating the quanta index. This is the largest index and, since you just captured 47GB of new data, it has a lot of new data to index. Unlike the actual capture phase, updating the quanta index involves millions (in your case hundreds of millions) of tiny reads and writes.

When capturing to a local hard disk, I'd expect a capture of 47GB to take another 10 minutes or so to index all the new data. I don't know anything about your network or your NAS drive, but how fast it can process these small read and write requests will substantially impact the speed at which it will take to finish the index. For example, if your network+NAS combination were working at 1/20 the speed of a local hard drive it would take 3-4 hours to finish indexing. If it works at 1/100 the speed (which is not unheard of) it could take 15-20 hours.

Start up the Activity Monitor and see if QRecall is working. You should see substantial network activity (lots and lots of small packets). If you do, I'd suggest letting QRecall continue working. If not, let me know and we can look into the situation in more detail.

The good news is that this doesn't have to be done again. On the next capture only the newly added data will added to the quanta index, and will be much faster.

thank you, James. i actually got impatient and force-quit QRecall before i got your reply and have started the whole backup process over, but when this one gets to the "closing archive" stage i will watch it in Activity Monitor and let you know if QRecall's just sitting idle.

regards,

Walt

(An old thread, I know )

I'm having essentially the same question - what is done during a "Closing archive". I have a huge archive (670 GB), holding the backups of two complete machines, living on a NAS.

E.g. just now, I had the QRecall Activity window telling me:
"Closing archive
1.41 MB, 29% duplicate
51 items, 412 KB/min"

with the counter running for 45 minutes in the "Closing archive" state; capturing itself (according to the log entries) took about 7 minutes (incl. opening archive from the NAS, locating local changes and writing them).

I watched, as you said, for lots of small network action, however did not find that to occur: on average maybe 1 to 2 packets/s. I tried to sample QRecallHelper from the Acticity window, but that failed due to some unknown error. I then turned to Instruments, and found that:

1. any significant reads/writes only happened in the last minute or so
2. the QRecallHelper process took 100% of CPU usage all the time
3. the QRecallHelper process took about 700 MB of physical memory

From Instruments, I was able to do some sort of sampling, and most of QRecallHelper's threads were probably waiting on some semaphore and handling the AutoReleasePool or something like that (I'm no ObjC programmer), while there was only one with symbols starting with NSHash... that probably indicates that the time goes into hash table management (updates, finding free buckets etc.), but not into network transfers as you described earlier in this thread for an initial capture.

Is this expected behaviour? Is there anything I could do to speed up the Closing Archive stage? Could that operation be parallelized somehow (probably not if all operations need to be performed on the same hashtable)? Mind you, the captures do succeed - I'm just curious what happens at that stage - it seems to be completely work done in-memory.

You can (temporarily...) download an Instruments document (10 MB) of the last 15 minutes of the above "Closing Archive" operation in case it might be of interest from here.

Regards, Christian

Christian,

Standard disclaimer: There's a lot that happens when an archive is being closed. How long will depend on how many new quanta were added to the archive, how many unused data blocks (aka "holes") are in the archive, the speed of your data connection with the archive, and so on. It could take 10 seconds or 10 minutes.

However, I sheepishly admit that you're probably encountering a serious problem with the names index that's present in the current release.

As the names index grows in size—and it sounds like you're is larger than most—the amount of time spent reading, updating, and writing the names table becomes enormous. I've seen QRecall sit and spin for 40 minutes just sorting the names index.

For the most part I've fixed this already. The new names index routines are anywhere from 10 to 60 times faster than the current release version. Unfortunately, there's another bug in the names index that I really need to get fixed before I release a new version, and I just haven't had the time to do that yet.

If you want to verify this, you can use the 'sample' tool to sample the QRecallHelper process when it gets stuck closing the archive. You'll have to run it as root (i.e. sudo sample ...) if QRecall is pre-authorized to use administrative privileges. Send me the sample file and I can tell if it's the same problem.

If this is causing you a great deal of grief, you're welcome to try—at your own risk—the unreleased QRecall 1.1.1 that I've given to a few users experiencing other problems, also fixed in this version.

Thanks James for the insight. No, it doesn't cause me too much of a grief - I just wanted to know I had a few times where I couldn't leave my machine awake until the closing was finished before putting it to sleep, and I think after waking it up again several hours later the capture did not always complete successfully. But there may have been other factors involved, so I'm not sure. It's just that I do not feel confident putting my machine to sleep while there is still a capture process going on.

I can do a sample next time with sudo, but I think the Instruments (an app included in Apple Dev Tools) file I linked to in my last post should contain already one, if you want to have a look at it right away. Just double-click the file and in Instruments, click on the "Sampler" instruments lane, then expand the first thread (0xde03).

I'll just wait for the new release, since the archive in question is my actual working archive which I do not want to lose/corrupt.

Regards, Christian

Christian Roth wrote:I think the Instruments (an app included in Apple Dev Tools) file I linked to in my last post should contain already one, if you want to have a look at it right away.

I downloaded your instruments file and took a look. Sure enough, nearly 100% of the first thread's time is spend in NSHashInsertIfAbsent, which is the function that's called to add one entry to the name index hash table.

I'll just wait for the new release, since the archive in question is my actual working archive which I do not want to lose/corrupt.

I'll keep you posted. Hopefully, I'll have some time soon to squash that lingering bug and get a maintenance release out.