Pierpaolo,

QRecall 1.3 has a number of new scheduling features in the works that I think will address your situation. Specifically, a new event schedule that will run an action when an application launches or quits. This way, you can simply schedule your capture action to run whenever DevonThink quits.

The next version will also have some scripting options that I'm looking for users to evaluate.

A beta of QRecall 1.3 should be available for testing by early September.

pirem71 wrote:Another separate feature request: menu bar drop-down list is now displaying next scheduled action; it would be nice if it could show the scheduled time of that action (as done in Actions window).

This is already on the to-do list.

Ralph Strauch wrote:I had reselcted the archive in the action before I posted, so was pretty sure it wasn't that.

I feel compelled to point out that changing an action document after the action starts has no effect on the action that's already running.

If the action had been started and was reporting "Waiting for archive," the process for that action has already been created and won't be influenced by changes to the action's definition until the next time it runs. You would need to (1) change and save the action document, (2) stop the running action, (3) run the action again.

As Adrian pointed out, it's possible that the action is pointing to a different archive. QRecall uses OS X alias objects to track the location of archives, and they can be tricked into referring to the wrong file.

I suspect, however, that some other process has the file open for reading.

Browsing, verify, and recall actions access the archive using shared read permission. That means that any other process can be reading from the same archive, but no process is allowed to write to it.

Merge and capture actions access an archive using exclusive write permission. That means that no other process is allowed to access the file for reading or writing until the action is done.

If an action can't obtain the access permission it needs, it displays the "Waiting for archive" message and ... waits.

To find out what other processes are accessing your files, open a Terminal window and issue the following command:

sudo lsof | fgrep 'FragmentOfArchiveName'

The lsof tool "lists the open files" on your system and the process name that has them open.

Here's an example from my system:

sudo lsof | fgrep 'Teacup'

QRecallHe 6548          james  txt       REG              14,19     67108868       8798 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/negative.index

QRecallHe 6548          james    5u      REG              14,19 155538852120       8806 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/repository.data

QRecallHe 6548          james    6r      REG              14,19     56868912       8802 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/package.index

QRecallHe 6548          james    7r      REG              14,19     75446504       8793 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/filename.index

QRecallHe 6548          james    8r      REG              14,19        25944       8792 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/displayname.index

QRecallHe 6548          james    9r      REG              14,19    402653240       8795 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/hash.index

QRecallHe 6548          james   10r      REG              14,19     67108868       8798 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/negative.index

QRecallHe 6548          james   11r      REG              14,19        83438       8796 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/hash_adjunct.index

This is telling me that a QRecallHelper process has a bunch of files open inside my Teacup.quanta archive bundle. This is normal, as the QRecallHelper was performing a verify of the archive at the time.

One all-too-common scenario is when an archive is being shared using file sharing. A process on a remote computer opens a file, and then the connection (ethernet or Wi-Fi) is lost. The Apple File Server process keeps the local file open, thinking that the remote computer still has the file open. When this happens, the lsof command will tell you that the AppleFile process has the file open.

The solution is to simply restart the computer sharing the file (closing all open files), or to stop and then restart the file server (again, causing the file server to first close all files). The later can be done in the Sharing preferences pane.

Steven M. Alper wrote:Loving the interface changes. It's really looking beautiful.

Thanks!

I seem to recall seeing something in the notes about the last update that changes were made to make QRecall OS 10.8 compatible. Have you tested it under the release version?

Yes. It's been tested under the Mountain Lion golden master. Among other tests, QRecall was used to capture a Mountain Lion boot volume. That volume was then erased and QRecall was used to restore the entire volume; the restored volume successfully booted and runs without any known issues.

The biggest change made was for compatibility with Mountain Lion's new Gatekeeper system. QRecall is now digitally signed with a developer ID certificate issued by Apple, so it will be recognized as an application from an "known" developer.

If it's all looking good, I suggest you add QRecall the the compatibility chart at http://roaringapps.com/apps:table . It doesn't appear on the list (even for 10.7 compatibility).

Thanks for the tip. I'll look into it.

Ralph,

This is a known problem. It's a message race condition between the action and the application or monitor window that's keeping an eye on that action.

You've already checked the log, but I'll repeat this advice for anyone else reading. If you get a "lost communications" error it just means that the user interface lost it's communications link with the running action. It might mean something horrible has happened (like the action has crashed), or it might just mean the link between to two went dead. The first thing to do is to look at the log and find the action. It the log for that action says it finished (as in "Capture finished") there's nothing to worry about.

This loss of communications between the UI and the running action has been popping up since Snow Leopard, and seems to be worse in Lion. (Apple has been getting agressive about conserving power when the computer is idle, but sometimes things are so quiet now the monitor thinks the action has died.)

I've recently changed some code to address this issue and it seems to be helping. I haven't recorded a single "lost communication" error in the past week on any of my test machines.

If the fix is stable, it will roll out with version 1.2.1, which should be released within a week.

Adrian Chapman wrote:I'm not suggesting that QRecall is at fault per ce, but I wonder if there is a connection somewhere.

Very likely. The compact action moves a lot of data. In fact, it will typically rewrite almost every block in the archive. Contrast that with a typical capture or merge action, which might modify no more than one ten thousandth of the archive.

If you have cross-linked files, the compact action will be the one to expose it.

Tellan,

If I understand your question correctly, QRecall does not skip items during a recall. It will skip over items that it is reasonably sure have not been modified during a capture action, but the same rules can't be applied during a recall action.

When restoring items, QRecall replaces items in their entirety with the version captured in the archive. To improve performance, it will read existing items and compare their data with the version in the archive. The existing item will be overwritten only if it is different, but the ultimate result is that every items on the destination volume will either be rewritten or compared in its entirety.

During a recall, QRecall does not skip items that merely appear to be the same. An existing item's metadata is insufficient to guarantee that the item is identical to the version stored in the archive. The only way to be sure that the restored items are identical to their captured versions is to write, overwrite, or compare all of them.

Adrian Chapman wrote:Unfortunately the Drobo-FS has its own internal file system and from what little I know, the method of checking can be destructive of data! Great!

From the sounds of this Drobo support article you might be able to fix it with OS X.

Just a follow up:

While reviewing the rest of your diagnostic report, I noticed this error during your last verify action:

2012-06-30 14:07:41.546 Failed

2012-06-30 14:07:41.546 cannot read file

2012-06-30 14:07:41.546 Error: I/O error (bummers)

2012-06-30 14:07:41.546 Length: 524288

2012-06-30 14:07:41.547 Pos: 242321696144

2012-06-30 14:07:41.547 File: repository.data

An I/O error means the drive was physically unable to read something from the drive.

It could mean hardware problems, or could also be caused by a corrupted volume structure (a file position get translated into a track/sector that doesn't exist and the drive responds with an I/O error). Because of the ambiguity of I/O errors, it doesn't really narrow down what the exact problem is. But it is one more piece of evidence that points to a problem with the NAS.

Adrian,

Thanks for posting your problem and for the diagnostic report.

My analysis is that you're having data integrity problems with your NAS drive. My first guess would be cross-linked files, which can cause writes into one file to pollute a different file with unrelated data.

The first problem was encountered on 2012-06-28 14:08 when the verified failed:

2012-06-28 15:44:17.686 Archive contains unused space that has not been erased

2012-06-28 15:49:13.313 Pos: 245397938824

2012-06-28 15:49:13.313 Length: 1555152

Under normal circumstances, QRecall fills each unused area of the archive with zeros. It then wraps a small record header around each area to mark where it begins and how long it is.

During the verify, QRecall confirms that all unused areas of the archive are, indeed, still filled with zeros. In your case, there was something other than zeros in that region of the archive. This indicates that some other process has written data into that region of your archive or the sectors that store that data have become damaged or corrupted.

Subsequent attempts to repair the archive show similar problems, but this time the damage was in regions of the archive that contained real data:

2012-06-28 17:53:37.933 2464 bytes at 248688532320

2012-06-28 17:54:20.271 13601632 bytes at 250084036664

2012-06-28 17:55:19.349 32744 bytes at 252668562552

2012-06-28 17:55:19.349 16 bytes at 252668595320

2012-06-28 17:58:06.164 4104 bytes at 259718257104

The next repair reported more lost data:

2012-06-28 21:00:20.971 16400 bytes at 245687567768

2012-06-28 21:00:53.052 20280 bytes at 246963311616

2012-06-28 21:00:53.395 2664 bytes at 246963332080

The really important thing to note here is that the file positions in the second report are at positions before the ones in the first report. If the data on the drive was stable, the first repair would have reported these errors too. Instead, the first repair read this area of the archive and found it to be sound, but the next time your ran the repair the previously sound area now has data errors.

There are two explanations for this.

The drive system could simply be dying a slow death, and is randomly scrambling or corrupting data.

More likely is the volume structure of the NAS is damaged. If the allocation map is invalid, it could cause newly written data (i.e. the index files that get recreated during the repair) to inadvertently overwrite perfectly good data in the primary archive data file. In effect, the repair is destroying the archive.

The solution to the later problem is to use a disk repair utility to ensure the volume structure of the NAS is OK.

As you continued to run more repair actions, QRecall continued to find new area of your archive that was damaged. A damage to a few records can indirectly impact the contents of multiple layers, so it won't take long before most of the layers in your archive are affected by at least one of these problems. You can review the repair log to discover exactly which files/folders were affected (in most cases, it was just a few files that were lost during each repair).

David Ramsey wrote:Something tells me it's not an SQL-type relational with BLOBs...

No, it's not.

Conceptually, it could be. QRecall's archive is organized very much like a database. The structure, however, is a custom one, designed for speed and efficiency in performing the billions of lookups need to recapture files. It's also designed to detect and survive random data loss caused by inadvertent writes, data corruption, or media failure. The latter requirement is why QRecall isn't based on an existing database engine.

David Ramsey wrote:I suppose I can simply delete the lower archive dated 5/14...

Select both owners and choose Combine Items....

David,

Here are some thoughts:

Did you change your identity key or replace/reformatted a drive around May 19?

Go to the very top level of your archive and make sure you only have one owner and one volume. An "owner" is defined by an identity key. Everything you capture belongs to that owner. If you change your identity key, a new owner will appear at the top level of your archive and all items captured after that will belong to the new owner. The same thing can happen to volumes if you replace/repartition a hard drive. If that's the case, you can "fix" this situation with the Combine Items... command. It stitches together the history of two owners or two volumes which actually represent the same item.

Also double check your capture actions to make sure they are capturing the right things. Actions keep track of the items they capture using OS X aliases, which are very smart but can occasionally be fooled into referencing the wrong item. If that's the case, simply remove the incorrect items and re-add the items you want to capture.

If you still think that items aren't getting captured, or are getting captured but not showing up, post again and we can dig a little deeper.

David Ramsey wrote:I can recall items from either of these layers, but can't find a way to expose later layers. Pulling the shade control lets me exclude the latest layer, but that's it.

Obviously I am missing something terribly obvious.

It's not obvious at all, David, or I'm sure you would have found it.

Excellent question. The answer is that QRecall is "helping" you by automatically hiding layers unrelated to the items you're currently browsing.

There are two ways to see the other layers:

Go browse a different folder, or better yet, a volume.

Go to the View menu and check Show All Layers. You should now see all of the layers in your archive, and you'll notice that all but two are grey.

When you browse a particular folder, QRecall finds all of the layers that contain copies of items in that folder. The folder that you happen to be browsing contains items that were captured in two layers. None of those items have been recaptured in any other layers. The remaining layers are dimmed, to indicate that they don't contain anything related to the items in that folder; or if you happen to have the Show All Layers unchecked, they are hidden altogether.

The idea is that when you browse a particular folder, the layers list and the graphic view on the left are trimmed down to just those layers that are relevant to the items you're looking at.

Neil Lee wrote:I've got a really large archive (500G+) that I'm trying to open to restore a file from, and when I open it in QRecall the app just hangs. I assume it's crunching away at something but there's no UI, no window, and no feedback telling me as such.

Also, I don't know how long it should take for QRecall to open an archive this size, but it's been sitting there for 20 minutes now with no sign of activity.

Lee,

Something is wrong. You should, at the very least, see an empty window while the archive is opening. If the application is locked up, then it probably locked up before it started to open the archive.

Please do this for me: Open the Terminal application, and issue the following command.

sudo /Applications/QRecall.app/Contents/Resources/sample.sh

(If your QRecall application is somewhere other than the /Applications folder, then adjust the path accordingly.)
Enter your admin password and wait for the script to finish.
Force Quit the QRecall application. Do this from the Dock, via Command+Option+Esc, or using Activity Monitor.
Launch QRecall again and see if it launches on its own. If so, send a diagnostic report.

Could there be some kind of dialogue window added so it's obvious what QRecall is actually doing?

Absolutely, and that's on the to-do list.

Having said that, however, this isn't your problem. When opening an archive, you should immediately see the archive window, and the details will fill out as the initial data is loaded. Even on a very large archive on a relatively slow mediam (i.e. wireless), it still shouldn't take more than 30 seconds before you can start browsing. I just opened a 5 TB archive on my system. The window appeared in approximately 2 seconds, and the browser had loaded and was ready to go in 10. This archive stores data for 4 computers, 10 volumes, and has over 30 million items in over a hundred layers. Admittedly it's on a very fast eSATA RAID-5, but you get the picture.