CaB Studios wrote:Am I missing something?

You're probably missing detail.

Move the detail slider in the log window to the right. My guess is the missing actions will appear in the log.

QRecall only captures changes. If there are no changes, no new layer is added to the archive. Also, the log messages for a capture action that didn't capture anything are demoted to "minutia" so it doesn't down out the more significant log messages about things that did happen.

Someone just pointed out to me that there's a couple of companion programs that extend the utility of Growl notifications. Prowl will forward Growl notifications to your iOS device.

And a new program, Bark, will redirect Growl notifications to Mountain Lion's notification center. Native Mountain Lion notification center integration is planned for an upcoming release of QRecall.

CaB Studios wrote:Been testing QR out over the last 24hrs and I'm blown away with the speed and interface of the software. Remarkable job!

We're pleased to hear that.

One feature which is lacking (or at least I cannot see it) is the ability to email an admin once an archive has completed.

In my previous job I used Areca, an open source bit of software which was slow compared with yours but did the job. It had the ability to email me when a job was complete and I'm kinda missing that at the moment.

Is there a chance this can be implemented?

E-mail notification isn't implemented for a number of reasons. From a technical standpoint, it's actually very awkward. The QRecall helper process runs in the background, as a root process, even while you're logged out. In this execution environment it doesn't have access to any of the built-in frameworks for sending mail messages. That means that the SMTP engine would have to be built into the process, which presents a whole new set of problems.

QRecall does offer a number of notification mechanisms, most of them focused on alerting you to any issues that may have arisen. The QRecall menu item shows you when actions are in progress. Any failure will persist in both the monitor and archive status windows, and the QRecall menu item's icon will change to make that more obvious.

QRecall is also integrated with Growl. Separate Growl messages for successful and unsuccessful completion allow you to configure success and failure notifications any way you like.

The archive status window will also alert you when no capture or verify action has run for a period of time, which covers the case where these actions aren't running at all for some reason. The status window state is persistent, so it will note problems even if those problems occurred while you were logged out.

Chris,

Please send a diagnostic report (Help > Send Report...). There's clearly something unusual going on, and the details of the failure will help.

Also, please describe the volume you're storing the QRecall archive on. Size, Internal or external drive, directly connected or NAS or file server, and most importantly how is the volume formatted?

Chris Petit wrote:I was curious how verifying worked.

In practical terms, it works by making sure that nothing in the archive has been altered or is missing. It doesn't need the original files to accomplish this.

In technical terms, a verify action reads and checks the integrity of every record in the archive. To understand why this works, you need to know a little about how an archive is constructed.

An archive is made up of a multitude of small records written to a file called the repository. Every record is stored with an Adler 32-bit checksum along with some redundant information. QRecall can use this information to determine if any data in that record has been disturbed or altered, even a little bit. Even empty records in an archive have checksums that can tell QRecall if something has been written where it shouldn't.

The first phase of a verify action consists mostly of reading every record in the repository and verifying those checksums. All of the information about a captured file (what directory it is in, its metadata, and every block of its data) are stored as individual, verifiable, records. So if all of the records are intact, QRecall can prove that none of the folders, files, or any of the data in those files, has been altered or lost.

The rest of the verify action is spent doing two things. The first is to make sure the structure of the archive is sound. For example, records refer to other records (a directory record may contain a list of records that represent the files in that folder). This is mostly a sanity check, as it's highly improbable that all of the records would be OK while a link between records is wrong. But it's not impossible, so QRecall checks.

The other thing the verify action is doing is making sure that all of the auxiliary index files in the archive package agree with the information in the repository records. The individual records of the repository are very robust and designed to be resistent to data loss, but are cumbersome to access quickly. The index files make finding and updating information in the repository much quicker, but they're only good if they accurately reflect what's in the repository. QRecall reconstructs all of the information in the index files from the information in the repository records, and then compares that to the files in the archive package. If they match, the index files are given a clean bill of health.

And that's verifying in a nutshell.

Adam Horne wrote:I want to copy my archive over to HD3 a few times a week. It seems a little dangerous to make an archive of an archive, but what do you recommend?

I don't know about "dangerous"?I have a couple of archives that I archive regularly. I would describe it as "cumbersome" because to recover something one must first recall the archive, then open that archive to recall the files.

The fundamental problem you're going to run into is that there's no way to sync a large file without reading the entire file and/or making a copy of the entire file, every time you sync. It's the nature of synchronization programs.

Most synchronization programs performs syncs in one of two ways. The simple way is to compare the two files (by looking at their modification date or actually reading both files) and then duplicating one of them if they are different. The end result is a copy of the entire file.

More sophisticated ones (like rsync) read the data blocks of each file, generate a "fingerprint" (checksum) for each block, and then compare those fingerprints. If they match, the blocks are the same. If not, the non-matching blocks are copied. This might seem to be terribly efficient, but most implementations work by first making a duplicate of the original file before modifying it, and then deleting the original. This is the safe way to synchronize the two, but doesn't save any data copying. The result is that the entire file still gets copied every time, and also requires that you have twice the amount of disk space as the file.

Making an archive of the archive is actually the most efficient. QRecall uses technology similar to advanced synchronization programs, but in a different way. The archive database already stores the "fingerprints" of all the data blocks already in the archive. So it's possible to capture the blocks that haven't changed without having to physically read or duplicate the unchanged data. Only the changed blocks need to be copied, and that's exactly what QRecall does.

My advice is not to do either. If it's a drive connected to the system you're backing up, simply create a second set of actions to back up your same items, using a much more leisurely (once/twice a week) schedule. You don't gain anything in terms of security or efficiency trying to backup the backup, this method is easy to recover from, it's just as safe as the "daily" archive, and it's ultimately more efficient.

Pierpaolo,

That's a situation I certainly didn't anticipate.

Currently, there's no mechanism for ignoring the capture status of a single archive. You can forget the archive in the status window (right-click > Forget), but it will reappear in the status window the next time you verify it.

I'll have to give some thought to a way of dealing with this kind of situation in the future. For now, you can create a "dummy" capture action the runs, say, once a week and captures a single item that's not likely to ever change. It doesn't matter what it is, but as long as it doesn't change the archive will never be updated. But it will update the last-captured status of the archive, and make that annoying red indicator go away.

Adam Horne wrote:I've used QRecall for a while on my iMac. Is there anyway to continue to use that archive?

Absolutely. QRecall archives are designed to store items for multiple volumes from multiple systems.

Simply set up your new computer to capture to the same archive. You can repurpose your existing QRecall actions simply by selecting a new target archive location in the your action documents. If you're setting things up from scratch, just choose the existing archive when you run the capture assistent on your new system.

A new volume will appear in the archive. Navigate to that volume to browse or recall items on your new computer. If you're using a different identity key, that volume will appear under the new owner.

Much of my files and folders were a copy over onto the new Mac Mini.

Since most of your documents have already been captured in the archive, capturing your new system won't cause the archive to grow much, since the majority of the data will be duplicate.

It would be good cool if I could continue to add to the archive and just add the new actions for what we discussed.

QRecall is very cool.

Adam,

Here's my two cents:

Adam Horne wrote:I've recently purchased a new mac mini with a 256 GB solid state drive.

Congratulations.

For all the other files, I've purchased a 4TB drive that I've set to Raid 0 for speed.

I assume that you mean two 2TB drives that you've RAID'd together.

I also bought a 3TB drive that I plan to use a backup for selected files on my raid drive.

I was thinking of partitioning my 3TB drive into 2 sections (1) being 256GB and (1) being the rest of the HD and having SuperDuper create a clone of my mac mini drive on the 256GB drive.

Mirroring of your 256GB SSD to a second partition is, by itself, a poor backup strategy. There are excellent reasons for mirroring your startup volume, but as a backup it's not robust. You eventually end up with two scenarios:

You mirror your SSD automatically (say, daily). If something gets messed up and you don't discover it before the next mirror, you just end up with two copies of an OS that's messed up.

You mirror manually. This will eventually result in a mirror that's way out of date with your startup volume. If you mess up your startup volume, your backup is now weeks or months out of date.

With that in mind, here's my suggested setup:

Configuration
Set up QRecall to capture everything (your entire SSD and your media volume to your backup partition) to a single archive. I can't think of any reason for creating two archives.

The OS will actually be a small part of that archive, but will give you the ability to regress back to any recent version of your OS, applications, mail, preference files, etc. (not just yesterday's or last month's).

Space management
Since you're capturing (potentially) 4.25TB into a 2.75TB volume, you run the risk of filling it up quickly.

You can enable compression if space on the backup partition starts to get tight.

You could exclude non-critical media files (like music or movies that you have backed up on CD/DVD elsewhere).

Pierpaolo,

QRecall 1.3 has a number of new scheduling features in the works that I think will address your situation. Specifically, a new event schedule that will run an action when an application launches or quits. This way, you can simply schedule your capture action to run whenever DevonThink quits.

The next version will also have some scripting options that I'm looking for users to evaluate.

A beta of QRecall 1.3 should be available for testing by early September.

pirem71 wrote:Another separate feature request: menu bar drop-down list is now displaying next scheduled action; it would be nice if it could show the scheduled time of that action (as done in Actions window).

This is already on the to-do list.

Ralph Strauch wrote:I had reselcted the archive in the action before I posted, so was pretty sure it wasn't that.

I feel compelled to point out that changing an action document after the action starts has no effect on the action that's already running.

If the action had been started and was reporting "Waiting for archive," the process for that action has already been created and won't be influenced by changes to the action's definition until the next time it runs. You would need to (1) change and save the action document, (2) stop the running action, (3) run the action again.

As Adrian pointed out, it's possible that the action is pointing to a different archive. QRecall uses OS X alias objects to track the location of archives, and they can be tricked into referring to the wrong file.

I suspect, however, that some other process has the file open for reading.

Browsing, verify, and recall actions access the archive using shared read permission. That means that any other process can be reading from the same archive, but no process is allowed to write to it.

Merge and capture actions access an archive using exclusive write permission. That means that no other process is allowed to access the file for reading or writing until the action is done.

If an action can't obtain the access permission it needs, it displays the "Waiting for archive" message and ... waits.

To find out what other processes are accessing your files, open a Terminal window and issue the following command:

sudo lsof | fgrep 'FragmentOfArchiveName'

The lsof tool "lists the open files" on your system and the process name that has them open.

Here's an example from my system:

sudo lsof | fgrep 'Teacup'

QRecallHe 6548          james  txt       REG              14,19     67108868       8798 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/negative.index

QRecallHe 6548          james    5u      REG              14,19 155538852120       8806 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/repository.data

QRecallHe 6548          james    6r      REG              14,19     56868912       8802 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/package.index

QRecallHe 6548          james    7r      REG              14,19     75446504       8793 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/filename.index

QRecallHe 6548          james    8r      REG              14,19        25944       8792 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/displayname.index

QRecallHe 6548          james    9r      REG              14,19    402653240       8795 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/hash.index

QRecallHe 6548          james   10r      REG              14,19     67108868       8798 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/negative.index

QRecallHe 6548          james   11r      REG              14,19        83438       8796 /Volumes/Warehouse 13/Backups/Red King/Teacup.quanta/hash_adjunct.index

This is telling me that a QRecallHelper process has a bunch of files open inside my Teacup.quanta archive bundle. This is normal, as the QRecallHelper was performing a verify of the archive at the time.

One all-too-common scenario is when an archive is being shared using file sharing. A process on a remote computer opens a file, and then the connection (ethernet or Wi-Fi) is lost. The Apple File Server process keeps the local file open, thinking that the remote computer still has the file open. When this happens, the lsof command will tell you that the AppleFile process has the file open.

The solution is to simply restart the computer sharing the file (closing all open files), or to stop and then restart the file server (again, causing the file server to first close all files). The later can be done in the Sharing preferences pane.

Steven M. Alper wrote:Loving the interface changes. It's really looking beautiful.

Thanks!

I seem to recall seeing something in the notes about the last update that changes were made to make QRecall OS 10.8 compatible. Have you tested it under the release version?

Yes. It's been tested under the Mountain Lion golden master. Among other tests, QRecall was used to capture a Mountain Lion boot volume. That volume was then erased and QRecall was used to restore the entire volume; the restored volume successfully booted and runs without any known issues.

The biggest change made was for compatibility with Mountain Lion's new Gatekeeper system. QRecall is now digitally signed with a developer ID certificate issued by Apple, so it will be recognized as an application from an "known" developer.

If it's all looking good, I suggest you add QRecall the the compatibility chart at http://roaringapps.com/apps:table . It doesn't appear on the list (even for 10.7 compatibility).

Thanks for the tip. I'll look into it.

Ralph,

This is a known problem. It's a message race condition between the action and the application or monitor window that's keeping an eye on that action.

You've already checked the log, but I'll repeat this advice for anyone else reading. If you get a "lost communications" error it just means that the user interface lost it's communications link with the running action. It might mean something horrible has happened (like the action has crashed), or it might just mean the link between to two went dead. The first thing to do is to look at the log and find the action. It the log for that action says it finished (as in "Capture finished") there's nothing to worry about.

This loss of communications between the UI and the running action has been popping up since Snow Leopard, and seems to be worse in Lion. (Apple has been getting agressive about conserving power when the computer is idle, but sometimes things are so quiet now the monitor thinks the action has died.)

I've recently changed some code to address this issue and it seems to be helping. I haven't recorded a single "lost communication" error in the past week on any of my test machines.

If the fix is stable, it will roll out with version 1.2.1, which should be released within a week.

Adrian Chapman wrote:I'm not suggesting that QRecall is at fault per ce, but I wonder if there is a connection somewhere.

Very likely. The compact action moves a lot of data. In fact, it will typically rewrite almost every block in the archive. Contrast that with a typical capture or merge action, which might modify no more than one ten thousandth of the archive.

If you have cross-linked files, the compact action will be the one to expose it.