I use Bare Bones's Yojimbo. There has been recent discussion on its list about using Time Machine to back up its database. A recent message from a Bare Bones person states:

> Before things get too far out of control, I want to clarify some
> facts about how Time Machine and Yojimbo.
>
> Yojimbo is built on CoreData, the same underlying technology as
> Aperture, and several other products. Because of issues related
> to how Time Machine and CoreData manage files on disk, Apple
> recommends excluding Aperture data from Time Machine backups,
> and managing Aperture backups independently:
>
> <http://docs.info.apple.com/article.html?artnum=306853>
>
> For the moment, we are recommending the same thing.

This leads me to wonder whether QRecall faces the same 'problem' as Time Machine does, and that it should therefore not be used to back up such data when a programme that uses CoreData is running.

In my case this is a hypothetical question as I have QRecall set to run when I am logged out. I imagine however that some users may want it to run while they are logged in.

-- Charles

Charles Watts-Jones wrote:This leads me to wonder whether QRecall faces the same 'problem' as Time Machine does, and that it should therefore not be used to back up such data when a programme that uses CoreData is running.

You assume correctly. You wouldn't want to capture your Yojimbo database while Yojimbo was running, or your Aperture database while Aperture was running.

The fundamental problem with CoreData, and most modern databases, is that the files in the database aren't always up-to-date while the database engine is using them. For speed, databases keep a lot of their changes in memory and only occasionally write those changes back to disk. If the files are copied while the database is in use, the files are an incomplete — and probably invalid — picture of the database.

Note that this problem affects other non-CoreData databases too. Entourage, MailSmith, and similar applications use their own database engines which have the same limitation.

In my case this is a hypothetical question as I have QRecall set to run when I am logged out. I imagine however that some users may want it to run while they are logged in.

Capturing while logged out is certainly the safest solution at the moment.

The problem isn't quite as sweeping as "being logged in" or "while CoreData is running" (technically, CoreData runs all the time). The problem should be limited to whether a particular database is in use. So quitting the application in question (Aperature, Yojimbo, Entourage, ...) should close the database and synchronize any changes with the files on disk. At that point, it's safe to capture that particular database for backup purposes.

There has already been a feature request for skipping captures while a particular application is running, or to start a capture whenever a particular application quits. So look for new action rules or filters that address this problem more directly in future versions of QRecall.

James Bucanek wrote:Capturing while logged out is certainly the safest solution at the moment.

And still utterly inconvenient for me as a method of ensuring integrity with any backup software. Sigh.

There has already been a feature request for skipping captures while a particular application is running, or to start a capture whenever a particular application quits. So look for new action rules or filters that address this problem more directly in future versions of QRecall.

It's definitely a tricky problem to completely resolve without being able to dynamically detect vulnerable conditions, but maybe eventually it becomes easier to identify candidate applications for special QR handling.

Sometimes I get the impression certain backup software developers aren't even aware of issues like this so I definitely appreciate that you are.

If you're lucky, maybe your program has an option to create a backup of its data store at timed intervals, or could be persuaded to do so via applescript and a cron job?

It does seem that timed-interval backups of an application data store is a capability that any app that relies on CoreData should have. I doubt many do.

steven