Yes, there are internal identifiers, but you shouldn't have to worry about it (too much).

Each archive is associated with a unique ID, which is used to monitor its status. When you duplicate your archive, you'll have two archives with the same ID. At least, for a while...

The next time you do something with the duplicate archive (verify, capture, whatever), just make sure the original archive is mounted at the same time. QRecall will see that you have two archives with identical IDs and will automatically assign your second archive a new ID.

If, for some reason, you *never* have both archives mounted at the same time, that could confuse QRecall. To fix that situation, simply control/right+click on the duplicate archive in the Finger, choose Show Package Contents, and trash its status.plist file. QRecall will create a new one, with a new ID, and all will be right with the world.

For anyone wondering, Mark and I went back and forth, over the course of several days, trying to figure this out. Mark was really helpful in getting to the root of this problem, and I thank him for his persistence.

It turns out, Mark had relocated his home folder to a different volume and that volume was set to ignore ownership and permissions. Apparently, new security policies in OS X 10.10 don't allow the launchd service (the part of OS X that keeps background processes running, among other things) to start a background agent process if the path to that agent executable passes through a symbolic link that's on a volume that doesn't enforce ownership and permissions.

I know, right?

Enabling ownership for the home folder volume fixed the problem?and is probably a good idea anyway. I'll add some code so that future versions of QRecall will warn unsuspecting users about this situation.

Ming-Li

Thanks for taking the time to detail the differences you found between QRecall and Time Machine. I've seen similar discrepancies here, which reinforces my suspicion and Time Machine has its own internal set of exclusion rules, beyond those flagged by the system's "do not backup" list.

I can also appreciate Time Machine's aversion to backing up log files and databases. Without QRecall's block-level de-duplciation, these would quickly consume all of the available backup storage.

Luckily, we don't have that problem.

Mark,

Thanks for posting your problem. (I also got your message via email, but I'll respond here.)

First, try sending a diagnostic report from the desktop computer you're having problems with (in the QRecall app, choose Help > Send Report...). Also open up your Console app. In the system.log, filter the output with "qrecall", copy anything that pops up, and send those to me too, either via email or in the comments of your diagnostic report.

I suspect, however, that your problem is permissions. The issues you describe ("scheduler service could not be contacted" and "Show in menu bar" not working) all depend on background processes. QRecall installs several, but the two key ones are the QRecallScheduler and the QRecallMonitor.

These processes are installed and registered with OS X's launchd service in your ~/Library/LaunchAgents/ and /Library/LaunchDaemons/ folders. For security reasons, launchd is extremely picky about the location, ownership, and permissions of these folders. If they are anything except what launchd expects, it will refuse to launch anything installed there. (launchd will complain about this in the system.log, which is why I had you look there.)

My first suggestion would be to run the "Repair Disk Permissions" function in the Disk Utilities app, launch QRecall again (to reinstall), and then restart your system. If your QRecall monitor appears in the menu bar, I suspect your troubles are behind you.

If that doesn't fix it, write/post again and we'll get more specific in the investigation.

Ming-Li,

I'm surprised too, but I'm beginning to suspect that Time Machine has some exclusion rules hard-wired, and those aren't reflected in the set of folder it claims to exclude.

Sadly, your version of Recall also has a bug that fails to exclude the /System/Library/Caches folder when you select the "Exclude caches" option. (This is fixed in a later version, which also correctly excludes any networked cache folders too.)

The manual exclusion you added will work just fine for now.

Thanks for bringing this to my attention.

Norbert Karls wrote:May I follow up on this asking if there has been any progress already?

I?m pleased to write that QRecall development is?once again, thankfully, finally?in high gear.

I'm putting the final touches on the command line tool, which does a lot more than I orignally planned.

I have some new scheduling logic/features that I want to get working before I release it as a beta, along with some massive UI code clean-up to do.

There are also a bunch of random problems, glitches, and crashes that I?m still hunting down. Consequences of all that new code. *sigh*

And I have more testing to do. QRecall needs to be really stable before I spring the next version on my loyal users. This version will NOT be backwards compatible with earlier versions. Once you capture new data using 2.0, you can?t open that archive with an earlier version. So early adopators need to know they?ll be flying without a safety net...

Ralph Strauch wrote:After the repair I ended up with two layers marked "damaged" and one marked "incomplete,?

An ?incomplete? layer is one where the capture was interrupted (stopped/canceled) leaving some items uncaptured. This is mostly a warning that the items in that layer don?t fully include all of the changed items at that time. If you subseqently merge this layer with a more recent (complete) layer, this warning goes away becuase the more recent layer will contain the most recent items—and all of them.

A ?damaged? layer occurs when the repair action finds missing or inconsistent information about files or folder in that layer. When you merge it with a more recent layer, QRecall looks for successfully captured items in the more recent layer that supersede the damaged one in the earlier layer. If all of the damaged folders and items have been successfully recaptured, then the ?damaged? status should go away. If there are still any files or folder that QRecall is not satisfied have been completely recaptured, the merged layer retains the ?damaged? status.

Now I say ?should? because I think there?s a bug in this logic. QRecall is very conservative in how it determines that items have been successfully recaptured. If there?s any doubt, it retains the ?damaged? status on those items, and that layer. In my testing here, I?ve never been able to trick QRecall into thinking that a layer is still ?damaged? after all of the suspect items have been recaptued, but occationally users report that this is the case. (This has been a difficult issue to verify and debug becuase the information I need to verify the bug has to be gathered before the merge is performed, and no one reports this issue until after the merge.)

If you have a persistent ?damaged? layer, review the log for the repair action. It will identify exactly which folders and files it found problems with. After your next capture, simply browse the archive and verify that those items were successfully recaptured in the latest layer. If they were, you're in fine shape.

After my uncompleted Compact action, the unused space shown in the Status window jumped from less than 10mb to 106gb. If I understand Compact correctly, that was space that had been occupied by previously deleted files which the compaction process had erased, leaving scattered free space throughout the archive, and if the process had completed properly, those spaces would have been closed up by shifting everything to fill them in. Currently, though, qrecall simply uses that as available space within the archive. If this is correct, is there anything to be gained by running Compact again to close up those spaces, or is it just as efficient to simply let Qrecall fill them as over time?

That?s all true. QRecall will reuse that scattered free space for new captures. But just like volume fragmentation, the process eventually results in thounds, even millions, of tiny little blocks of space that can?t be effectively reused. These eventually become a drag on almost every action, since QRecall has to keep track of them all but can?t do much with them. Once you get your drive-unmounting issue resolved, I?d suggest taking another shot at compacting the archive, espeically with a 106GB free.

Ralph,

You can confirm the integrity of the archive by performing a verify. If the verify doesn?t complain, then everything in the archive is valid.

During the repair, however, you might have lost specific files or folders. If your drive is spontaneously disconnecting, then the archive probably has missing or incomplete data records that weren?t written before the drive disconnected. The repair will detect this. Look in the log file for items like ?Damaged directories,? ?Damaged files,? and ?Incomplete files.? These will list all of the files and folders the repair process couldn?t reassemble. These are the items you should be concerned about.

You can also look in the archive browser for layers marked as ?-Damaged-?. If the repair found any inconsistencies or missing data in a layer, it will mark that layer as ?damaged?. It will remain ?damaged? until it?s merged with a later layer that contains fresh captures of all of the suspect items. (The ?damaged? state also directs QRecall to force a recapture of those items during the next capture action.)

Since your repair followed an unsuccessful compact action, the repair will also report a bazillion ?Duplicate record? errors. That?s because the compact works by copying records from higher positions to lower positions in the archive file. If you interrupt the compact, it can leave a copy of thousands, even millions, of records at two positions in the archive. These can be ignored. The repair action simply logs the inconsistency and then ignores the duplicate.

In the logs you sent, I do see a handful of missing or damaged items, mostly in your iTunes media backup directories. These should be recaptured, but I?m guessing the rest of your archive is intact. I can?t tell for sure because the log records uploaded by the diagnostic report is incomplete. (The diagnostic report only uploads the last few megabytes of log records, and the repair action is so full of ?duplicate record? entries, that it doesn?t go back further than the last repair you performed.) If you want me to investigate this further, please email me your complete ~/Library/Logs/QRecall.log file.

Kurt,

All great questions. QRecall is primarily focused on detecting and protecting you from damage to the archive. Currently, all records in an archive are checksummed and QRecall tests those checksums whenever those records are used. This allows QRecall to detect if archive data has been corrupted, preventing you from recalling corrupted data or compounding the corruption with new data.

The next (as-yet-unreleased) version of QRecall adds data-redundancy. This allows QRecall to detect and correct limited amounts of data corruption in the archive.

QRecall cannot, however, do much about data corruption in the source filesystem. If you need that kind of protection, you should be using a RAID or other filesystem that supports data redundancy and correction.

It?s not practical to re-read every byte of data of every source file during every capture. QRecall is, at its heart, an incremental backup utility. To do that efficiently necessitates the use of monitoring filesystem change events and comparing file metadata to determine which items needs to be recaptured. Recapturing all of the source data every time would turn every capture into an initial capture, requiring many (many!) hours to complete.

If, however, you still want to do that sort of thing, I?ve had a "?force? capture option on the drawing board for awhile. This would force QRecall to recapture all of the source items in their entirely, as if they had never been captured before. If that?s important to you, let me know and I?ll give it a +1.

There?s also been requests for a verify/compare action that would perform a byte-by-byte comparison of data on the filesystem with the data in the archive and note any differences.

Alex wrote:I assume if I install QRecall on the server with its "Start and run actions while logged out" enabled, the actions will run (I understand I'll need to have appropriate filesystem permissions for QRecall on the server).

Correct. You also don?t need to install an identity key on the server if all you?ll be running are maintenance actions (verify, merge, compact, repair, and so on).

If you have a fast server and a slow network, it?s recommended that you perform everything but captures directly on the server.

However, it doesn't appear that on a headless server a verification failure has any place to warn me. Given my experience so far with handling fileserver disconnects I'm guessing I'll see verification errors with some regularity.

Possibly. For most actions (capture, merge, and so on) a spontaneous interruption due to a network communications break, process termination, or loss of power, will leave the archive in a state where it will be auto-repaired on the next regular action. That includes the verify action. So even if you regularly lose connection with the file server, or crash your laptop, the next regularly scheduled action will/should restore the archive to its previous state and pick up where it left off.

Caveat: There are critical moments in an archive?s life where a terminal interruption will leave the archive is a state that can?t be auto-repaired. For those (rare) situations, a repair will be necessary.

The discussion of autorepair in the manual focuses primarily on the client repairing during capture.

All actions will first attempt to auto-repair the archive before proceeding. The one exception is the Repair command when you explicitly tell it to not perform an auto-repair before repairing.

Tip: If all you want to do is auto-repair an archive, run a verify action. If it manages to get started, then any auto-repair that needed to be performed was successful, and you can now cancel the verify.

However, in my logs I see messages like:

Action 2014-11-14 18:00:01 Failure Capture failed
Action 2014-11-14 18:00:01 The index of the archive is incorrect and needs to be recreated.

That?s a more serious issue that would require a repair (or reindex) to correct.

- This old thread http://forums.qrecall.com/posts/list/118.page#488 mentions email notifications, was that ever implemented?

QRecall support Growl, and the latest Growl supports several email notification extensions. (Note that probably won?t help you on your server if you don?t leave a user logged in, but it would likely work on your clients.)

- Is it possible to chain verify to repair on the server action?

That?s on the to-do list for the next version of QRecall.

- If the server verification fails is this guaranteed to trigger a warning and eventual autorepair on the status on the client? Will the client's status window show the verification state produced by the server (i.e. is the verification failure persisted to the archive for other readers)?

An action that fails (but does not terminate) will update the status of the archive, and that status will (eventually) appear in all status windows that are displaying the status of that archive. So yes, a fatal verify error will eventually percolate back to the client?s status window.

Note that this doesn?t apply to auto-repairable conditions. The conditions that leave the archive in an incomplete state are unexpected terminations that (obviously) don?t finish and don?t update the archive?s status. The next action will attempt to auto-repair the archive. If successful, no harm no foul. If not, it will update the status to ?needs repair? and stop.

- Will the server repaired archive trigger recapture of corrupted files in the archive from the client?

Yes. If the repair process finds anything out of place in a layer, that layer is marked as ?damaged?. A capture that references a damaged layer ignores all file change history, exhaustively recurses through the entire directory tree, and fully recaptures all suspect files.

Alex,

Here?s what the Send Diagnostic report includes:

The comments and email address you enter into the Send Diagnostic dialog

An anonymous system profile

The last 20MB of QRecall log entries

Your preference settings

The actions you have defined

Any crash reports stored in ~/Library/Logs/CrashReporter who name starts with QRecall.

That information is then ZIP?d and uploaded to the diagnostic report server. The shell script that prepares this is named ?buildreport.sh? and is part of the QRecall application package.

Since most of that information is useful for diagnostics, it helps to send it all. If you?re still unconformatable using the built-in report upload feature, do this:

Create a folder (it can be anywhere, even on the desktop).

Save a plain ASCII text file inside that folder named ?comments.txt? containing whatever comments you have about the report.

Open the Terminal.

Use the terminal to navigate to the buildreport.sh script inside the QRecall.app bundle.

In Terminal, execute the buildreport.sh script. The first argument must be the empty folder your created in the first step.

When the script is done, the folder will contain a report.zip file containing the diagnostic information. If you?re satisfied with its content, send that zip file to support (mailto:support).

Or, just send whatever you want to support and we?ll figure it out.

A gray layer (or a missing layer when Show All Layers is off) means that the layer doesn?t contain any items contained in the browser window. It?s designed to help you ignore layers that don?t apply to the items you?re browsing.

If all recent layers are grey, it?s most likely that you?re browing an older set of files.

Everything in an archive has an owner, determined by your identity key. Within each owner are the volumes that have been captured, and inside those are the individual files and folders.

Use the navigation bar at the bottom of the browser window to navigate to the top of the archive. Here you will see all of the individual owners who have captured files to this archive. My guess is that on February 11, 2014 you either changed identity keys or migrated to a new hard drive. From that moment on, everything captured to the archive was added to the new identity key or volume. But the items belonging to the old owner or volume are still there, but in a different branch of the archive. This is the one you are probably browsing.

Find the owner for the current identity key and the volume that contains recently captured items. You?ll find your recently captured items there.

Also see the Help > QRecall Help > Guide > Advanced > Combine Items topic for how to join two sets of items captured with different identity keys or from different volumes into a single owner/volume.

Rosco Mahoney wrote:So QRecall 2.0 is still on the horizon? Maybe not too long?
Either way I am really looking forward to it.

Rosco,

Like the Great Pumpkin, QRecall 2.0 will rise again.

I?m happy to report that I delivered the final chapter of my new book, Learn iOS 8 App Development, to Apress yesterday. I expect near-full-time development of QRecall to resume in a week or two. I?ll post something on the forums when the next beta looks close.

Ralph Strauch wrote:... and got a status window message that the archive required repair. Since that's a long process with a terabyte archive, I decided to rerun the backup first. It completed successfully and has been working since. The Archive Status window, though, refuses to reset and continues to show an "archive needs repair" message and a red circle, as does the status window on my other computer.

That?s correct. When a capture, merge, compact, or other action that is modifying the archive encounters a fatal problem reading or writing the archive data, it sets a ?needs repair? flag in the archive?s status file. This is what triggers the message you?re seeing in status window.

The only actions that will clear that flag are verify and repair. Another successful capture, merge, or whatever does not mean the archive is fixed. None of these actions will repair anything that?s broken in the archive (except those things that can be fixed by auto-repair). If the next capture was successful, it just means you dodged the bullet that got your last capture, but it doesn?t mean the archive is in perfect working order.

I even told the Status window to forget the archive, thinking that would clear it, and the repair needed message still came back.

The Ignore command just makes the status window forget about that archive?s status. The archive, however, still maintains its status internally and the next action (capture, compact, verify, ?) will restore the archive?s status in the window in its entirely, which includes the remembered ?needs repair? flag.

Is the Status window indicating an underlying problem with the archive that needs repair, even though it seems to be working OK, or is this just a problem with the status report?

It may, or may not, be a problem with your archive. Since your failure was due to a network communications error, it?s possible there?s nothing actually wrong with your archive. But QRecall doesn?t know that, and it won?t reset that ?needs repair? status until it can verify that. And it only verifies the integrity of the entire archive during the verify and repair actions.

If it really bugs you, you could cheat. The archive?s status is maintained in the status.plist file in the archive?s package. Use the Ignore command in the status window to cause the QRecall application to discard its copy of the status, and then open the archive package and delete the status.plist file. QRecall will forget everything it knows about the status of that archive.

Or, you could just wait and run a repair.

Since updating to Yosemite, I'm seeing a couple of other anomalies in the log. The first is repeated "Cannot connect with scheduler" messages, followed by reinstallation of com.qrecall.monitor.plist

The interprocess communications is even more problematic in Yosemite. I?ve completely rewritten this logic in QRecall 2.0.

and the second is an "Unable to determine path to VM swap files; assuming /var/vm" message within each backup log. (I don't see either of the log entries in my other computer.)

I?m still doing research, but it appears that Yosemite has eliminated the vm_swap process. It?s now probably baked into the core OS. As such, there?s probably no way of customizing the location of your swap files in 10.10. This has little or no impact on QRecall. If it can?t find the vm_swap process it simply uses the standard VM swap location. In the next version of QRecall, I?ll remove this test for Yosemite users, but for now just ignore that message.

And as long as I'm writing, I'm curious about what's reflected in the "Ignored some changes" entries I see frequently?

OS X?s filesystem events history is a fantastic feature, but has known flaws. If QRecall relied solely on filesystem events to check for modified files, there are files it could premanently miss. (And yes, there are situations where Time Machine will neglect to backup files indefinitely.) QRecall combats this with a limit (which defaults to about a week) that it will trust the filesystem history information. Once that time period expires, it ignores the filesystem history and preforms an exhaustive scan of the entire filesystem. It also logs the message you see so you know when that happened.

Prion wrote:does QRecall run under Yosemite?

All testing so far indicates that the currently released version of QRecall (1.2.3[8]) is fully functional in Yosemite. I?ve been running QRecall on Yosemite since the first developer release, and have only seen a couple of small cosmetic issues. Testing has included normal day-to-day capture operations, as well as full restores of a bootable Yosemite system.

If anyone reports any Yosemite related bugs, I?ll release a patch. But so far, no one has.