Ming-Li Wang wrote:One problem remains: I still can't restart QR scheduler after forcing it to quit in the Activity Monitor. Quit and restarted QR several times, but the scheduler never came up. A diagnostic report has been sent.

I'll look into this.

QRecall's scheduler is managed by the launchd service. launchd is set to run the scheduler all the time, unless the QRecallScheduler process exits with a non-zero status. launchd will also throttle, or eventually stop running, the scheduler if it repeatedly crashes or is forcibly terminated too often.

If you're force quitting (SIGKILL) the scheduler, I would expect it to restart. If you're soft quitting (SIGTERM), the scheduler should immediately clean up and quit with a status of 0; launchd should immediately restart it. If you interrupt (SIGINT) the scheduler, it will deliberately exit with a non-zero status so that launchd does NOT restart it. It will also exit with a non-zero status under certain circumstances, such as with some internal errors.

Relaunching the QRecall application won't help, because it's launchd's run the process. If launchd has decided not to run the scheduler, you'll need to log out and back in (if "Run actions while logged out" is turned off), or restart the system (if "Run action while logged out" is turned on), to get it going again.

Anyway, diagnostic reports will help. They should tell me if the scheduler is terminating early, terminating with a non-zero status, or repeatedly crashing.

Other issues:

1. I used "sudo qrecall captureprefs list -r -s /" to check the QR Capture Preferences settings, and it aborted with the following error midway through the process:

* failed to read directory

This isn't surprising, because you don't have permission to read those directories. Use sudo qrecall captureprefs list -r -s / instead.

I'll put an item on the wish list for qrecall captureprefs -r to continue if it can't read the contents of a directory.

2. There are a few dozen of "Unable to communicate with helper" warnings and one "Unexpected problem; scheduler stopping immediately" error in the log.

Hmmm. The "scheduler stopping immediately" error might be the reason your scheduler isn't running...

Accurately identifying volumes is a tricky business. The name of a volume is essentially useless for identifying it; there are probably 50 million hard drives all named "Macintosh HD" in the world. Plus, renaming a volume doesn't make a new volume.

Awhile back, OS X began assigning each volume a UUID (Universally Unique Identifier), where possible. This is used in aliases and bookmarks to reliably identify a volume, and is QRecall's preferred method. The UUID is used, for example, to determine what device to mount when trying to resolve a link to a file that's on an unmounted volume.

If you install a new drive, or repartition a drive, OS X will assign that volume a unique ID.

Mirroring a drive, block-level cloning of a volume, and similar techniques can, however, result in two volumes with the same UUID. This can result in confusion. OS X tries to combat this by spontaneously changing a UUID if it discovers it isn't unique. But that typically only happens if you clone a drive and then mount it alongside its original.

On filesystems/devices that can't support UUIDs, QRecall falls back to using a combination of the volume's size, the date it was created, and (as a last resort) its name.

So to answer your question, in a roundabout way, some of the things you're doing (installing new hardware, repartitioning) will result in QRecall treating the new volume as a brand new volume. Other techniques, like block-level cloning, may trick QRecall into thinking the new volume is the same one it has seen before.

If you're interested, you can see the UUID (if any) of a volume using the diskutil info command.

Ming-Li Wang wrote:While I don't particularly mind QR starting a new set from time to time, this practice does have its down side: when digging through the archive for some earlier versions of a file, I can't reach all versions of the same file easily.

Hence, a question: is there a way to "merge" the sets? If not, please consider it a feature request. Thanks!

QRecall > Help > Guide > Advanced > Combine Items.

Bryan Derman (of Derman Enterprises) has developed a set of scripts that will incrementally backup a QRecall archive to optical media or disk images.

You can download these scripts, read the documentation, and get the full history from his blog entry Off-Site and Archival Storage for QRecall Backups.

In a nutshell, these scripts divide a working QRecall archive into smaller segments which are stored on disk images. Each segment / disk image can be burned directly to an optical disk, stored across smaller hard drives, uploaded to a server, or added to a cloud synchronization service. As the archive grows, new segments are automatically burned/created. The result is a set of discs or images that, in the future, can be reassembled to recreate the original archive.

There are a couple of caveats. First, you must turn off periodic merge and compact actions. These scripts depend on a feature of how QRecall adds new data to an archive; newly captured data is always appended to the end of an archive, until the archive is merged and compacted. So by not regularly merging, new segments can be written/burned for just the new data that's been appended to the archive. This can go on indefinitely, or at least until your archive runs out of disk space.

At some point, the archive will need be merged and compacted, at which time you'll have to reburn/recopy a new set of discs/images from the beginning. But with adequate disk space, combined with QRecall's efficient data de-duplication, this might only be necessary once a year, possibly even less.

Bryan has done an amazing job with these shell scripts. They're flexible, can be customized, are very well documented, and handle a variety of DVD and BluRay disc sizes. Bryan has made these scripts freely available and encourages feedback from anyone interested in trying them out. Currently, they have only been tested with QRecall 1.2.

When Bryan approached me, looking for a way to backup an archive using optical media, I dismissed the idea as impractical. Because of the way QRecall archives are normally merged and compacted, it would require that the entire set of backup images/discs to be repeatedly re-burned. But Bryan convinced me that it's perfectly manageable to not merge the archive and let it grow indefinitely. And there are business practices that would make this a necessity, in situations where all data must be kept in perpetuity.

I encourage anyone who is looking for an off-site backup solution for their archive to give these scripts a try.

Bryan and I will be working together to integrate this functionality into a future version of QRecall.

Johannes,

My stock answer to this question is "No, you can't run two versions of QRecall simultaneously."

The problem is that all versions of QRecall are going to use the same set of resources (preferences, schedule, action documents, status files, archives, and so on). And much more complicated is the fact that QRecall installs a number of system services. These services are all identified by name ("com.qrecall.scheduler" for example). OS X doesn't allow two or more services to be registered with the same name, and QRecall (for security reasons) will refuse to use any service or component that's not the exact same version. So there can only be one scheduler service running, and only the same version of QRecall will talk to that scheduler, which effectively limits the usable versions to one.

Having said all of that, there might be a one-time-only exception as we transition from 1.2 to 2.0. QRecall 2.0 uses a different (safer, more secure) launch mechanism for running privileged commands, like a capture. Because of this, it might be possible to install QRecall 1.2 in user account A and QRecall 2.0 in user account B as long as both users have the "Run actions when logged out" option turned off.

In this situation, all QRecall user services (monitor, scheduler, system service, spotlight importer, and so on) will be installed in that user's account, and just for that user. The only system service that need to be installed is the privileged launch mechanism, but since 1.2 and 2.0 use different mechanisms, they shouldn't stomp on either other.

Warning: This is entirely theoretical, so install at your own risk. If you run into problems, please let me know that you're also running a parallel copy of QRecall 1.2 in any bug reports.

Ming-Li Wang wrote:I've since removed the action but now I get two ghost items in the QR pop-up menu that refuse to go away. (screenshot attached below)

This is an unrelated bug that I'm working on.

Workaround: Quit the QRecall Monitor process to make them go away.

Ming-Li Wang wrote:Got "internal program error" midway through a compact action. The Log shows "unexpected buffer contents" messages. Report filed.

Got it, and thanks for the test archive. I don't think I would have found the bug without it.

Yes, this is why we do beta testing.

This archive had just the right (wrong?) sequence of reads and writes needed to confuse the redundancy buffer management. The fix for this will be in b11.

Ming-Li Wang wrote:If one make a change to the number for the "Keep deleted items for at least ___ [unit]" setting in the archive settings window, then click the "Cancel" button to dismiss the window, the old value would not be retained as expected. Instead, the new value would be kept.

For the record, the value wasn't being set in the archive. The dialog was simply failing to reset the edit field with the old value before presenting it again.

When the Archive Settings window is opened, in addition, the focus (cursor) rests in the number box with the whole number highlighted. If one hits the ESC key at this time (in an attempt to cancel the dialog), the number would be replaced with the letter "I". Hitting ESC again would change it back to the old value, but clicking outside the box would make the "I" stick and get an error message about invalid number.

I can't do much about this, as this is OS X's auto-complete / spelling / whatever interface.

I have fixed the code so that non-numeric values will present a standard validation dialog, and you'll get the choice of editing it or reverting to the original value.

Ming-Li Wang wrote:I didn't know anything about "alias" in QR, and therefore have never used it intentionally. Is it the same thing as Finder alias?

Yes, they're the same thing. The alias files created by the Finder are just alias records written to disk. OS X provides a number of semi-opaque data structures for remembering where a document has been stored. The alias records were a carry-over from Carbon (OS 9), so that gives you some idea of how old they are. The URL Bookmark structure performs the same task for modern software.

Anyway, I was able to reproduce and fix all of the alias->bookmark conversion problems you found in b10.

Ming-Li Wang wrote:Voila, now you can see for yourself that the "Items Excluded by Time Machine" option has been reverted back to enabled state.

I'm pretty sure this is fixed in b11.

I think the basic bug, which is why it was so confusing, was the inability of the archive to determine when the settings had actually changed. (Settings are only rewritten to disk when they change.) Some combinations would trigger a save, others wouldn't...

Ming-Li Wang wrote:4. b10 seems pretty stable, but this morning (about 9am, Taiwan time) I found the Monitor window on the desktop, listing the verification for an archive as the next action, but the scheduled time for that action was 4:40 am. QR was very sluggish. The rainbow wheel spinned for a few sec. with each move (clicking on the menu icon, clicking a menu item, etc.). Restarting the Monitor didn't help. I had to kill all QR processes in the Activity Monitor. The log shows a lot of "Unable to communicate with helper" warning since 4:10 this morning.

If something like this happens in the future, please send a diagnostic report before killing any of the processes. The Send Report command in QRecall 2.0 now samples all of the running QRecall processes to see what they were doing at the time, but if you kill them first that information is lost.

Btw, I couldn't get QR scheduler to restart after having it killed in Activity Monitor (relaunching QR didn't restart the scheduler).

Please send a diagnostic report. There might be a clue as to why the scheduler didn't restart.

5. Starting an action manually in the "Actions" window doesn't start it immediately. See the attached screenshot--next action at 11:39, system time at 11:43--for an example. The action was finally executed at 11:45, which was the nearest time for a scheduled action (for another archive). It's the same when I tried to execute another action (a Verify action) manually.

I discovered this about an hour after I released b10. The change to the scheduler that stopped it from running all the time has gone too far in the other direction, and now it doesn't check the scheduler often enough. *sigh* This should be fixed in b11.

6. I can't disable the "Cache Files" standard exclusion for one of my archives. I've unchecked the item and saved, many times over, but it always remains checked the next time I open the archive settings window. Very weird, for there is no such problem for other archives.

I can't reproduce this, but I recently fixed another bug with how filter definitions are stored in the archive document which might explain why this is happening. Try it again when b11 is released.

7. Speaking of cache files (and items excluded by Time Machine), v2 has retained the same behavior from v1 of excluding the whole "Caches" folder (not just its contents). Would you consider changing it to something equivalent to the new "contents exclusion" mode? Time Machine's standard exclusion plist specifically says some folders should be retained (while their contents are discarded), and my experience was the system would behave abnormally if some of the folders were not there after restoring from a backup.

Testing has shown that all of the "well known" folders in /Library, ~, and ~/Library (like the Caches folder) will automatically be recreated as needed. Most of the excluded folders that must be preserved are the core BSD folders (like /dev and /private/var/vm). If those folders don't exist, the system won't boot.

Of the folders that are automatically recreated, I'm inclined to leave the behavior as it is on the theory that it's likely to cause less problems since the folder will be lazily created with the correct ownership and permissions. The only exceptional case would be if you've set some special ownership, permissions, extended attributes, or Access Control Lists on the folder and want those preserved. If that's the case, turn off the "a la carte" exclusion setting in the archive and use the capture preferences to exclude just the contents of those folders.

When I have a chance, I'll look into how difficult it would be to exclude just the contents of the "a la carte" excluded folders.

8. When trying to check my QRecall Capture Preferences settings with the "qrecall captureprefs list -r -s ~/Documents/" command, I got the following error message:

2015-08-02 12:07:28.975 qrecall[8920:60370] *** Assertion failure in char *DDNewCPath(const char *, const char *)(), /Users/james/Development/Projects/Quantum Recall/Common/Source/Filesystem/DDPath.m:696
* parent path (/Users/wang/Documents/) ends in '/'

Removing the trailing slash solved the problem, but you might want to add some fault tolerance codes or tweak the error message a little to make it less scary.

Error messages are supposed to be scary.

This was just a bug, now fixed. The captureprefs subcommand was failing to sanitize its path arguments.

Ming-Li Wang wrote:I rebooted and reinstalled b10 again, with my old Actions folder in place, and QR launched without problem after installation, but again no conversion was performed. I thinks it's because the 2nd installation checked and realized v2 has been installed before, and skipped the conversion step.

Correct. QRecall sets the QRUpgradeLevel preference once it has finished performing the (rather expensive) v1.x to v2.x update. If you ever want QRecall to perform this update again, use this terminal command:

defaults delete com.qrecall.client QRUpgradeLevel

I've since done more tests, and can confirms that v2 would crash on my system after installation when upgrading from v1 to v2, unless I empty the "Actions" folder beforehand.

That's what I see too. But the OS X crash logs are remarkably uninformative in this case, so I can't tell exactly where/why it's crashing. If you have copies of those old (pre-upgrade) action documents, could you please email them to support@qrecall.com?

What I don't get is: v2, on first run, should only try to convert customized exclusions into QRecall Capture Preferences, according to the release notes.

Actually it's modifying a whole bunch of things. And I don't put everything in the release notes; they're long enough as it is.

One of the things that gets updated for every action are the alias records. When upgrading to 2.0, QRecall tries to convert every legacy alias record into a modern URL bookmark structure. I say "try," because the item the alias refers to has to be reachable for the conversion to be successful. Any items that aren't reachable are left as legacy alias records (which QRecall 2.0 can still use).

That's all for now. Off to play with the new beta (thank you for the speedy release).

Keep us posted!

Ming-Li Wang wrote:Sorry, no logs were kept, but I'm pretty sure the problem is reproducible on my system. If it happens again with b10 (if I have time to try it), I'll keep the log.

No big deal, but if you do install the next version and it crashes, send a diagnostic repot (Help > Send Report). The report will include any QRecall crash logs, which would also be helpful.

Ming-Li Wang,

Thanks for taking the time to beta test QRecall. You certainly get points for persistence.

Ming-Li Wang wrote:Not to be detered, I tried to use drag-n-drop (from finder) instead, and was delighted to find that worked. The joy was shortlived, however, as I soon realized the "-" button was broken, too, and there was no way to remove an excluded item. Can't edit them out from the plist file either as they are now encoded therein.

Fixed.

Next I tried to use the new QRecall Capture Preferences service, only to find it nowhere to be found, not even in System Services Configuration (the one at "System Preferences - Keyboard - Shortcuts - Services"). Reinstalling didn't help.

It would appear that your initial install of QRecall 2.0 failed spectacularly. If you still have the QRecall.log for that time period, please forward it to support@qrecall.com. (You'll want to compress it first, as it can often be rather large.)

The QRecall Capture Preferences service was there, finally, though I had to enable it manually in System Services Configuration.

That, unfortunately, is now standard for OS X. New services are disabled by default; you must explicitly tell OS X which services you want to use.

Time to test it out. I set several folders on my system partition to "Do not capture contents" with the "Exclude from all archives" option enabled, and then back it up to a new archive. It went smoothly, but several excluded folders were still backed up despite the setting. I soon realized the extended attribute wasn't really set because those were privileged folders (owned by root). QR really should ask for permission to set the XA, or at least give a warning about not being able to set the XA. It did neither.

Sorry about that. When I went to check this out, I even found a note in the code: "TODO: Handle xattr set failure here." Obviously, I hadn't gotten back to that.

Anyway, I went ahead and implemented a warning dialog that will now pop up if QRecall can't set the capture preferences for an item.

It was never my intent to allow the QRecall service to set the capture preferences for items not owned by the user, although it's theoretically possible to do via an authorization request, and I'll add that to the wish-list.

A few suggestions:

1. The ability to save a folder without capturing its content has been one of my most wanted features. Restoring the system partition with QR has been a pain without this feature as I always have to spend much time recreating the excluded system folders with the right ownership and permissions.

The new QRecall Capture Preferences service is thus a great addition, but it needs to be able to work on privileged folders, and I hope the same functionality can be extended to per-archive exclusions.

To set the capture preferences for an item you don't own, use the sudo and qrecall command line tools:

sudo qrecall captureprefs exclude /Path/to/excluded/item

2. The "Exclude from all archives" option is confusing. I first thought it meant excluding from all archives instead of just this archive, only to realize soon that this is not a per-archive exclusion, so the "Do not capture" setting can't be archive specific. I had to dig out the release notes to know its true meaning. I would suggest something like "Override archive setting" instead.

I'm afraid it's going to be confusing either way.

Essentially, the "exclude item" setting is QRecall's version of OS X's "exclude item from backup" attribute, designed principally for Time Machine. When you configure your archive, you can choose to honor or ignore OS X's "do not backup" flag, and you choose to honor or ignore QRecall's "exclude item" setting.

So, basically, you have two ways of marking items to be excluded from a capture (OS X's and QRecall's) and you have the choice of excluding those items from the capture, or ignoring the settings and capturing either, or both, sets.

I'll try to make this clearer when I get around to updating the help. And I might change the name of the "Exclude from all archives" setting to something like "Setting cannot be ignored," or something like that.

3. I don't know how difficult this is, but I truely hope there is a way for v1 and v2 betas to co-exist on a system, so I could try out the betas without messing with existing archives and backup routines.

If it was easy, I'd do it myself. (Probably the most frustrating part of QRecall development is not being able to use QRecall all the time.) The main problem is that QRecall installs system services that are identified by name, and only one service with that name can be running. And for security, QRecall insists that all installed components must be the same version. This makes it an all-or-nothing deal.

Thanks for the great work. V2 does look promising.

Thanks. I look forward to you trying it again soon. The next (b10) release should include the fixes I mentioned here.

Steve,

I would expect the update from 2.0.0b8 to 2.0.0b9 to fail, as b8 has the flawed update code in it. The real test will be if the update from 2.0.0b9 to whatever version follows is successful.

I know I've got my fingers crossed...

Dear beta testers,

If you have encountered a situation where QRecall says that it's "Waiting for archive" when there's no obvious reason why it should, please send a diagnostic report.

Also note that if the archive isn't actually busy, the "waiting" situation should resolve itself after a few minutes. If it doesn't fix itself after ten minutes, definitely send a diagnostic report.