Ming-Li Wang wrote:7. Speaking of cache files (and items excluded by Time Machine), v2 has retained the same behavior from v1 of excluding the whole "Caches" folder (not just its contents). Would you consider changing it to something equivalent to the new "contents exclusion" mode? Time Machine's standard exclusion plist specifically says some folders should be retained (while their contents are discarded), and my experience was the system would behave abnormally if some of the folders were not there after restoring from a backup.

It looks like you're getting your wish, after all.

To be compatible with OS X 10.11 (El Capitan), the filtering of most system folders (trash, cache, /dev, mountpoints, and so on) has changed. QRecall 2.0.0b15 now captures those directories, but excludes their content.

With El Capitan's new System Integrity Protection features, many of these directories are protected and have very specific permissions and file flags. Starting with OS X 10.11, if any of them are missing or have the wrong permissions, the system won't boot.

Johannes wrote:Any hints?

Nothing leaps to mind. Send another diagnostic report (since you rebooted) and I'll dig into it.

Bruce Giles wrote:I'm already looking through a pair of the finest reading glasses that WalMart can supply (3 for $7.98!), and trust me, it doesn't help.

I've added a to-do item for the UI rework to make the "indeterminate" state of the progress indicator more obvious. I'm considering disabling the view instead of using its indeterminate state, or try setting a different color at the same time (with apologies to the color blind), so that the indeterminate state would appear grey or clear. I might even make the choice an advanced option, should you prefer one solution over the other ...

By the way, this brings up another issue that's not specific to the beta, as I've seen it in Rev 1 of QRecall as well. Note that the attached screenshot shows it's capturing a file called "screenshot_0000.png". That QRecall capture has now been running for more than 24 minutes, and the activity window STILL shows it working on screenshot_0000.png. I've checked the actual size of that screenshot, and it's only 281K, which QRecall ought to be able to capture in a fraction of a second. So what is QRecall really doing?

There are a couple of reasons, much of which is boring technical details about how QRecall's different threads communicate with each other and how they communicate what they're doing with the activity monitor process.

There is one, really big, status message flaw that has been only recently addressed in the QRecall beta. During a capture, an item's icon and display name are gathered by a separate process. In the not-too-distant past, and so that this doesn't slow down the capture thread, the item's icon and data are captured in parallel. The side-effect of this arrangement is that the message (containing the item's icon and display name) weren't displayed until after the item was captured. So the messages really should have said "Captured file.txt" not "Capturing file.txt".

In the beta, the capture message is now updated as soon as the first thread collects the item's display information. So (for larger items) it will now accurately reflect the item currently being captured.

There's still another issue, however, in that the status message only updates when there's something to report. And by "something" I mean a file to capture or other maintenance activity ("closing archive", "updating indexes", and so on). So if there are a large number of items that haven't changed, the last "Capturing ..." message may stay in the status window for an inordinate amount of time while QRecall searches for the next item to capture. Furthermore, the progress indicator bar is only updated when the status message changed, so that too will remain unchanged.

The way to address this would be to add a "Searching for changes..." or some similar message that would appear during long gaps between captured items. My reaction to this is roll my eyes and say "Oh, great, something else to slow down the capture process," but it might be worthwhile if it keeps the interface from being confusing.

If QRecall spent all that time doing some kind of maintenance on the archive file, it seems like it ought to say so in the log and in the activity window.

It does, but more than likely the capture was just looking for the next item to capture and didn't really have anything else to say.

I spotted that the other day, too.

The fix will show up in the next release.

Johannes wrote:I excluded three folders for the archive manually. When I realised that I should have excluded the iTunes library too, it won't stick. I could add it to the list, but the next capture will start again archiving iTunes. I canceled, opened the archive, checked the settings: iTunes is not in the exclude list. Tried it again, it was added to the list. Closed the archive, re-opened it, checked settings: iTunes no longer in the list. Strange. I had to exclude the folder by setting its preferences via command line tool.

Another bug in the archive's settings dialog. The fix didn't make it into b14, but should be in b15.

While browsing to find out why my archive has started exploding, I noticed that the file sizes in various places do not "match". Folder or Layer sizes where much bigger or smaller than the content. I did not check every item of course but it was strange to find only one GB of data in al Layer that claimed 6 GB. But maybe that is a lack of my understanding of QR's magic.

It's really hard to say. It's sometimes hard to reconcile the size of items in the archive because of complications like invisible items, packages, and other filesystem curiosities.

When I browse my archive I see two disks under my user with all the folders like it should be. But the inspector for both disks looks different. One shows the properties like kind, size, owner etc - as it should. The other says: Kind: Placeholder and no further infos beside the list of the layers.

When you capture an entire hard drive, there will be a volume item in your archive. If you only capture something within the hard drive (like your home folder), all of the enclosing folders leading to that item are stored as "placeholders" in the archive. These items have not been captured—QRecall has no information about them beyond their name. They exists solely to place the captured item at the correction location in the volume's hierarchy.

Given your earlier comment that you suddenly started capturing the whole volume, but then removed that from your captured items, I suspect the "Volume" item is an artifact of capturing the whole volume. You can verify this by looking at what layers it spans. If so, you can delete this item from your archive. The "placeholder" volume should contain your regularly captured sub-items.

QRecall Capture Preferences is not in the service menu. I searched in the system preferences>keyboard, no luck either. Only the two "old" entries capture to/reveal in Qrecall archive are there. Tried restarting. No luck.

In your ~/Library/Services folder you should have a symbolic link named QRecallService.service that points to the QRecallService.service resource in your QRecall application bundle (typically /Applications/QRecall.app/Contents/Resources/QRecallService.service). The log file you sent me says that there was an error removing the old symbolic link, which might have prevented the new one from being installed. Try removing it and running QRecall again.

Ming-Li Wang wrote:After digging into the various records associated with those verification actions scheduled for early Sat. and Sun. morning, however, I found several of them--the last one on Sat. morning, at around 4:40am, and all 3 this morning--did not have a "Verify finished" line. And sure enough, those are exactly the 4 archives listed as "Verified 13 days ago" in the status window.

That's exactly what happened; the verify (and one merge) action never completed.

This is another side-effect of the monitor application process going to sleep while the computer is still running. I've identified this particular code path and have a fix, which will appear in 2.0.0b14.

Manual verifications on all 4 finished successfully as well.

Yes, because these commands were started while the monitor was awake.

Now, I don't want to disable status warnings in the menubar icon; nor do I want to "forget" the archive, but the red dot bugs me, and might numb me someday. So please consider allowing some customization there in the future. Thanks!

Well, it's a good thing you pay attention to those red dots, because this time it really was a problem!

There is an issue with archives that are captured/verified very infrequently—or never—in that they acquire a red dot that never goes away. I have this situation on my server. I have an archive with a capture of my emergency recovery volume. (Should my emergency recovery drive ever die—and I've replaced it twice now—I can quickly have it ready to use again with a single restore.) Since the emergency volume never changes, it never gets recaptured. So eventually, I get a red dot warning me that it hasn't been captured in a while.

I have a feature on the to-do list that will add a "Forget Capture Status Until Next Capture" option to the status window, so you can ignore that particular warning without forgetting about the entire archive.

Bruce Giles wrote:I don't think Apple could have made it any more subtle. I much preferred the old barber-pole pattern, but I realize that QRecall is just using what Apple provides there.

In the next iteration of the UI I was contemplating using the even-smaller progress indicator. I might have to include a free pair of reading glasses with every copy of QRecall.

So, just out of curiosity, what icon was supposed to be there? I went back and looked at QRecall 1.2.3.8 which is still running on one of my other computers, to see what it did. In that case, the top level item is the name of the computer, as defined in the Sharing panel of System Preferences. It happens to be an old iMac, and the icon is an iMac icon, so that's appropriate. But in b13, the top level item isn't the computer's name, it's MY name. In fact, the computer name isn't even in the list. The next level below my name is the name of the hard drive. So what should the top level icon be -- a human silhouette, or something like that?

At the top level of a QRecall archive are the owners. An owner is defined by an identity key and can be synonymous with you, the system, or the identity key. Everything captured using that identity key is contained within that owner.

The owner's icon is the "computer" icon from your system. The name is defined in the QRecall > Preferences > Identity Key. When you first enter your identity key, the name defaults to your computer's display name. You can change it to anything you want to identity the source of the captured items. Also note that you can later change the name of the installed identity key at any time, should you decide you want something else.

I spent some more time with this problem yesterday and I've determined that the legacy function I was using to obtain static system icons (like the icon for your computer system, the default icon for hard drives, and so on) no longer works reliably when the process is running as root—which is exactly what the QRecallHelper runs as during a capture. The call doesn't fail, but instead returns a completely blank icon, which QRecall (lacking any artistic judgement) dutifully captures.

Apple has provided a modern replacement for this function, but it's only available to processes running as the user and within the user's GUI bootstrap; QRecall capture processes don't meet either of these requirements.

I think I've developed a workaround, which I'm testing now. Unfortunately, if you've captured items and have a blank icon you're going to have to wait for those blank icons to get replaced with updated ones, which should happen during subsequent captures and merges.

But they will if the desktop computer is connected to a UPS (and monitoring the battery state via a USB cable), right? The release notes seem to indicate that.

If your UPS system has the software drivers and hardware required to communicate the status of the UPS to the power management system, then the answer is yes; QRecall will treat a desktop machine running off a UPS exactly as it would a laptop running off of battery power.

I was going to argue back on this one. I was absolutely certain you had used the word "scrap" in several places, in which case, "scrapping" would be appropriate. I went back and looked again and couldn't find it. Then I read ahead and discovered that Ming-Li Wang noted it as well, and you've apparently fixed it.

George Orwell would be proud.

I have to say I prefer "scrap" and "scrapping" to "scrape" and "scraping". I think of "scrap" as a synonym for "destroy". For example, "The wrecked car was scrapped." I don't think of "scrape" that way. But it's really not a big deal either way. I can learn to live with it. If I really have to.

I really wanted to make a statement about what QRecall was doing (scraping off items), not insinuate that the items being deleted were rubbish (scrap).

Ming-Li Wang wrote:I thought one could get the same result by simply clicking on the Next or Schedule column first before clicking on the Archive column. No?

Sadly, no. The sorting is relatively stable, so it might work until the table was updated again. But the next time the status of an action changes the entire table is reloaded, and the sub-order of items will change in unpredictable ways.

I thought as much, but didn't speak because English is not my first language.

English is my only language and I still find it confusing.

The cause for confusion probably lies in the beta release notes, however. In the section titled "Excluding, Ignoring, and Scraping Items", "scrap"--not "scrape"--is used numerous times as a verb.

LOL. I checked the interface so carefully, and completely missed this in the release notes!

People with dyslexia should not be allowed to write.

Fixed. (At least I'm pretty sure. Someone else will need to check it.)

Bruce Giles wrote:

When capturing, the progress bar in the activity window goes to full (100%) immediately at the start of the capture, and then it never changes after that. I have this same problem in the current release version of QRecall also. (Just as I was getting ready to post this message, an action started another capture, and this time the progress bar seems to be working correctly. Is that something you fixed in b13, or have I got some kind of inconsistent thing going on here?)

That's not 100%, it's "indeterminate." OS X progress views have an "indeterminate" mode to indicate that the amount of progress is unknown or unknowable. When the look of OS X got "Yosemited." the familiar marching barber-pole pattern was replaced with an (very) subtle gradient that drifts slowly across the screen. On small controls, it's nearly indistinguishable from the progress view at 100%.

When QRecall begins a capture operation it starts two threads: the main capture thread and a secondary "estimate" thread. That second thread tries to quickly rattle through the directories and change history in order to estimate the amount of work the capture thread will actually have to do. Until the estimate thread is done, the progress indicator is set to "indeterminate." Sometimes the estimate can be made very quickly, but in other cases the estimate will take nearly as long as the actual capture, in which case you might not see the progress indicator move at all.

My old backup drive was getting a bit flaky, so I bought a new external drive, and set up new archives for beta testing. I have not yet tried to convert my old archives. When i opened the new archive window for the first time after the first capture, it opened in icon mode. There was a blank icon with my name under it. I had to double-click the blank area to open up a new view showing the hard drive, and then I got an icon on the drive. The lack of an icon to go with my name was disconcerting -- at first I didn't even see it and thought the archive action hadn't worked. Did you intend to have some kind of icon there, or was it supposed to be blank?

To state the obvious, that's a bug. I'm still not sure where the problem is, and I'm beginning to think there are several bugs. In some cases it appears that the icons for the volume and computer aren't being captured properly. In other cases, the static system icons for a volume and the computer that are supposed to display when a captured icon isn't available aren't loading for some reason. Its been difficult to resolve because neither of these problems occur on my system with any regularity. Regardless, it's on the short list of bugs to look into. The issue is entirely cosmetic, as the icons captured/displayed for items in the archive aren't used to recall them.

Will the capture assistant be updated to support the new event conditions? I like the new conditions. I'm testing the one that waits to back up my virtual machine folder until after VMware Fusion quits, and so far, it seems to be working.

I consider most of the new action schedules and conditions to be power-user features, so I don't have any plans to add them to the assistant—which is really just to get you up and running with QRecall as quickly as possible. The assistant in 2.0.0b13 does add the new power state conditions to the actions it creates so that most will stop automatically if the power capacity of the battery drops below 20%. (On desktop computers, these conditions will never trip.)

Feature request: Two-column sorts of the Actions window. I'd really like to see actions sorted first by the Archive column, and then by either the Next or the Schedule column.

I'll add that to the list. Most of the UI enhancements are being addressed in version 2.1, so I'll probably get to it then.

In the release notes, "scraping" should be spelled as "scrapping", with two "p"s. Mentioned so you can get it right in the help files.

I bow to your superior skills in proofreading, but I have to call you out on this one. It's "scraping" (one 'p'), the noun form of "scrape": "to push or pull a hard or sharp implement across a surface or object so as to remove dirt or other matter..."

But it really doesn't matter. You can think of the new feature as "picking up scraps" that have already been captured, or "scraping off" items you don't want to keep on your hard drive.

Excellent news!

Steven J Gold wrote:Deduplication (and therefore, QRecall) should be my friend, so I've jumped onto the Beta hoping to become a user.

Welcome aboard! Now we'll see if we can get you to stay.

But QRecall pales in comparison to TimeMachine in one important aspect: Impact of QRecall on system usability.

I rarely notice when Time Machine is doing it's hourly stuff,, but QRecall severely impacts my computer. Suddenly, my Mac's operations like web browsing, file manipulation -- anything which requires disk access -- slow down and I wonder "What the heck's going on?" and the answer is QRecall. Especially for operations like compacting an archive or verifying.

I freely admit that the CPU and memory requirements involved in comparing every block of file data to a corpus of potentially millions of previously captured blocks, calculating checksums, generating error detection and correction codes, compression, and other tasks are vast compared to that of simply copying data from one drive to another. Keeping QRecall from interfering with your other work, therefore, can be a challenge.

(1) Time Machine sets the "use low-priority I/O" flag which gives precedence to non-Time Machine I/O, thereby lessening the impact of a backup. Could this be made a settable option in QRecall?

QRecall already does this. But, again, QRecall is doing orders of magnitude more work than Time Machine, so it still takes a toll on performance.

(2) There are times I *really* need all the performance I can get from my Mac (like when running Windows in a virtualized environment) and many of QRecall's operations take *hours* to complete. I've learned the hard way quitting QRecall in the mist of a long operation (like a compact) may have dire consequences!

I sure hope there are no dire consequences! (Unless you're force quitting a merge or compact action, in which case there can be dire consequences.) If you need to stop an action, use the stop button in the monitor window or send the QRecallHelper process a quit (TERM) signal, and not a force quit (KILL) signal. The stop or TERM signal will tell QRecall to find a break in its work, gracefully stop, and close the archive.

Could we *please* have a "Pause" button? Something which would pause QRecall until a "Resume" button is pressed?

In the QRecall Monitor window, there's a little stop (X) button in the upper right corner of each action progress pane. Click-and-hold (or Control+click) that button and you'll get a pop-up menu. In that menu you'll find two sets of commands of interest.

First is the Pause For > submenu. You can select to pause (suspend) the action for a brief time period between 5 minutes and 4 hours. Once you've paused an action, the menu item changes into a Resume command that will end the pause mode immediately.

There are also Stop and Stop and Reschedule commands. The first simply stops (quits, TERM signals) the process, identical to simply clicking the stop button. The second stops the action, but prompts you to immediately set a time at which the action is restarted. (This command only applies to actions started via an action document, because only actions can be scheduled.)

And here are some other tips:

In the QRecall Monitor menu (or in the QRecall application's Action menu, if you're not using the monitor menu), you'll find a Hold all Schedules > submenu that will suspend all of the scheduled actions for a period of time. This command affects action that are scheduled to run but haven't started yet, while the Pause for > commands suspend an already running action.

Time consuming actions can be scheduled to run late at night or early in the morning. You can add power requests to the schedule so that the computer starts up when the action is about to run and/or shuts down or goes to sleep when it's finished.

The really time consuming actions, like Compact, don't have to be run very frequently. Compact really doesn't need to be run more than once a month. How often you run merge actions is a matter of taste, but these too can be deferred for days or weeks with no significant impact on your archive. In fact, merge is more efficient if run less frequently. The other big time consumer is verify, but that can be paused or stopped without any consequences.

If you have a really large (more than 2 TB) archive, the new "Defer Data De-duplication" feature added in QRecall 2.0 is there to address the performance issues of capturing data to a large archive by putting off the de-duplication work until you compact the archive. This is because the effort required to de-duplicate data grows exponentially as the archive gets bigger. By deferring de-duplication, you can capture data all week long very quickly, and then de-duplicate it en masse over the weekend.

And if you want to get really tricky, there are new application-based conditions you can add to action schedules. For example, you could ignore (not run) actions whenever your Windows emulation application was running. It's also not a good idea to try and capture virtual machine images while they are in use. Instead, you can create an event schedule to capture these after your Windows emulation application quits.

Let us know if this helps and if you have any other questions or suggestions.

Adrian Chapman wrote:In version 2 the ability to use the new "capture on event" schedule seems ideal using "Run when capture items change" coupled with a delay, but I can't see how I can configure things so that they aren't captured by the hourly action but can still be captured to the same archive using the "Run when capture items change" action.

You're not being dense; it's impossible.

If I understand your question, the scenario is thus: You have top-level folder that you periodically capture in its entirely.

Contained within this top-level folder you have either (a) a folder who's changes are insignificant and you aren't interested in capturing regularly and/or (b) a folder who's contents are extremely important and you want to capture those much more frequently than the top-level folder.

For this example, lets say your top-level item is your home folder. You've created an action that captures your home folder every hour. Your home folder contains an iTunes folder (a), that you're not worried about capturing more than once a day, and a Working Projects folder (b), that you want to immediately capture if there are any changes.

(a) is easy to setup. Use the QRecall Capture Preferences service to set that folder to "Ignore Changes". In the case of the iTunes folder, you'd probably set it to ignore changes for 24 hours, or set it to ignore changes during the hours you typically work.

(b) can be addressed by setting up a second capture action, using a "Capture Items Changed" event. You'd create a capture action for the Working Projects folder and set it to, say, capture 2 minutes after any change.

You don't have to do anything to exclude/ignore the Working Projects folder from the hourly home folder capture. The Working Projects capture action will capture all the changed items in that folder as soon as they change. So when the home folder capture gets around to running, it's unlikely that there are any uncaptured changes in the Working Projects folder. QRecall only captures changes, so there will be nothing in the Working Projects folder to capture.

What you can't do is have a folder that is both ignored (a) and frequently captured (b). That's logically inconsistent.

Ming-Li Wang wrote:1. Those event-driven actions take place more often than I expected due to meaningless (non)change to .DS_Store files.

This, sadly, I can't do much about. OS X's file change event service doesn't report exactly what changed in the filesystem, just that "something in folder XYZ changed." So it's impossible to prevent changes to specific items from running an action.

I would like to exclude, e.g., all .DS_Store files, all *.bak files, all *.*-journal files, and all "minidump" folders, just to name a few.

This has been on the to-do list for awhile.

I've long intended to add a filter mechanism that would allow you to use file globbing and/or regex to exclude items. But it's such a geeky power-user feature that I keep putting it off.

Since you asked, I'll give it a +1.

2. The current explanation for the option "Ignore new events for ___ (min./hours/days/weeks)" &helips;

There are two issues here. First, I haven't updated the help for the new 2.0 features yet, so the behavior of the "Ignore" period for the new "Capture Items Changed" event hasn't been documented yet.

Second, the terminology is a little off because the "changed items" event are stateless and the whole events schedule mechanism is designed around stateful events, like mounting and unmounting a volume, logging in and out, or launching and quitting an application. Said another way, when a folder's content changes (the event) it stays changed forever (until captured). When an application launches, that launch event ceases to exist when the application quits again.

Because of these differences, how events and event state are treated during the "Ignore" period of event schedule is?as you've discovered?quit different for change events vs. stateful events.

The two need different terminology, and I plan to give them different interfaces, but that's probably going to have to wait until 2.1 when the UI gets an overhaul.

Ming-Li Wang wrote:B12 went through the grueling early Sat. morning schedule (moved to this morning as usual) for the first time here and survived. It's still working fine, and doesn't give me spinning rainbow wheels.

Finally, some progress!

From the diagnostic report you sent, I can confirm that all of the actions ran successfully.

Found some peculiar entries in the Log Window, however.

1. Five consecutive "could not communicate with helper" entries at address 8:46am, right about when I touched my computer for the first time in the morning. There were 6 of them yesterday morning (I forgot to add Tuesday morning to the group of "Sat. morning actions", so I didn't report back yesterday morning), also at about the time I touched my computer for the first time.

I was able to determine the cause of these messages, and it goes a long way to unraveling a slew of odd issues other have encountered.

Recent versions of OS X have gotten extremely aggressive about suspending apps that it doesn't think are relevant. In the past, the QRecall Monitor process could reasonably expect to get distributed notifications and timer messages when the system was idle, but apparently that's no longer true. Based on the log entries, it appears that OS X is electing to completely suspend the QRecall Monitor application when your display is asleep. Here's the sequence of events that I found:

At 3:20 AM a capture action ran.
20 seconds later, the capture action finished and terminated.
At 8:46 AM the monitor app receives a notification from the capture action that it would like be monitored.
The monitor app tries to connect with the capture action (the one that terminated 5 hours ago) and fails (no surprise).

This explains the spat of "Could not communicate with helper" errors that you're seeing when you wake up your display. Even though the capture action sent a notification to the monitor app at 3:20, OS X hung on to that message and didn't deliver it until 8:46.

2. There are 2 errors associated with the actions that took place overnight. The first at 4:49:55am, "Helper did not start" after a successful "merge" action. Then "helped did not start" again at 5:34:55am, which stopped a "verify" action. Both errors gave the same description (with different timestamps, of course):

These message are both erroneous and have the same root cause as the "Could not communicate..." errors.

Both the merge and verify actions also tried to establish a connection with the monitor app (this time using a direct Mach port connection, rather than using distributed notifications?for complicated reasons having to do with bootstrap namespaces). Again, instead of delivering those messages, OS X left the QRecall Monitor app suspended so it never processed or responded to those messages. The messages eventually timed out, and threw an exception which cause the "Helper did not start" error to be logged.

The "Helper did not start" error is actually a mistake in this context. It's supposed to report that helper's main run loop never got started, but it can also get logged if the run loop starts but then throws an uncaught exception, which is what happened here. I've fixed that bug. Now the "Helper did not start" error only gets logged if the helper encountered a fatal error before its main run loop is started, and not afterwards.

I've addressed the two communications problems—the monitor waking up with a request to monitor an action that's already finished and terminated, and the situation of an action trying to send status messages to a monitor app that's suspended and unresponsive.

All of these remedies will appear in (lucky) 2.0.0b13, which should get released before the end of the week.

Thanks again for all of the great feedback and diagnostic reports. I probably would have never found this on my own!

Ming-Li Wang wrote:One question: how reliable is OSX/HFS's file change mechanism?

There are broadly two ways to use the filesystem change event information: to determine if something changed and to determine what changed.

The first use is rock solid. If anything within the watched folder hierarchy changes, OS X will notify the observing process (the scheduler) which will start the action.

The second use is where things get tricky. There are some extreme edge cases where filesystem events can't be used to reliably determine exactly which folders contain changes. This is why backup software like Time Machine can run into trouble. QRecall's capture uses a mixed solution that uses filesystem change information to skip folder during most captures (which is fast). But that trust has a time limit and, once exceeded, QRecall will perform an exhaustive examination of the items in every folder (which slow, but sure).

Do I need to compliment the action with, e.g., daily capture?

No. The "items changed" event simply triggers the start of the capture action, it doesn't tell the action what to capture. So it doesn't matter if you start the action using a daily schedule, with an "items changed" event, or using the command line tool, the capture action will still capture the same set of changes.