All layers earlier than the two 1-year layers defined will be merged into a single layer, but that will be considerably more than 2 years ago.

When you run the merge today (October 14, 2015), here's how QRecall will calculate your merge.

You're keeping the most recent 7 days, so any and all layers captured after Oct 7 will be left as they are.

In the next tier, you have 90 day layers. QRecall starts at Oct 7 and looks backwards for the first full day, which is Oct 6. All layers captured on Oct 6 will be merged into a single layer. It repeats this for the remaining 89 day layers:

Day 90: Oct 6, 2015
Day 89: Oct 5, 2015
Day 88: Oct 4, 2015
...
Day 1: July 9, 2015

Your next tier has 52 week layers. Starting with July 9, 2015 QRecall looks for the nearest full week, which ended on Saturday July 4, 2015. All layers captured between Sunday June 28, 2015 and Saturday July 4, 2015 will be merged into a single layer. This repeats for the previous 51 weeks:

Week 52: Sat June 28, 2015 - Sun July 5, 2015
Week 51: Sat June 21, 2015 - Sun June 27, 2015
Week 50: Sat June 14, 2015 - Sun June 20, 2015
...
Week 1: Sat June 29, 2014 - Sun July 5, 2014

The next tier keeps 12 month layers. Again, started at the beginning of the last tier (June 29, 2014), QRecall searches backwards for the first full month, which is May, 2014.

Month 12: May, 2014
Month 11: April, 2014
Month 10: March, 2014
...
Month 1: June, 2013

Finally, you have 2 year layers. These will start on the previous full year before June, 2013:

Year 2: 2012
Year 1: 2011

Finally, all layers captured before January 1, 2011 will be merged into a single layer.

Every thing else: 2006(*) - 2010

I hope that helps.

(*) Why 2006? Because that was the first public release of QRecall. Obviously, you can't have any layers before that.

If anyone is using, or interested in, Bryan Derman's scripts for backing up QRecall archives onto optical media, you'll be happy to know that there's a new version available: Off-Site and Archival Storage for QRecall Backups

Bryan is now using QRecall in combination with his scripts to secure the data on several mission critical systems. If you have similar needs, please give these (free!) scripts a try and let Bryan know what you think.

My advice would be to wait until I fix this bug

I'm pretty confident that there's no data loss, just migration pains.

Ming-Li Wang wrote:The latter. The date stamp for layer 0 is 2015-01-31, so definitely 1.2.x.

This seems to be the problem. I've identified several issues with archives that were originally created with 1.2 and later updated to 2.0.

I'm still investigating, but this problem seems to be related to how volumes are identified in 2.0 vs. 1.x. This disagreement causes newly captured items to migrate into a new directory hierarchy that, in turn, trips a number of internal consistency checks. In your case, a delta directory record (a database record that records changes in a previously captured folder) can only refer to a another directory record that is both (a) in an earlier layer and (b) at the exact same location in the volume/folder hierarchy. The shifting volume identifiers cause the second test to fail, making the inherited directory record "unreachable."

The verify may also report "layer catalog tree" errors, which basically means the folder structure in the index and the folder structure the verify thinks the archive should have aren't identical.

I have seen this with another archive.

Important question: Is this an archive created with the 2.0 beta, or is this a 1.2.x archive that you started using with 2.0?

Just FYI, I think I've got a solution in the works.

There were about four different, and subtle, problems that intersected the code which deals with users who have relocated their home folders to a different volume. I have definitely fixed a number of bugs for those folks. Now I just have to determine if I've also fixed the related issue for users with regular home folders.

I should have a new version out by tomorrow.

Ming-Li Wang wrote:7. Speaking of cache files (and items excluded by Time Machine), v2 has retained the same behavior from v1 of excluding the whole "Caches" folder (not just its contents). Would you consider changing it to something equivalent to the new "contents exclusion" mode? Time Machine's standard exclusion plist specifically says some folders should be retained (while their contents are discarded), and my experience was the system would behave abnormally if some of the folders were not there after restoring from a backup.

It looks like you're getting your wish, after all.

To be compatible with OS X 10.11 (El Capitan), the filtering of most system folders (trash, cache, /dev, mountpoints, and so on) has changed. QRecall 2.0.0b15 now captures those directories, but excludes their content.

With El Capitan's new System Integrity Protection features, many of these directories are protected and have very specific permissions and file flags. Starting with OS X 10.11, if any of them are missing or have the wrong permissions, the system won't boot.

Johannes wrote:Any hints?

Nothing leaps to mind. Send another diagnostic report (since you rebooted) and I'll dig into it.

Bruce Giles wrote:I'm already looking through a pair of the finest reading glasses that WalMart can supply (3 for $7.98!), and trust me, it doesn't help.

I've added a to-do item for the UI rework to make the "indeterminate" state of the progress indicator more obvious. I'm considering disabling the view instead of using its indeterminate state, or try setting a different color at the same time (with apologies to the color blind), so that the indeterminate state would appear grey or clear. I might even make the choice an advanced option, should you prefer one solution over the other ...

By the way, this brings up another issue that's not specific to the beta, as I've seen it in Rev 1 of QRecall as well. Note that the attached screenshot shows it's capturing a file called "screenshot_0000.png". That QRecall capture has now been running for more than 24 minutes, and the activity window STILL shows it working on screenshot_0000.png. I've checked the actual size of that screenshot, and it's only 281K, which QRecall ought to be able to capture in a fraction of a second. So what is QRecall really doing?

There are a couple of reasons, much of which is boring technical details about how QRecall's different threads communicate with each other and how they communicate what they're doing with the activity monitor process.

There is one, really big, status message flaw that has been only recently addressed in the QRecall beta. During a capture, an item's icon and display name are gathered by a separate process. In the not-too-distant past, and so that this doesn't slow down the capture thread, the item's icon and data are captured in parallel. The side-effect of this arrangement is that the message (containing the item's icon and display name) weren't displayed until after the item was captured. So the messages really should have said "Captured file.txt" not "Capturing file.txt".

In the beta, the capture message is now updated as soon as the first thread collects the item's display information. So (for larger items) it will now accurately reflect the item currently being captured.

There's still another issue, however, in that the status message only updates when there's something to report. And by "something" I mean a file to capture or other maintenance activity ("closing archive", "updating indexes", and so on). So if there are a large number of items that haven't changed, the last "Capturing ..." message may stay in the status window for an inordinate amount of time while QRecall searches for the next item to capture. Furthermore, the progress indicator bar is only updated when the status message changed, so that too will remain unchanged.

The way to address this would be to add a "Searching for changes..." or some similar message that would appear during long gaps between captured items. My reaction to this is roll my eyes and say "Oh, great, something else to slow down the capture process," but it might be worthwhile if it keeps the interface from being confusing.

If QRecall spent all that time doing some kind of maintenance on the archive file, it seems like it ought to say so in the log and in the activity window.

It does, but more than likely the capture was just looking for the next item to capture and didn't really have anything else to say.

I spotted that the other day, too.

The fix will show up in the next release.

Johannes wrote:I excluded three folders for the archive manually. When I realised that I should have excluded the iTunes library too, it won't stick. I could add it to the list, but the next capture will start again archiving iTunes. I canceled, opened the archive, checked the settings: iTunes is not in the exclude list. Tried it again, it was added to the list. Closed the archive, re-opened it, checked settings: iTunes no longer in the list. Strange. I had to exclude the folder by setting its preferences via command line tool.

Another bug in the archive's settings dialog. The fix didn't make it into b14, but should be in b15.

While browsing to find out why my archive has started exploding, I noticed that the file sizes in various places do not "match". Folder or Layer sizes where much bigger or smaller than the content. I did not check every item of course but it was strange to find only one GB of data in al Layer that claimed 6 GB. But maybe that is a lack of my understanding of QR's magic.

It's really hard to say. It's sometimes hard to reconcile the size of items in the archive because of complications like invisible items, packages, and other filesystem curiosities.

When I browse my archive I see two disks under my user with all the folders like it should be. But the inspector for both disks looks different. One shows the properties like kind, size, owner etc - as it should. The other says: Kind: Placeholder and no further infos beside the list of the layers.

When you capture an entire hard drive, there will be a volume item in your archive. If you only capture something within the hard drive (like your home folder), all of the enclosing folders leading to that item are stored as "placeholders" in the archive. These items have not been captured—QRecall has no information about them beyond their name. They exists solely to place the captured item at the correction location in the volume's hierarchy.

Given your earlier comment that you suddenly started capturing the whole volume, but then removed that from your captured items, I suspect the "Volume" item is an artifact of capturing the whole volume. You can verify this by looking at what layers it spans. If so, you can delete this item from your archive. The "placeholder" volume should contain your regularly captured sub-items.

QRecall Capture Preferences is not in the service menu. I searched in the system preferences>keyboard, no luck either. Only the two "old" entries capture to/reveal in Qrecall archive are there. Tried restarting. No luck.

In your ~/Library/Services folder you should have a symbolic link named QRecallService.service that points to the QRecallService.service resource in your QRecall application bundle (typically /Applications/QRecall.app/Contents/Resources/QRecallService.service). The log file you sent me says that there was an error removing the old symbolic link, which might have prevented the new one from being installed. Try removing it and running QRecall again.

Ming-Li Wang wrote:After digging into the various records associated with those verification actions scheduled for early Sat. and Sun. morning, however, I found several of them--the last one on Sat. morning, at around 4:40am, and all 3 this morning--did not have a "Verify finished" line. And sure enough, those are exactly the 4 archives listed as "Verified 13 days ago" in the status window.

That's exactly what happened; the verify (and one merge) action never completed.

This is another side-effect of the monitor application process going to sleep while the computer is still running. I've identified this particular code path and have a fix, which will appear in 2.0.0b14.

Manual verifications on all 4 finished successfully as well.

Yes, because these commands were started while the monitor was awake.

Now, I don't want to disable status warnings in the menubar icon; nor do I want to "forget" the archive, but the red dot bugs me, and might numb me someday. So please consider allowing some customization there in the future. Thanks!

Well, it's a good thing you pay attention to those red dots, because this time it really was a problem!

There is an issue with archives that are captured/verified very infrequently—or never—in that they acquire a red dot that never goes away. I have this situation on my server. I have an archive with a capture of my emergency recovery volume. (Should my emergency recovery drive ever die—and I've replaced it twice now—I can quickly have it ready to use again with a single restore.) Since the emergency volume never changes, it never gets recaptured. So eventually, I get a red dot warning me that it hasn't been captured in a while.

I have a feature on the to-do list that will add a "Forget Capture Status Until Next Capture" option to the status window, so you can ignore that particular warning without forgetting about the entire archive.

Bruce Giles wrote:I don't think Apple could have made it any more subtle. I much preferred the old barber-pole pattern, but I realize that QRecall is just using what Apple provides there.

In the next iteration of the UI I was contemplating using the even-smaller progress indicator. I might have to include a free pair of reading glasses with every copy of QRecall.

So, just out of curiosity, what icon was supposed to be there? I went back and looked at QRecall 1.2.3.8 which is still running on one of my other computers, to see what it did. In that case, the top level item is the name of the computer, as defined in the Sharing panel of System Preferences. It happens to be an old iMac, and the icon is an iMac icon, so that's appropriate. But in b13, the top level item isn't the computer's name, it's MY name. In fact, the computer name isn't even in the list. The next level below my name is the name of the hard drive. So what should the top level icon be -- a human silhouette, or something like that?

At the top level of a QRecall archive are the owners. An owner is defined by an identity key and can be synonymous with you, the system, or the identity key. Everything captured using that identity key is contained within that owner.

The owner's icon is the "computer" icon from your system. The name is defined in the QRecall > Preferences > Identity Key. When you first enter your identity key, the name defaults to your computer's display name. You can change it to anything you want to identity the source of the captured items. Also note that you can later change the name of the installed identity key at any time, should you decide you want something else.

I spent some more time with this problem yesterday and I've determined that the legacy function I was using to obtain static system icons (like the icon for your computer system, the default icon for hard drives, and so on) no longer works reliably when the process is running as root—which is exactly what the QRecallHelper runs as during a capture. The call doesn't fail, but instead returns a completely blank icon, which QRecall (lacking any artistic judgement) dutifully captures.

Apple has provided a modern replacement for this function, but it's only available to processes running as the user and within the user's GUI bootstrap; QRecall capture processes don't meet either of these requirements.

I think I've developed a workaround, which I'm testing now. Unfortunately, if you've captured items and have a blank icon you're going to have to wait for those blank icons to get replaced with updated ones, which should happen during subsequent captures and merges.

But they will if the desktop computer is connected to a UPS (and monitoring the battery state via a USB cable), right? The release notes seem to indicate that.

If your UPS system has the software drivers and hardware required to communicate the status of the UPS to the power management system, then the answer is yes; QRecall will treat a desktop machine running off a UPS exactly as it would a laptop running off of battery power.

I was going to argue back on this one. I was absolutely certain you had used the word "scrap" in several places, in which case, "scrapping" would be appropriate. I went back and looked again and couldn't find it. Then I read ahead and discovered that Ming-Li Wang noted it as well, and you've apparently fixed it.

George Orwell would be proud.

I have to say I prefer "scrap" and "scrapping" to "scrape" and "scraping". I think of "scrap" as a synonym for "destroy". For example, "The wrecked car was scrapped." I don't think of "scrape" that way. But it's really not a big deal either way. I can learn to live with it. If I really have to.

I really wanted to make a statement about what QRecall was doing (scraping off items), not insinuate that the items being deleted were rubbish (scrap).

Ming-Li Wang wrote:I thought one could get the same result by simply clicking on the Next or Schedule column first before clicking on the Archive column. No?

Sadly, no. The sorting is relatively stable, so it might work until the table was updated again. But the next time the status of an action changes the entire table is reloaded, and the sub-order of items will change in unpredictable ways.

I thought as much, but didn't speak because English is not my first language.

English is my only language and I still find it confusing.

The cause for confusion probably lies in the beta release notes, however. In the section titled "Excluding, Ignoring, and Scraping Items", "scrap"--not "scrape"--is used numerous times as a verb.

LOL. I checked the interface so carefully, and completely missed this in the release notes!

People with dyslexia should not be allowed to write.

Fixed. (At least I'm pretty sure. Someone else will need to check it.)

Bruce Giles wrote:

When capturing, the progress bar in the activity window goes to full (100%) immediately at the start of the capture, and then it never changes after that. I have this same problem in the current release version of QRecall also. (Just as I was getting ready to post this message, an action started another capture, and this time the progress bar seems to be working correctly. Is that something you fixed in b13, or have I got some kind of inconsistent thing going on here?)

That's not 100%, it's "indeterminate." OS X progress views have an "indeterminate" mode to indicate that the amount of progress is unknown or unknowable. When the look of OS X got "Yosemited." the familiar marching barber-pole pattern was replaced with an (very) subtle gradient that drifts slowly across the screen. On small controls, it's nearly indistinguishable from the progress view at 100%.

When QRecall begins a capture operation it starts two threads: the main capture thread and a secondary "estimate" thread. That second thread tries to quickly rattle through the directories and change history in order to estimate the amount of work the capture thread will actually have to do. Until the estimate thread is done, the progress indicator is set to "indeterminate." Sometimes the estimate can be made very quickly, but in other cases the estimate will take nearly as long as the actual capture, in which case you might not see the progress indicator move at all.

My old backup drive was getting a bit flaky, so I bought a new external drive, and set up new archives for beta testing. I have not yet tried to convert my old archives. When i opened the new archive window for the first time after the first capture, it opened in icon mode. There was a blank icon with my name under it. I had to double-click the blank area to open up a new view showing the hard drive, and then I got an icon on the drive. The lack of an icon to go with my name was disconcerting -- at first I didn't even see it and thought the archive action hadn't worked. Did you intend to have some kind of icon there, or was it supposed to be blank?

To state the obvious, that's a bug. I'm still not sure where the problem is, and I'm beginning to think there are several bugs. In some cases it appears that the icons for the volume and computer aren't being captured properly. In other cases, the static system icons for a volume and the computer that are supposed to display when a captured icon isn't available aren't loading for some reason. Its been difficult to resolve because neither of these problems occur on my system with any regularity. Regardless, it's on the short list of bugs to look into. The issue is entirely cosmetic, as the icons captured/displayed for items in the archive aren't used to recall them.

Will the capture assistant be updated to support the new event conditions? I like the new conditions. I'm testing the one that waits to back up my virtual machine folder until after VMware Fusion quits, and so far, it seems to be working.

I consider most of the new action schedules and conditions to be power-user features, so I don't have any plans to add them to the assistant—which is really just to get you up and running with QRecall as quickly as possible. The assistant in 2.0.0b13 does add the new power state conditions to the actions it creates so that most will stop automatically if the power capacity of the battery drops below 20%. (On desktop computers, these conditions will never trip.)

Feature request: Two-column sorts of the Actions window. I'd really like to see actions sorted first by the Archive column, and then by either the Next or the Schedule column.

I'll add that to the list. Most of the UI enhancements are being addressed in version 2.1, so I'll probably get to it then.

In the release notes, "scraping" should be spelled as "scrapping", with two "p"s. Mentioned so you can get it right in the help files.

I bow to your superior skills in proofreading, but I have to call you out on this one. It's "scraping" (one 'p'), the noun form of "scrape": "to push or pull a hard or sharp implement across a surface or object so as to remove dirt or other matter..."

But it really doesn't matter. You can think of the new feature as "picking up scraps" that have already been captured, or "scraping off" items you don't want to keep on your hard drive.