Ralph Strauch wrote:I?ve had a couple of strange backups this week, with qrecall capturing what seemed like an extreme amount of data almost certainly from files that had not changed.

Ralph,

Thanks for the diagnostic report. Out of curiosity, I wrote a script to extract the total capture action time, number of folder changes, total amount analyzed (new+changed files), and the amount of new (unique) data added to your archive.

Legend:
Circles: time the capture took to complete
Squares: number of changed folders. The spikes are an artificial number indicating where QRecall ignored the folder history and performed a deep scan of the filesystem
Orange bars: The amount of data (in GB) captured (read from new or changed files)
Red bars: The amount of unique data (in GB) added to the archive

As you can see, the long capture times correlate, pretty closely, with either (a) a deep scan of the filesystem or (b) an inordinate amount of captured data. Some of the longest capture times naturally occurred when QRecall captured the most data (the biggest four were 8GB, 12GB, 18GB, 59GB, and 139GB).

The real question is why QRecall is capturing all of this data when you believe that not much data has changed. A few of the longer captures are clearly legitimate. The 12GB and 18GB captures, for example, captured 9GB and 8.5GB of new data (respectively). So in those instances, there was at least 8GB of new data that had to be added to the archive.

The suspicious captures are the 59GB (6GB or 10% new data) and the 139GB (2GB or 1.4% new data). In these cases, it would appear that a substantial amount of the data did not change but was recaptured because QRecall though it might have changed.

Why QRecall thought the files change is anyone's guess at this point, although you could compare the file info for the files that were captured with that of pervious layers for clues. Any change in the file's creation/modification date, length, name, etc. will cause QRecall to recapture that file's data.

Another, often overlooked, cause for recapturing a file is renaming a folder. Let's say I have a "Working VMs" folder that I rename "Project VMs". QRecall sees this as a deleted folder and a new folder, which it then captures as if they were all new files. Of course, 99% of the data in the "new" folder will be duplicates to what had already been captured in the "deleted" folder, but the files are still recaptured in their entirety, which takes time.

In conclusion, I don't see anything amiss, from QRecall's perspective. The question is what's touching or renaming files that might cause QRecall to recapture large swathes of data.

Ralph Strauch wrote:Please confirm that this admonition applies only to Restore and not to Capture.

This issue applies to both restore and capture of a bootable OS X system volume. Note that it does not apply to volume containing the QRecall archive, and it doesn't matter (as much) if the volume just contains document files.

The files in a bootable OS X system require very specific ownership, permission, attributes to be set or it will not function. When QRecall captures a bootable OS X system, it records these properties in the archive. When it restores the files, it replicates the original file's ownership, permissions, and all attribute properties.

On a volume set to "Ignore Ownership and Permissions," the filesystem neither reports nor honers the ownership, permission, or attributes of files. When queried, all files appear to belong to the user that's requesting the information. All attempts to change the ownership, or set certain attributes, are ignored.

If you attach a bootable OS X system volume to another computer, and the volume is set to "ignore ownership and permissions," the volume will be captured but the metadata required to make it a bootable OS X system volume will have been lost. Similarly, if you take a successfully captured OS X system volume and restore it to a volume that ignores ownership, the necessary ownership and attribute will not be restored (because the filesystem will ignore them) and the volume will not be bootable.

Which brings us to a common problem when restoring a bootable volume, and why I posted the original comment. If you (a) capture you startup volume, (b) format a new drive, (c) attach it an external volumes and (d) restore your startup volume to the new drive, you have to be careful that the newly formatted volume didn't default to "ignore ownership," which is typical for external volumes. The lastest version of QRecall will warn you that you are recalling items that were captured on a volume that honors ownership and permissions, but you are now attempting to restore them to a volume that ignores it.

Sidebar: the "ignore ownership" attribute is not recorded on the volume. It's a flag (in a database) maintained by each system. When you attach a volume, OS X looks up the volume's identifier in its database and applies the "ignore ownership" flag based on that information. Thus, a volume that "ignores ownership" on one system might honor ownership when plugged into a different system. Similarly, if you wipe and reinstall your OS, or boot from another volume, the "ignore ownership" setting for your volumes could change.

I hope that this adequately explains the situation.

Bruce,

So far, I haven't run into any problems. I'm running more tests now, and hope to have a definitive answer this weekend. If I find anything, I'll push out a new version.

Ralph,

It does, indeed, sound like the capture process was stuck. I can't think of any normal activity that would take that long to finish, even over wireless on a really large archive. The diagnostic reports probably won't help too much, as I really need to see a sampling of the process at that time, but I'll look for anything suspicious.

As for the iMac, the diagnostic report will include all of your active actions, so I'll be able to tell you what's going on there. That is, just as soon as I get back to the office, which will be a few days from now.

Bill,

Every layer that contains your file represents a complete and separate copy of that file. Until you merge layers, you already have every "previous copy" of that file.

Use the bottom layer shader in the browser to rewind your archive. With the recent layers hidden, you'll see the earlier copies of your file. To retrieve an earlier version, use the recall/restore command or just drag and drop it where you want it.

I suggest looking at the Introduction section of the help (Help > QRecall Help > Guide > Introduction) and then follow the link to Understanding Layers.

Remember that QRecall does not copy files. It captures only unique data. So it's not possible to "force a full backup"; QRecall would see that all of that data has already been captured and would do nothing.

David Ramsey wrote:

James Bucanek wrote:

The "Show Deleted Items" view option was not turned on.

Hm. It was turned on. Why should that make a difference?

When you enable "Show Deleted Items" it changes the way folders/volumes are restored. The restore action restores not only the captured items, but all previously deleted items still in the archive. This can pollute the OS with tens of thousands of previously deleted files, which often make the resulting system unusable.

This was probably the issue, and also why the restore took so long. In the next version of QRecall, you'll receive a stern warning if you attempt to do this.

After formatting the new drive, and before restoring, make sure the "Ignore ownership and permission" option was not checked.

Checked where?

Select the volume in the Finder and choose Get Info.

I assume you restored the volume by holding down the option key and choosing Archive > Restore Volume To... (or by dragging the archive volume icon, dropping it into the restore volume, and choosing the "Replace Volume" option when the dialog popped up).

No; I backed out to the first level of the archive, right-clicked on the volume, and selected "Restore". Is that valid?

That was probably OK. The plain vanilla Restore command will restore the captured volume to its original volume. After reformatting a drive, however, the volume appears to be a new volume and QRecall might not recognize it as the original volume. In this case, the Archive > Restore Volume To... is the command of choice.

Did you try blessing the volume after restoring it?

Shades of System 9! Not even sure how I'd do this on OS X...but no. I didn't have any way of mounting the volume.

There's a bless terminal command, but I don't think that is your problem.

David Ramsey wrote:1. My backup currently has 57 layers. Is that an unusually high number? Should I merge more often?

The number of layers will have some impact on performance, but doesn't fundamentally change how files are restored. As far as QRecall is concerned, there's no difference between 2 layers and 200 layers. And for the record, I have archives with over 500 layers that restore just fine.

2. What's going on when the barber pole is animating?

There's a quirk that affects very large restore actions. There's a separate process that estimates the amount of data that's going to be restored. It takes awhile to generate that estimate, which is why you initially see the barber poll. The estimate process, however, can underestimate the amount of data to restore. Once the restore has restored the amount of data in the estimate, it switches back to the barber poll because at this point it doesn't know how much data remains to restore.

3. Any ideas on why a whole-volume restore didn't work?

I don't know. In theory, you shouldn't have any problem restoring a bootable OS X volume. Here's some things to look out for:

The "Show Deleted Items" view option was not turned on.

After formatting the new drive, and before restoring, make sure the "Ignore ownership and permission" option was not checked.

I assume that both the capture and restore were authorized to use administrative privileges.

I assume you restored the volume by holding down the option key and choosing Archive > Restore Volume To... (or by dragging the archive volume icon, dropping it into the restore volume, and choosing the "Replace Volume" option when the dialog popped up).

Did you try blessing the volume after restoring it?

Charles Watts-Jones wrote:While back-ups run at an acceptable speed, verify operations are terribly slow at around 6.5 MB/min.

6.5 MB/min is dreadfully slow. Are you sure you don't mean 6.5MB/sec? That's about what I would expect from WiFi.

Verifies are I/O bound. If you have 100Mb ethernet, connecting via ethernet will give you about 10MB/s. If you have Gb ethernet, you should get about 80-100MB/s.

I don't know what your topology is, but consider having a different computer verify the archive. I have several laptops and an iMac that all capture via WiFi, but I use the server that's connected directly to the shared RAID to perform all of the weekly verify and merge actions. Everything is fast.

Charles Watts-Jones wrote:When using an external drive QRecall was able to back-up when I was logged out at night. Is it possible to arrange this with a NAS (which runs 24/7)? Or must I leave myself logged in 24/7 too?

It depends on how the volume mounts. If the volume mounts as a physical hard drive, QRecall will mount it before starting an action. If the mounts as a remote/network volume, then that requires an account and password, which requires the keychain, which requires that you be logged in and your keychain unlocked when the action runs.

A little experimentation should tell you which method works.

Testing the NAS I made a new archive on the NAS. I never used it and have deleted it. Status Window remembers it and lights its red button to show that the archive has never been used. How do I get Status Window to 'forget' that archive?

Control+click/right-click on that item in the status window and choose Forget from the pop-up menu.

Ralph Strauch wrote:I haven't been compacting because I was under the impression that qrecall would use empty space in the archive as it became available, ...

QRecall does (normally) reused the empty space in an archive. Over time, however, these bits of unused space become more and more fragmented. Eventually, QRecall must keep track of hundred of thousands, if not millions, of tiny, empty, records. It's this overhead that you're encountering.

Compacting consolidates this empty space by relocating active records so they are adjacent. It also rearranges records so that like records are closer together, which makes accessing the archive more efficient.

Is there any simple way tell how much empty space there is in an archive?

Open an archive and make sure nothing in the item (or layer) browser is selected. Open the Info window and it will show you a summary for the entire archive, including the amount of unused space (if it's been recently calculated).

But the problem here is not the amount of unused space, it's the amount of fragmentation, and the browser stats won't tell you that. You can infer the amount of fragmentation by looking at the size of the fill.index file inside the archive.

Ralph,

Thanks for the samples and the diagnostic reports.

In the first report and sample you sent, something definitely seems amiss, but I'm at a loss to explain why. The helper process appears to be stuck trying to access the record index file, but there's no obvious reason why it should be having trouble doing that.

In the second sample, QRecall is actually hard at work. It would appear from the log that your archive has accumulated a lot of empty space. QRecall keeps track of the unused portions of your archive and reuses it. As the number of unused areas increases, so does QRecall's overhead. It must periodically stop and clear the unused space that has accumulated?QRecall deliberately erases all deleted data from an archive. This is what it was doing when you made the second capture. This maintenance can occur at any time during a capture and it would appear that the capture is "hung up."

This also exacerbates another problem. If you terminate (force quit) a QRecall process, then next action has to auto-repair the archive before it can proceed. Among the auto-repair tasks is to re-erase all of the empty space in the archive. In your logs, I'm seeing that QRecall it taking upwards of 20-30 minutes to accomplish this, which I suspect means there's a lot of empty space in your archive.

I would suggest reindexing the archive first: choose the File > Repair Archive command, open the archive, and then check the Use auto-repair and Reindex only options.

When the reindex is finished, open the archive and compact it (Archive > Compact). This will consolidate and recover all of the empty space in the archive and optimize the record order for better performance.

I highly recommend that you create a compact action and schedule it to run occasionally (say, once a month) for optimal performance.

Ralph,

Thanks for the diagnostic report. But as you might imagine, the log doesn't help much in determining why a process has become stuck.

If this happens again, it would help greatly if you could capture a sample of the process. A capture process runs as root, so this must be done through the command-line. Launch Activity Monitor and get the process ID (PID) of the QRecallHelper process that's running as root (make sure you have the All Processes filter set). Substitute that PID number for "PID" in the following command:

sudo sample PID 20 -file ~/Desktop/QRecallHelper.sample.txt

It will prompt you for your admin password, and then sample the process for 20 seconds. When it's done, send me the QRecallHelper.sample.txt file that's now on your desktop.

Jonathan Edwards wrote:It's good to know I'm not the only one with the keyboard focus issue in Column View. I thought I'd provide a screenshot of this issue all the same and in doing so have discovered what looks like a second bug:

"Folder 2B" is not being shown at all in this view. It was captured and resides in Folder 2, as shown in both the equivalent Icon View and List View screenshots.

That's strange. I'll look into it. A lot of the column view code is getting rewritten for the next version, so I'll see if something like this is still happening.

Also attached are two Layers View screenshots, as requested, showing the graphics corruption.

That's even stranger. It looks like the view is not being completely redrawn. Does the situation change/improve if you resize the window, particularly if you make the window smaller?

Also, please send a diagnostic report from QRecall. This will include a snapshot of your hardware configuration (particularly, your GPU).

Jonathan Edwards wrote:I see that changing the advanced setting QRCompactErasesExpiredDeletedItems to false prevents "deleted" items outside of the "Keep deleted items" setting from being permanently deleted by a Compact action, which is great.

Is there an equivalent setting for the Merge action please? Thus, keeping all "deleted" items whilst retaining the additional benefits of the Merge and Compact actions. A hidden alternative to a "Keep deleted items forever" option in the archive settings perhaps?

There is no "keep forever" option but, as I posted previously, you can set the keep deleted items time to 999 months, which is effectively "forever." Deleted items will be retained for 80+ years.

If you've done this, the QRCompactErasesExpiredDeletedItems setting will have no effect, because you won't have any deleted items that have expired.

On a separate note, there appears to be a file browser bug when using the arrow keys in column view:

That's a known bug.

I wonder if this is due to an incompatibility with the Macbook Pro's Retina display?

Nope. It's a quirk of how QRecall handles keyboard focus.

That may also account for the distorted 3D Layers view I'm seeing. Changing the Automatic graphics switching preference in Energy Saver has no affect.

This sounds like an unrelated issue. If you could send me some screenshots, that would be very helpful.

This is still invaluable software, thanks!

That's great to hear!

My only comment is to remember the immortal words of Roy, from The IT Crowd:

"Have you tried turning it off and back on again?"