I'm surprised you can't delete the QRecall archive, a document you wrote to the Time Capsule.

A time capsule runs a (slightly older) version of Apple's file server software, the same file service that powers personal file sharing on the Mac. There are some peculiarities when it comes to Time Machine files (the OS sets special "immutable" flags the prevent you from messing with the organization of the backed-up files.) But when you're accessing it as a network volume, you're just an AFP client. You should be able to delete any file you originally wrote.

The only thing I can think of is something in the permissions or ownership of the files has changed, or you're accessing the volume with a different user account than was used when the archive document was originally created. Knowing what the ownership and permissions for the archive bundle and files are may provide a clue. (ls -lae@ /Volumes/Path/To/Server/Archive.quanta)

Bob,

My guess would be VM storage. You can verify this by looking at the QRecallHelper processes in Activity Monitor. There will be two: run running as root and a second running with the UID of the user that installed QRecall. The one running as root is the capture action, while the second one is used to gather metadata about the files being captured.

The current version of QRecall still uses the legacy Carbon API for file services. There's a particularly annoying memory leak with some of the Launch Services APIs, which causes the second (not root) version of QRecallHelper to use excessive amount of RAM when it has to process a lot of files ... like when you're taking an initial backup of a large volume.

What happens is that the second QRecallHelper process leaks memory, which causes the VM backing store files to grow, eating up disk space on your system volume. Try stopping the capture and just restart it. It will pick up where it left off (if you want to be extra neat, merge the two layers when the second capture is finished). If you see the size of the VM store drop dramatically after stopping the action, that's what the problem is.

On typical (incremental) captures, the amount of the memory leak is so small that no one notices, which is why neither Apple nor I have ever invested the time to address it. Plus, the next version of QRecall does't use the legacy Carbon APIs, so eventually it won't be an issue.

Rob Schumann wrote:Could you please clarify the following:

>> Make sure View ? Hide Invisible Items, Hide Deleted Items, and Hide Package Contents have been chosen.

I know, it's confusing.

If the menu item says "Hide Invisible Items," then invisible items are currently being shown. If it says "Show Invisible Items," then the invisible items are currently hidden.

When restoring most things, you'll want to hide deleted items, invisible items, and package contents. You'll have to show invisible items to dig into the /Library and ~/Library folders, and you might find other invisible items in your home folder of interest.

Rob Schumann wrote:The Applications folder will be there already, so do I just select the folder within the QRecall archive and 'restore' it over the top of the Applications folder on the target disk, or do I select the contents of the folder in QRecall and restore them inside the Applications folde on the target disk?

Hopefully, you've simply erased your boot volume and installed a clean version of Mavericks. This will help, because QRecall will still recognized the volume as being the one you previously captured. If not, you'll need to modify these instructions by manually selecting and recalling items, rather the restoring them—restore automatically locates the original item on the original volume for you.

You'll want to selectively replace just the items you've captured, leaving the Mavericks version of Apple's apps in place. Here's how you do it:

Download and install the QRecall app.

Preauthorize it to use administrative privileges.

Open your existing archive and navigate to the Applications folder.

Make sure View ⇒ Hide Invisible Items, Hide Deleted Items, and Hide Package Contents have been chosen.

Choose Edit ⇒ Select Existing Items.

Choose Edit ⇒ Invert Selection.

Choose Archive ⇒ Restore....

This will select and restore all of the applications that do not exist in your new Applications folder. If you're cleaning house, review the list first and unselect any applications you don't want to reinstall. Repeat with the Utilities subfolder.

I would suggest repeating these steps for each major subfolder in your root-level /Library folder. (You'll have to turn on Show Invisible Items to get to that.) You'll want to restore any Fonts not already installed, Preference Panes not already installed, Preference files not already installed, and so on. Again, review each list before restoring and consider unselecting really old items and folders/files that appear to belong to applications you're no longer using. If you guess wrong, you can always delete anything you restored or later restore anything you missed.

For the user a/c, I'm assuming that I need first to create the user with the same name as previously before restoring the account files. Again... do I then restore the account folder over the top of the newly created account on the target drive, or just the contents?

You'll want to simply replace your entire home folder. If you've created your new home folder with the same name, and it happens to have the same user ID (typically 501) as your original, the easiest solution is to select your home folder (inside the Users folder) and restore it.

If your new home folder has a different name, drag the contents of your captured user folder into your new home folder. Make sure you have Show Invisible Items turned on and include your ~/Library folder in that set.

If, for some reason, your new user has a different user ID than your previous account, then things get a little sticky. You can see what your new uid is by opening up a Terminal window and entering the id -u command. It should be 501. In QRecall, select your home folder (or anything inside it) and use the Inspector palette to find the owner. It should be yourshortname(501).

If they don't match, I'd suggest un-pre-authorizing QRecall and tell it not to use administrative privileges when restoring those items. QRecall will then restore those items using your new user's account. You'll have to ignore the thousands of warnings your going to get as QRecall complains that it can't completely restore the ownership and permissions of those files, but for the most part that's OK. It could be a problem for some special files (like the contents of the hidden ~/.ssh directory which requires very specific ownership and permission settings to function properly).

Lastly, do I need to boot from an external drive in order to do this (especially the user a/c)

No, but I'd suggest restarting immediately afterward, especially after restoring the root /Library folder.

Ralph Strauch wrote:I think that one or two of the failures you noted resulted from my disconnect a backup in process by inadvertently shutting the laptop lid. Sometime a backup in process seems to survive this, and other times it doesn't.

When the laptop wakes up again, it becomes a race condition between the TCP/IP stack trying to reconnect with the server and reestablish the session it had and the next filesystem request timing out. The filesystem tries to give the network stack time to recover, but it doesn't always happen that way.

Ralph,

Looking at the latest diagnostic report you uploaded, I see three additional capture actions, all of which failed because of I/O errors that occurred trying to read or write to the archive.

You said that you're access the archive on a shared network volume via WiFi. The three most likely causes of I/O errors in this situation would be:

(1) The physical drive is experiencing hardware/controller problems

(2) The volume directory structure of the archive volume is corrupted

(3) You're encountering network connectivity failures (you'r loosing the WiFi connection)

It's my understanding that you also capture to this same archive from other computers, without any problems, so 1 and 2 are unlikely, but I would still run a volume repair on the archive volume and restart the file server to just to make sure.

If the I/O errors are being caused by wireless network communications, then you can fiddle with that. Try changing the channel on the base station in an attempt to reduce interference, adjust the transmitter power, and so on. (A network scan utility is handy for finding out what channels other WiFi zones in your vicinity are using and pick a channel as far from those as possible.)

As an experiment, you can connect your MacBook to the base station using an ethernet cable and see if the captures are successful. That would duplicate the fileshare/network arrangement and eliminate only the WiFi as a variable.

P.S. Given the number of failed captures, I'd recommend running a verify on the archive first, and repair it if any anomalies are found.

Ralph Strauch wrote:The 1am scheduled iMac backup on 11/28 failed due to a permissions problem.

And I find that strange. Going back through your earlier log records, I find quit a number of QRecall actions that fail due to permission errors, or some variation thereof. The errors I see are "refnum," "permission," and "EACCES" errors. None of which make much sense, and makes me suspect the file server or the operating system's interaction with it.

I wasn't paying much attention to the computer on Thanksgiving and didn't notice these failures, which led to another 1am iMac backup failure on 11/29, logged as "archive file out of sequence," followed by a 7:30am failure on the MBP logged as "header file length invalid."

The earlier capture (on 11/28) had completed, but while the archive was being finalized, that permissions error prevented QRecall from correctly closing the archive.

Whenever QRecall opens an archive, it verifies that all of the files in the package are current with one another. Since the package index files couldn't be finalized on 11/28, the files were out of order and couldn't be trusted. Both the MacPro and the MacBook refused to use the archive.

Question: The logs on both computers told me that the index is incorrect and needs to be recreated. I just repaired the archive, since I've found that reindexing always seems to require a repair anyway, so the reindex step is just a waste of time. Does it ever make sense to do a reindex rather than just doing a repair?

Looking back through your logs, I can only see one Reindex operation, and that one was successful. There are many problems that a reindex won't fix (invalid data, bad checksums, and so on). However, there are a lot of issues, like an out-of-sequence index file, that the reindex will correct.

The advantages of a reindex are speed and the promise that no data in the repository.data file will be altered. The amount of improved speed is variable. If you're reindexing a file on a local hard drive, you might see significantly better performance from a reindex over a repair. If you're access the volume over a network or slow external drive, you probably won't see any advantage at all and a repair would be your best choice in all cases.

The second advantage is obscure. If you're super-worried about not losing any data in the archive, a reindex will not alter any data in the main repository.data file. A repair, on the other hand, will erase any data it doesn't like or doesn't appear to belong to any file (depending on what repair options you choose), which could result in loss of data.

Ralph Strauch wrote:I've had a couple of backups fail recently due to hash map problems -- failure to open the hash map in one case and failure to close it in another. In both cases I reran the backup and it ran without problems.

This could be caused by virus scanning software. As QRecall updates the index files in its package, the virus scanner will see it as a new file and attempt to scan it. Depending on the timing, this may prevent QRecall from relocating the file before the scan is complete. I've had a number of users encounter this, and the only solution (for the current version of QRecall) is to prohibit the virus scanning software from scanning the volume that contains the archive.

Several backups recently have reported an inability to open an email attachment for backup. The unopenable files were ones that the Sophos Quarantine Manager reported as malware, so this was a good thing. It's not something I've seen until recently, though.

You Quarantine manager might be preventing any software from accessing the file, which would include QRecall. I'd be very interested in seeing your QRecall log file where these events occurred. So at your convenience, please send a diagnostic report.

bill steer wrote:How Can I display all files in an archive, without folders, sorted by size ?

Bill,

There's no broswing option that will flatten the filesystem hierarchy so you can see all of the files in a single list.

You can, however, sort the browser items by size (in descending order). If you're displaying all layers, each folder will report its aggregate size (the total size of all of the items captured in that folder). The largest folders will appear first. Open up the biggest folders to find the biggest files.

Ralph Strauch wrote:I?ve had a couple of strange backups this week, with qrecall capturing what seemed like an extreme amount of data almost certainly from files that had not changed.

Ralph,

Thanks for the diagnostic report. Out of curiosity, I wrote a script to extract the total capture action time, number of folder changes, total amount analyzed (new+changed files), and the amount of new (unique) data added to your archive.

Legend:
Circles: time the capture took to complete
Squares: number of changed folders. The spikes are an artificial number indicating where QRecall ignored the folder history and performed a deep scan of the filesystem
Orange bars: The amount of data (in GB) captured (read from new or changed files)
Red bars: The amount of unique data (in GB) added to the archive

As you can see, the long capture times correlate, pretty closely, with either (a) a deep scan of the filesystem or (b) an inordinate amount of captured data. Some of the longest capture times naturally occurred when QRecall captured the most data (the biggest four were 8GB, 12GB, 18GB, 59GB, and 139GB).

The real question is why QRecall is capturing all of this data when you believe that not much data has changed. A few of the longer captures are clearly legitimate. The 12GB and 18GB captures, for example, captured 9GB and 8.5GB of new data (respectively). So in those instances, there was at least 8GB of new data that had to be added to the archive.

The suspicious captures are the 59GB (6GB or 10% new data) and the 139GB (2GB or 1.4% new data). In these cases, it would appear that a substantial amount of the data did not change but was recaptured because QRecall though it might have changed.

Why QRecall thought the files change is anyone's guess at this point, although you could compare the file info for the files that were captured with that of pervious layers for clues. Any change in the file's creation/modification date, length, name, etc. will cause QRecall to recapture that file's data.

Another, often overlooked, cause for recapturing a file is renaming a folder. Let's say I have a "Working VMs" folder that I rename "Project VMs". QRecall sees this as a deleted folder and a new folder, which it then captures as if they were all new files. Of course, 99% of the data in the "new" folder will be duplicates to what had already been captured in the "deleted" folder, but the files are still recaptured in their entirety, which takes time.

In conclusion, I don't see anything amiss, from QRecall's perspective. The question is what's touching or renaming files that might cause QRecall to recapture large swathes of data.

Ralph Strauch wrote:Please confirm that this admonition applies only to Restore and not to Capture.

This issue applies to both restore and capture of a bootable OS X system volume. Note that it does not apply to volume containing the QRecall archive, and it doesn't matter (as much) if the volume just contains document files.

The files in a bootable OS X system require very specific ownership, permission, attributes to be set or it will not function. When QRecall captures a bootable OS X system, it records these properties in the archive. When it restores the files, it replicates the original file's ownership, permissions, and all attribute properties.

On a volume set to "Ignore Ownership and Permissions," the filesystem neither reports nor honers the ownership, permission, or attributes of files. When queried, all files appear to belong to the user that's requesting the information. All attempts to change the ownership, or set certain attributes, are ignored.

If you attach a bootable OS X system volume to another computer, and the volume is set to "ignore ownership and permissions," the volume will be captured but the metadata required to make it a bootable OS X system volume will have been lost. Similarly, if you take a successfully captured OS X system volume and restore it to a volume that ignores ownership, the necessary ownership and attribute will not be restored (because the filesystem will ignore them) and the volume will not be bootable.

Which brings us to a common problem when restoring a bootable volume, and why I posted the original comment. If you (a) capture you startup volume, (b) format a new drive, (c) attach it an external volumes and (d) restore your startup volume to the new drive, you have to be careful that the newly formatted volume didn't default to "ignore ownership," which is typical for external volumes. The lastest version of QRecall will warn you that you are recalling items that were captured on a volume that honors ownership and permissions, but you are now attempting to restore them to a volume that ignores it.

Sidebar: the "ignore ownership" attribute is not recorded on the volume. It's a flag (in a database) maintained by each system. When you attach a volume, OS X looks up the volume's identifier in its database and applies the "ignore ownership" flag based on that information. Thus, a volume that "ignores ownership" on one system might honor ownership when plugged into a different system. Similarly, if you wipe and reinstall your OS, or boot from another volume, the "ignore ownership" setting for your volumes could change.

I hope that this adequately explains the situation.

Bruce,

So far, I haven't run into any problems. I'm running more tests now, and hope to have a definitive answer this weekend. If I find anything, I'll push out a new version.

Ralph,

It does, indeed, sound like the capture process was stuck. I can't think of any normal activity that would take that long to finish, even over wireless on a really large archive. The diagnostic reports probably won't help too much, as I really need to see a sampling of the process at that time, but I'll look for anything suspicious.

As for the iMac, the diagnostic report will include all of your active actions, so I'll be able to tell you what's going on there. That is, just as soon as I get back to the office, which will be a few days from now.

Bill,

Every layer that contains your file represents a complete and separate copy of that file. Until you merge layers, you already have every "previous copy" of that file.

Use the bottom layer shader in the browser to rewind your archive. With the recent layers hidden, you'll see the earlier copies of your file. To retrieve an earlier version, use the recall/restore command or just drag and drop it where you want it.

I suggest looking at the Introduction section of the help (Help > QRecall Help > Guide > Introduction) and then follow the link to Understanding Layers.

Remember that QRecall does not copy files. It captures only unique data. So it's not possible to "force a full backup"; QRecall would see that all of that data has already been captured and would do nothing.

David Ramsey wrote:

James Bucanek wrote:

The "Show Deleted Items" view option was not turned on.

Hm. It was turned on. Why should that make a difference?

When you enable "Show Deleted Items" it changes the way folders/volumes are restored. The restore action restores not only the captured items, but all previously deleted items still in the archive. This can pollute the OS with tens of thousands of previously deleted files, which often make the resulting system unusable.

This was probably the issue, and also why the restore took so long. In the next version of QRecall, you'll receive a stern warning if you attempt to do this.

After formatting the new drive, and before restoring, make sure the "Ignore ownership and permission" option was not checked.

Checked where?

Select the volume in the Finder and choose Get Info.

I assume you restored the volume by holding down the option key and choosing Archive > Restore Volume To... (or by dragging the archive volume icon, dropping it into the restore volume, and choosing the "Replace Volume" option when the dialog popped up).

No; I backed out to the first level of the archive, right-clicked on the volume, and selected "Restore". Is that valid?

That was probably OK. The plain vanilla Restore command will restore the captured volume to its original volume. After reformatting a drive, however, the volume appears to be a new volume and QRecall might not recognize it as the original volume. In this case, the Archive > Restore Volume To... is the command of choice.

Did you try blessing the volume after restoring it?

Shades of System 9! Not even sure how I'd do this on OS X...but no. I didn't have any way of mounting the volume.

There's a bless terminal command, but I don't think that is your problem.