QRecall

Greg,

Thanks for the suggestion. I've added it to the to-do list.

I'll probably implement this feature by performing an unbuffered double-read of all file data, and comparing the results. If they match then everything is good. If the data doesn't match, either read it a third time to settle the dispute or log an error.

Neil Lee wrote:Besides the report I'm sending right after I post this, is there anything else I can do to track down what happened?

Neil,

The report you sent doesn't provide any illuminating information, beyond the obvious fact that QRecall successfully captured your files at 11:35PM and again at 2AM. There's nothing unusual about the number files that were captured, the amount of time the capture took, or any similar anomalies.

When anything weird happens like that on my system, I start by repairing the volume using Disk Utility or <insert your favorite disk repair program here>.

If the files were inadvertently moved or renamed, rewinding the archive to that 2AM layer and searching for those files might uncover where they went. (Unless they were moved to the trash and you don't capture your Trash.)

Gary,

Thanks for that information. I haven't encountered that particular problem, but it makes sense. The dynamic library cache contains resolved copies of the various dynamic libraries in the operation system. As a sanity check, the cache remembers the modification date and the inode (nodeID) of each library file. Following a restore, the nodeIDs of the libraries will be different, causing the OS to ignore the version in the cache.

I need to investigate this a little. Given that the files in /var/db/dyld are cache files, it might make sense for QRecall to simply exclude them from the capture and let the OS rebuild them following the restore.

The man page for update_dyld_shared_cache says

This tool is normally only run by Apple's Installer and Software Update, as they are the only official ways for OS dylibs to be updated. But if for some reason you used another mechanism to alter an OS dylib, you should manually run update_dyld_shared_cache.

Restoring a startup volume from a backup would certainly fall into this category, but it's impractical for QRecall to try and run update_dyld_shared_cache tool following a restore.

Johannes wrote:(I run without authorising because the folder only contains my stuff). I was surprised because QRecall was able to capture the file without admin privileges.

This is not surprising at all. Your user account can read this file, and all of its attributes, because the file belongs to 'admin' and file's owner has read privileges. That's why QRecall can capture the item.

When it comes to restoring the item, it can restore all of the data and the original ownership (because it still belongs to 'admin'), but it can't set the group to wheel or grant wheel read or execute privileges. That's because the wheel group doesn't belong to admin, so setting the group would escalate the privileges for this file. Without authorization, QRecall doesn't have the authority to do that.

It seems all I lost is group and world access and the original date. And I have some finderInfo attributes out of midair:
com.apple.FinderInfo:

00000000  62 7A 79 20 51 6D 52 43 00 00 00 00 00 00 00 00  |bzy QmRC........|

Now that's probably a bug, or really a side effect of not having sufficient privileges to restore the file. The 'bzy' type is a special file type that tells the Finder that the file is "busy"?that is, it's being copied or whatever?and the Finder is not supposed to let you try to move, trash, or open that item until the process is finished. Normally the 'bzy' type is removed when the rest of the file's properties are restored, but that got interrupted because of permissions.

Johannes wrote:Great! Any timeframe?

Yes, I hoped to have it done by November.

Are you still interested in feedback for the current UI? Or are things already settled for the new UI, so we better not waste time on the old stuff?

The new interface is radically different, so there's not much point critiquing it. On the other hand, if there's any features you really like about the old interface that you'd like to see preserved, speak up now.

Johannes wrote:Can't find this string anywhere in the log.

Make sure you have the log detail level control moved all the way to the right. This message is considered to be "minutia" and is only displayed at the very highest detail level.

Johannes,

Both of these features are on the way, but both require substantial changes to the user interface code—which is being rewritten right now.

Johannes,

I forgot to mention this before; if you really want to have fun with file system events, there's a secret (well, not so secret now) setting for logging FSEvent records to the QRecall log. Use the defaults command tool to set QRLogFSEvents to true:

defaults write com.qrecall.client QRLogFSEvents -boolean true

With QRLogFSEvents set, QRecall will dump all of the individual change event records that it receives from the OS. If QRLogFSEvents and QRLogCaptureDecisions are both set, it will also dump the change history tree to the log before the capture begins. This tree represents the the composite folder structure which QRecall believes contains changes based on the FSEvent records it received. Basically, any folder outside of this tree is assumed to be unchanged since the previous capture.

Richard,

I know it's hard to phrase the question, because these values are difficult to describe—but here goes:

There are three significant size numbers in QRecall. (1) The Size column in the layers pane is the total logical size of every item captured in that layer. This is, conceptually, the total amount of data contained in that layer.

(2) The size value that you see for individual items and folders in the item browser pane is the logical size of the file item, or the aggregate size of all items contained in the folder. These are, approximately, the same values you should see in the finder if you did a Get Info.

So if you capture a single 60GB file every day, both the size of the layer and the size of the item in the browser will say 60GB. Neither of these sizes reflect the amount of data that's shared with other items or layers. That's a really labor intensive value to determine, and an even more difficult value to explain, so I've never added this calculation to QRecall.

(3) To see amount of data that was duplicate, look in the log. At the end of each capture, the log records the logical size of the items that were captured—this becomes the size you see in the layers pane. If you expand that log line (you might need to slide you details control over to the right), you'll see a breakdown like the one shown in the attachment. This will tell you how much of that 60GB is new data and how much was duplicate.

I hope that helps.

James Bucanek wrote:This appears in the log as "Filesystem change history ignored."

Johannes wrote:
Here are the file records; ...

Thanks, Johannes! As soon as I looked at those records, I knew exactly what was wrong.

Please download and install QRecall 1.2.0(25) alpha, which fixes this bug. The problem was a mismatch between the way extended finder information was captured and how it gets restored. I'm embarrassed to say that this was caused when I merged two branches of the QRecall project to create 1.2.0a24. Some recent optimizations in metadata handling got into a24, while others didn't.

The good news is that the errent code was isolated to the restore logic. All of the information in the archive is correct, and if you restore the same items using a25 they should be OK.

Now I know what the Dump-Command is for. Is this alpha only?

This command (and a few others) only appear in the alpha and beta releases, and are intended for debugging problems just like this.

What is the diference between Archive>Dump and File>Dump??

One of them works.

Could this be a left over from Big/Little Endian issue between Intel and PowerPC machines? When I created this file I was on a PowerPC. Now I am on Intel. Just a thought ?

It's a good thought. There were a few file system structures in 10.4 and 10.5 that didn't get swapped property during the early migration to Intel, but to my knowledge all of these have been fixed. For cross-platform compatibility, all structures are stored big-endien in the filesystem. So anything originally written using a PowerPC machine should be correct on the disk. The only possible problem would be Intel software that fails to swap the bytes when it reads the data.

Good catch!

Johannes wrote:

James Bucanek wrote:
Set QRAuditFileSystemHistoryDays to 0.95. That will force a deep scan once a day

On what base will QRecall do the maths? Once a day for each Archive? For each Action? For each machine?

It's based on how long it's been since a deep scan was performed for the items being captured.

Will I get a deep scan for Action 2 and 3 every time with the above setting?

Yes.

QRecall keeps track of how long it has trusted FSEvents to determine changes for each (top-level) item captured in the archive. During the next capture, it compares the scope of what's being captured to what's in the archive and determines the minimum common overlap between the two. It then compares the "FSEvent trust" dates and finds the largest interval.

Let's say QRAuditFileSystemHistoryDays was set to 1.0. Now let's say you captured '/' two days ago and '/Users/you/Document' two hours ago, both using a deep scan.

If you were to capture 'Documents' again, the difference in trust dates would be 2 hours, so no deep scan is required. If you captured '/Users/you' the difference would be 2 days, because the oldest capture that encompasses '/Users/you' in the archive is '/', which is now two days old.

Now instead of capturing '/Users/you', let's say you captured '/Users/you/Documents/Projects' an hour later, QRecall would not perform a deep scan because 'Projects' is encompassed by '/Users/you/Documents', and the latest capture of 'Documents' trusted FSEvents for 2 hours, so the aggregate trust time is now 3 hours.

So as you can see, it's kind of complicated, but it also can't easily be tricked into trusting FSEvents for longer then it should.

Johannes wrote:ls -la@ in recall folder and in original folder (no difference):

Well, the file got restored correctly. That's good news.

xattr -l in recall folder and in original folder (looks like the same difference like with some other old files):

00000000  50 49 43 54 47 4B 4F 4E 01 00 00 46 FF 51 00 00  |PICTGKON...F.Q..|

00000010  00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00  |................|

00000020



00000000  50 49 43 54 47 4B 4F 4E 80 00 00 46 00 00 00 00  |PICTGKON...F....|

00000010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|

00000020

Similar, but not exactly. In this case one file has Finder flags of 0100 and the original has 8000. The value 0100 indicates kHasNoINITs (an obsolete OS 9 attribute), while 8000 mean kIsAlias. (Are these backwards by any chance?)

It would be interesting to find out what finder information QRecall read, and captured, when it got the file in the first place. And since you're running an alpha version of QRecall, you can do that.

:

Open the archive with this file and choose Archive > Dump. In the data section, check "File Package" "File Package: Details" and "Scan by Package". Uncheck every other option in the dialog. Click Dump (it may take awhile, during which QRecall will be frozen). Open the dump file with a text editor and find the file records for the 'bergkristal1.pict' file.

Johannes wrote:Any idea why they became Aliases when restoring (recalling) them?

I have no clue. I've never seen that happen.

What do the files look like if you list then in the terminal (ls -l)? And if you have GetFileInfo installed, it would be interesting to see what their file attributes are, and how they differ from the attributes of the original.

Also, try duplicating them in the Finder and see if the Finder still thinks they're alias files.

Johannes wrote:Is there a terminal command to clear their extended finder info?

I though there was, but I can't find one. I know you can delete ACL with chmod, but I don't know of a utility that will let you arbitrarily delete extended attributes. I guess that's why I probably end up writing a program to do it.