Bruce Giles wrote:Do I understand correctly that this only happens with folders containing applications?

To be honest, I'm not sure. I've only encountered this problems with applications, but I suspect that it might be related to bundles/packages in general. Again, I haven't been able to get a definitive reason from Apple as to why this is happening so I can't say for sure.

Bruce Giles wrote:Is this a good scheme, or am I missing something?

No, you didn't miss anything. It's a good scheme, but I'm working on a better one.

I current have experimental code that automatically throttles the shifted quanta detection up or down based on past performance and available resources. QRecall will randomly check blocks looking for shifted data. If it finds some, it will increase the amount of time it spends looking for more shifted data. If the data being captured doesn't contain any shifted data, it will throttle the shifted quanta detection back for better capture performance.

If this proves successful, I will replace the shifted quanta archive setting with a simple on/off setting, or possibly eliminate the setting altogether.

Also, regarding the two compression sliders, suppose capture compression is set to "faster larger", and compact compression is set to "slower smaller". Would a capture, followed by a compact, produce the same results as if I had set the capture compression to "slower smaller", and then not performed a compact?

Correct. The capture compression level is applied during the capture. If it is the same as the compact compression level, then the compact will not recompress the data added by the capture.

My advice: Unless you are running out of disk space, leave all compress settings off (faster larger). Compression can dramatically reduce the size of an archive, but it comes at a huge cost in performance. If you have free disk space, use that up before using compression.

Bruce Giles wrote:I've noticed that when QRecall displays files and folders captured in the archive window, it seems to be able to go only one layer deep when reporting folder sizes. That is, a captured folder that contains only files will have a folder size displayed, but a folder that contains other files AND FOLDERS, will display only "--" for the size.

Is this correct? Any way to get QRecall to report the correct size on all folders?

It's a bug (sort of). I plan to replace this with a mechanism that pre-calculates the size of folders during the capture so that the correct size of folders and packages are always displayed.

Right now, QRecall has to calculate the size by recursively adding up the size of all enclosed items. But that information is only read in when the contents of subfolders are displayed in the browser. If you expand every nested subfolder, then cause the top-level folder to redisplay (by scrolling or collapsing/expanding it), the correct size is displayed.

Bruce Giles wrote:When I try to archive my entire home folder, however, which is about 16 GB in size, it proceeds only so far (varies between about 2 GB and 6 GB) before I get a message that QRecall has lost connection with the process. Checking the logs shows that QRecallHelper has crashed.

This problem is due to a bug in Mac OS X 10.4. I am working with Apple to develop a workaround, but there isn't one yet.

The problem is with Apple's Launch Services system. This is the part of the OS that identifies objects on a file system. Given a file or folder, it will tell you what kind of item it is, what its localized display name is, what applications will open it, what icon it should have, and so on.

QRecall uses this facility to capture the icon of every file. However, if one calls Launch Services to obtain the icon of a bundle and that bundle is in a folder which has never been opened in the Finder, occasionally that call will crash:

Thread 1 Crashed:

0   ...ple.CoreServices.CarbonCore 	0x90b9e3fc CSStoreGetUnit + 56

1   com.apple.LaunchServices       	0x9193ed00 CSStringCopyCFString + 108

2   com.apple.LaunchServices       	0x9194fe90 _LSCopyBindingInfoForItemRefInfo + 1336

3   com.apple.HIServices           	0x91848fc0 ISBuildBaseImage() + 1480

4   com.apple.HIServices           	0x91848880 GetImageForFSItem + 528

5   com.apple.HIServices           	0x91848544 GetIconRefFromFileInfo + 568

The solution is annoying, but simple: Open the folder containing the offending item in the Finder. When the Finder displays an item for the first time it does something to update the local .DS_Store file and the Launch Services database which permanently fixes the problem for that item.

For example, when I install a fresh OS I find I must open up the Applications and Utilities folder once to display all of the applications it contains. Once I do that, QRecall can capture the /Applications folder without any problem.

If you find it difficult to locate which item(s) is having problems, create a test archive and manually capture folders one at a time until you find the one that crashes.

As for why your QRecall application is crashing, I'll have to investigate that. The crash appears to be associated with a window. Does it crash when you open a specific kind of window (like the log or actions window) or when you open a specific archive?

Al Pawlowski wrote:Is there a way to mark a file so that a merge would not delete it, even if deleted in layers being merged? If not could some kind of marking, say a file name additive like [ ] around the nam, be put into your "to do" list?

Is your primary concern that you want to make sure you keep the last copy of each file in the archive?

Trying to merge two layers (where a file exists in the earlier layer but doesn't exist in the latest layer) but keeping the deleted file doesn't fit the model used by QRecall. A layer is a snapshot of the what files do, and do not, exist at a single point in time. The only way to keep the older copy of the file would be to maintain a second layer -- which is exactly what you already had before the merge. So the short answer to this problem is don't merge.

I've considered this and similar problems in the past and have been toying with a number of possible solutions. But I haven't come up with any yet that are both simple and keep the layer information consistent.

Here's my suggestion: If you have a group of files that you want to keep the last version each day, use two archives.

Create one archive for your active documents that you capture on a daily or hourly basis and a second for everything else. For the first, have a rolling merge keep daily layers for that archive going back as far as you want (30 day, 6 months, ...). Unless you're working on large multimedia files, keeping a year or more of daily backups of your documents shouldn't take up too much space. Because of QRecall's efficiency, a layer with no changes takes up virtually no storage.

Use a second archive to store everything else on the volume. That can be captured daily or weekly, and you probably only need to keep a few daily layers before merging older layers into weekly or monthly layers.

This will let you restore your OS or applications folder with a fairly recent version should something happen to it, while being able to recall that last version of the day of any document during the past year.

I realize that I could have a copy of old "not being used" files where I was not likely to mess with them, but that is not as easy as keeping all my related files together. If I did have such a copy, it would be nice if recall had a marker to mark files you do not want captured that are in a directory that you do want captured.

When creating a capture action, you can add whatever files or folders you want to the filter section. Those items will be ignored (treated as if they did not exist) during the capture. This lets you exclude whatever you want from regularly scheduled captures.

By the way, great program concept and I am doing a capture right now to a FAT32 networked attached (NSLU) USB drive that seems to be working fine.

QRecall should be able to use an archive on virtually any file system that correctly supports file sizes larger 2GB.

Disclaimer: I've never tested QRecall on an SMB mounted volume and have no idea if it will even work. QRecall was designed to work with HFS+ volumes, although I'm open to getting it work with other filesystems as time permits.

With that out of the way, I'd be very interested in getting a copy of your log file (~/Library/Logs/QRecall/QRecall.log) that covers the time both for successful and unsuccessful captures. You can send them directly to james@qrecall.com.

I can imaging all kinds of things that might be going wrong. Looking at the logs will help me narrow down the possibilities.

One item on my to do list is to add a drawer to both action document windows and the main Actions window that would show a list of upcoming run times. The drawer in the action document would show the schedule just for that action, while the Actions window drawer would show a complete list of all upcoming run times.

Try this:

Open the actions windows in QRecall (Windows > Actions, or Command+1). Click on the Next column header. This will sort the actions table based their next scheduled execution time, essentially giving you a schedule of when actions will start.

That's typically the view I leave my Actions window in. Maybe I should make it the default.

Gary,

F25s are fun to own, even if you just use it to get the mail.

A rolling merge is close to what you want, but has one tiny hiccup; Any layers created today are considered part of the "recent" tier (even if most recent is set to 0) and are never merged.

To get what you're asking for, create a simple Merge action instead of a Rolling Merge action. Set the merge criteria to

From: 0 layers after first layer
Through: 0 layers before last layer

That action will merge every layer in the archive into a single layer. Schedule it run immediate after your daily capture.

I'm glad you enjoy the app.

Hello again,

Thanks for the log file. For some reason your QRecall preferences folder is owned by root so the QRecall application couldn't create a new Actions sub folder.

To fix it, open up the Preferences folder in your home Library folder. Find and select the QRecall folder then choose File > Get Info. In the Info window, expand the Owner & Permissions details, click on the padlock, and change the owner of the folder to garyporter. That should fix the problem.

In the mean time I'll investigate how you ended up with a QRecall folder owned by root.

I assume that this is your first installation of QRecall?
It appears from the log that the first thing you did was to authenticate QRecall to use administrative privileges, and then you ran a capture.
I'll repeat that sequence here and see if I can reproduce the problem.

Be sure to let me know if you have any other issues.

Cheers,

Hello Gary,

This problem showed up in 1.0.0(37), but I thought it was resolved. Could you e-mail me your log file (~/Library/Logs/QRecall/QRecall.log)? You can attach it and send it directly to james@qrecall.com.

A substantial number of log messages were added to diagnose what the problem might be. Hopefully it will explain what's going on.

Thanks,

More Info:

Apparently, a number of people are reporting problems accessing large files (>2GB) on a shared USB drive via an Airport base station.

http://discussions.apple.com/thread.jspa?messageID=4379916
http://discussions.apple.com/thread.jspa?messageID=4459563

It appears that the highest rate of success is when the USB drive is formatted using HFS Extended and the computer is connected via AFP. So my suggestion would be to make sure your drive isn't formatted FAT32 and check to see that you are not connected to the base station using SAMBA or some other protocol.

Also make sure that you've applied the latest firmware upgrade.

If you can verify a 2GB archive when the drive is directly connected to your Macintosh, and that same archive fails to verify when connected to the base station, you'll need to complain to Apple.

Hello Tobias,

How large is "large"? Is the archive larger than 2GB? 4GB?

Try verifying the archive while it's being shared by the AirPort base station and see if that's successful. If it isn't, remove the USB drive and plug it directly into your computer and try the verify again.

Steven M. Alper wrote:Then it's unclear to me why you would run an unattended verify.

Peace of mind.

The verify command was intended to detect corruption or damage caused by something external to QRecall. When it's done, it guarantees that the data in the archive is readable, valid, intact, and usable.

QRecall is ultra-paranoid about data. Every byte of data in an archive is protected by a checksum, which is verified whenever a block of data is read. The file records all have multiple interlocking verifications to ensure that the directory structure never gets out of sync. So (baring a software bug), it's essentially impossible for a capture or compact operation to add to the corruption of an archive. As soon as any inconsistency is found, the action would stop dead in its tracks.

The verify action is, however, more thorough. A capture only checks the data needed to accomplish the capture. A compact only checks the data that needs to be moved. A verify tests everything, from top to bottom.

So if you had a hard drive that suddenly developed a bad sector, or the disk had cross-linked data blocks, or some malicious program wrote a single zero in the middle of a 300 megabyte archive, a verify would find that while a capture or merge probably wouldn't.

Right now I have a Verify action set to run prior to a Compact action. But if the Compact action in essence will check for the integrity of a file, why would this make sense?

It makes more sense to run a verify following an action that changes the archive. That way you can immediately verify that, following the change, the structure of the archive is still OK.

I do it here because QRecall is still a work-in-progress. Over the past year I've introduced a couple of bugs that caused a capture or merge to vandalize an archive. Regular verifies help me catch those quickly. I also had one of my RAID drives die; the verify caught it before the OS or my RAID monitoring software did!

Steven M. Alper wrote:1. Would it be possible to add a condition which would prevent an action from occurring if a previous verify failed? It would be nice to be able to automatically stop actions which could potentially further corrupt a damaged archive.

That's not necessary. All QRecall commands constantly perform data integrity checks. If a command encounters any inconsistency in the archive data it will stop immediately. A verify is not necessary to determine this.

Once an archive is found to be inconsistent, flags are set in the file headers so that QRecall knows that the archive is bad. At that point, the only two commands you can perform on that archive are Reindex and Repair.

(Actually what happens is that every archive is marked as "bad" as soon as an action begins. If the action completes successfully and finds no inconstancies, the flags are set back to "good" and the archive is closed. If anything interrupts the action -- I/O error, crash, power failure, etc. --- the archive is left with the "bad" flag and will need to be repaired before it can be trusted again.)

2. What about adding a "repair if damaged" step to the verify action?

Repairing requires a number of decisions that should be considered carefully before running it and may have consequences that need to be reviewed once it's done. Running a repair automatically could hide all kinds of serious problems.

Besides, any backup system that needs to automatically repair itself from time to time is simply broken.

3. I'd love it if you could add Growl support so that completed actions and verification reports could be accessed more easily than launching the app and checking the logs.

That's been on my long list of features to add for months.