QRecallDownloadIdentity KeysForumsSupport
  [Search] Search   [Recent Topics] Recent Topics   [Hottest Topics] Hottest Topics   [Groups] Back to home page 
Messages posted by: James Bucanek
Forum Index » Profile for James Bucanek » Messages posted by James Bucanek
Author Message
Hello again,

Thanks for the log file. For some reason your QRecall preferences folder is owned by root so the QRecall application couldn't create a new Actions sub folder.

To fix it, open up the Preferences folder in your home Library folder. Find and select the QRecall folder then choose File > Get Info. In the Info window, expand the Owner & Permissions details, click on the padlock, and change the owner of the folder to garyporter. That should fix the problem.

In the mean time I'll investigate how you ended up with a QRecall folder owned by root.

I assume that this is your first installation of QRecall?
It appears from the log that the first thing you did was to authenticate QRecall to use administrative privileges, and then you ran a capture.
I'll repeat that sequence here and see if I can reproduce the problem.

Be sure to let me know if you have any other issues.

Hello Gary,

This problem showed up in 1.0.0(37), but I thought it was resolved. Could you e-mail me your log file (~/Library/Logs/QRecall/QRecall.log)? You can attach it and send it directly to james@qrecall.com.

A substantial number of log messages were added to diagnose what the problem might be. Hopefully it will explain what's going on.

More Info:

Apparently, a number of people are reporting problems accessing large files (>2GB) on a shared USB drive via an Airport base station.


It appears that the highest rate of success is when the USB drive is formatted using HFS Extended and the computer is connected via AFP. So my suggestion would be to make sure your drive isn't formatted FAT32 and check to see that you are not connected to the base station using SAMBA or some other protocol.

Also make sure that you've applied the latest firmware upgrade.

If you can verify a 2GB archive when the drive is directly connected to your Macintosh, and that same archive fails to verify when connected to the base station, you'll need to complain to Apple.
Hello Tobias,

How large is "large"? Is the archive larger than 2GB? 4GB?

Try verifying the archive while it's being shared by the AirPort base station and see if that's successful. If it isn't, remove the USB drive and plug it directly into your computer and try the verify again.
Steven M. Alper wrote:Then it's unclear to me why you would run an unattended verify.

Peace of mind.

The verify command was intended to detect corruption or damage caused by something external to QRecall. When it's done, it guarantees that the data in the archive is readable, valid, intact, and usable.

QRecall is ultra-paranoid about data. Every byte of data in an archive is protected by a checksum, which is verified whenever a block of data is read. The file records all have multiple interlocking verifications to ensure that the directory structure never gets out of sync. So (baring a software bug), it's essentially impossible for a capture or compact operation to add to the corruption of an archive. As soon as any inconsistency is found, the action would stop dead in its tracks.

The verify action is, however, more thorough. A capture only checks the data needed to accomplish the capture. A compact only checks the data that needs to be moved. A verify tests everything, from top to bottom.

So if you had a hard drive that suddenly developed a bad sector, or the disk had cross-linked data blocks, or some malicious program wrote a single zero in the middle of a 300 megabyte archive, a verify would find that while a capture or merge probably wouldn't.

Right now I have a Verify action set to run prior to a Compact action. But if the Compact action in essence will check for the integrity of a file, why would this make sense?

It makes more sense to run a verify following an action that changes the archive. That way you can immediately verify that, following the change, the structure of the archive is still OK.

I do it here because QRecall is still a work-in-progress. Over the past year I've introduced a couple of bugs that caused a capture or merge to vandalize an archive. Regular verifies help me catch those quickly. I also had one of my RAID drives die; the verify caught it before the OS or my RAID monitoring software did!

Steven M. Alper wrote:1. Would it be possible to add a condition which would prevent an action from occurring if a previous verify failed? It would be nice to be able to automatically stop actions which could potentially further corrupt a damaged archive.

That's not necessary. All QRecall commands constantly perform data integrity checks. If a command encounters any inconsistency in the archive data it will stop immediately. A verify is not necessary to determine this.

Once an archive is found to be inconsistent, flags are set in the file headers so that QRecall knows that the archive is bad. At that point, the only two commands you can perform on that archive are Reindex and Repair.

(Actually what happens is that every archive is marked as "bad" as soon as an action begins. If the action completes successfully and finds no inconstancies, the flags are set back to "good" and the archive is closed. If anything interrupts the action -- I/O error, crash, power failure, etc. --- the archive is left with the "bad" flag and will need to be repaired before it can be trusted again.)

2. What about adding a "repair if damaged" step to the verify action?

Repairing requires a number of decisions that should be considered carefully before running it and may have consequences that need to be reviewed once it's done. Running a repair automatically could hide all kinds of serious problems.

Besides, any backup system that needs to automatically repair itself from time to time is simply broken.

3. I'd love it if you could add Growl support so that completed actions and verification reports could be accessed more easily than launching the app and checking the logs.

That's been on my long list of features to add for months.
That's a good question.

The QRecall Activity window is independent of the QRecall application. So, technically, it doesn't belong to the QRecall application and isn't affected by its window order. Hiding or quitting the QRecall application won't hide or close the QRecall Activity window either.

Having said that, I'll look into the possibility of hooking the "Bring All to Front" command and bring the QRecall Activity window to the front at the same time.

You can always bring the activity monitor to the front by using the Window > Activity Monitor > QRecall Activity (Command+0) command.
All good suggestions. I personally don't like most manuals in PDF format unless they are intended to be read like a book; the Photoshop manual comes to mind. But other people feel differently. I probably won't do something like this in the near future because that means maintaining two different versions of the documentation, which would be counterproductive at this point.

The next time I see David I'll ask him.
Steven M. Alper wrote:Are Merges and Rolling Merges exclusively for manipulation of the archive itself? There's no capturing going on?

That's correct. A merge action only combines layers that have already been captured. Captures and merges are complements of each other: A capture adds a new layer, a merge removes old layers.

So in other words, I would likely need three actions per archive?:
1. A recurring capture
2. A recurring merge or rolling merge of the resulting archive
3. A recurring compact of the archive

And if you're paranoid (like I am), you'll want to schedule a recurring verify.

Note that I'm contemplating adding an assistant to QRecall that would help you set these up the first time.

[P.S. - I hate the way (at least your installed version of) JForum ignores the BB Code on first submission (actually displays the code) until you edit and resubmit.]

So do I. I'm working to see if I can upgrade to a version that has some of these problems fixed. I find I have better luck when I uncheck the "Disable HTML in this message" option.
Steven M. Alper wrote:I keep re-reading the Rolling Merge Action page, but seem to fall short in understanding exactly how the "Followed by:" merging works and what effects my choices will have.

Rolling Merge is really a simple idea, but it has been a struggle to explain it clearly. Let me explain the logic that QRecall uses and see if that makes it any clearer. Then I'll try to answer your other questions.

Each "tier" in the rolling merge defines a set of time periods, counting backwards from midnight today. Let's say you defined a rolling merge that has no recent layers, 5 day layers, 2 week layers, and 1 month layer. When that rolling merge is run on Friday May 4th, 2007, it would find the following time periods:

- 0 recent layers: nothing
- 5 day layers: One period each for April 29th, 30th, May 1st, 2nd, and 3rd.
- 2 week layers: One period each for the week of April 22nd and the week of April 15th.
- 1 month layer: All of the days in March, plus a second period for April 1st through the 14th; this represents the fractional month between the end of the last month layer and the beginning of the first week layer.
- Remaining layers: Everything before March 1st.

All of the layers in each time period is then merged into a single layer. If this archive only had one layer per day, then the first 5 day layers would do nothing as those days already have only one layer.
All of the layers between April 22nd-28th would merged into a single layer. (week layer)
Layers between April 15th-21st would be merged together. (week layer)
Layers between April 1st and the 14th would be merged. (partial month layer)
Layers between March 1st and the 31st would be merged together. (month layer)
Everything between the first layer and February 28th would be merged. (remaining layer)

If I said Keep most recent 30 days and left it alone, what happens to the 1st day once I've reached 31?

Assuming that you are created one layer each day, the 1st and 2nd layers would be merged together, leaving you with the 30 most recent layers.

Said more precisely, any layers captured in the past 30 days would be left alone. Any layers captured more than 30 days ago would be merged into a single layer.

If I say keep most recent 30 days, followed by 30 day layers, is that not the same as simply keeping the most recent 60 days?

If you create one layer a day, they are equivalent.

As a counter example, I have QRecall capture the source to the project files every 20 minutes while I work. At the end of the day there might be 20 new layers in the archive. After working on QRecall another 29 days I might have 600 new layers. (Yikes, that's a lot of layers!)

If the rolling merge was set to keep the most recent 30 days followed by 30 day layers, it would leave all 600 layers as they are because they were all captured in the past 30 days.
If the rolling merge was set to 0 recent days and 60 day layers, the 20-odd layers created each day would be merged into a single layer, reducing those 600 layers into 30.

If I say keep most recent 30 days, followed by 3 week layers, then:
- The 1st day gets bumped into the first week layer
- Once the first week layer has 7 "bumped" days, then, what, the first of those 7 days gets bumped to the second week layer? Or the whole first week layer gets bumped to the second week and the first week layer now contains only the next bumped day?

A complete week layer is a single layer. What happens the next day is that there is a gap -- a fractional time period -- between the most recent week and the end of the 30 day. Over the next 6 days all of the layers in that gap are merged into a single layer. At the end of the week, that single layer becomes the most recent week. At the same time, the oldest week falls off the end (rolls) into the first monthly period, or the factional portion of the month, and is absorbed there. The next day the process starts all over again.

If I set up like this, what happens?
  • Most recent 30 days
    3 week layers
    2 month layers
    1 year layer

  • Whatever layers you capture in the past 30 days are left alone. Layers in the recent period are never merged.

    All of the layers between the 30 day period and the next three full weeks will be merged into 3 or 4 layers. One for each full week, and one for any fractional week between the last week and the beginning of the 30 day period.

    Moving backwards from the earliest week, QRecall finds the next two full months and merges all of those layers into 2 or 3 layers (again, two full months plus a partial month if there is one).

    It then moves backward and finds all the layers captured during the previous calendar year and merges those into one layer. A second layer might be created if the earliest month layer wasn't January.

    All of the layers created before Jan 1st of last year are then merged into a single layer.

    If I don't do anything beyond "Most recent 30 days," the bumped days get merged with the original?

    Yes. The layers in the "keep" period would be left alone and anything beyond that will fall into the single "remaining" layer.

    Sorry, it seems clear until I try to think it through. Perhaps the help page could use some specific examples to help understand what the real effect of choices in the action will be?

    It does seem simple -- until you start to worry about the details. Personally, I rather people not worry about the details and simply focus on goal: What resolution and duration of incremental backups do you need?

    If, two months from now, it is important to be able recover both the version of a document from Thursday May 3rd and Friday May 4th, then you'd probably want daily layers going back 90 or 120 days.

    On the other hand, if you only care about getting back files in the past few days as well as that vacation video you deleted last November to clear up some disk space, then keeping the past 7 days plus 12 month layers is probably more in line with your needs.

    I keep struggling to find a simple way of explaining this. I'm considering adding some more detailed examples to the help file. My reluctance is that I try to keep the help short and to the point, and explanations like this get rather tedious.
    The missing file messages shouldn't be any cause for alarm, unless you think the files are important. If you are using your computer while a capture is in progress this is inevitable. Files are created and deleted constantly, and QRecall may prepare to capture a file then find that it's been delete before it can read it. These events usually aren't interesting, but I feel obligated to record them in the log none the less.

    The QRecall application wasn't designed to run as root. (In general, the GUI isn't designed to run as root and Apple frowns on it.) You may run into some odd behavior.
    Steven M. Alper wrote:Reading through the help files, I found the following:

    Thanks Steve, that's a great help. These corrections will appear in the next release.

    No problem -- I'm just glad thinks are working now.

    Beta identity keys are permanent. The beta application itself will expire, but your identity key won't.

    The only thing that should cause you to lose your identity key is if you lost the preference file/folder where your identity key is stored. The self-uninstall procedure (Command+Option+Shift+Q) should not delete your key, but performing the manual uninstall instructions found in the advanced topic of the help will. If you performed the manual uninstall in an attempt to diagnose your earlier problem, that would definitely erase your key. Identity keys are also user specific, so running QRecall from another user account or from a different boot volume will require that you reenter a key. Note that if you are running QRecall from multiple accounts on the same machine you can enter the same identity key for all of those users, or use a different identity keys to keep the data captured by each user separate.

    And I agree that the identity key entry screen is confusing. Apple added "default text" to text entry fields recently and I used that to display a sample identity key. However, Apple's method of displaying the default text (i.e. the value of the field if you don't enter anything) is to draw it dimmed. I've had two individuals think that there was already an identity key entered, and two more now that thought the field was disabled.

    I'll probably replace this with a prompt or simply leave the field blank in the next version.

    Thanks for the feedback.
    Here's a tip:
    If you are recapturing large disk images, you'll get much better performance if you turn off (lowest setting) shifted-quanta analysis in the archive's settings. This will greatly improve recapture performance.

    Disk images are an array of immovable blocks. Data doesn't "shift" in a disk image the way it can in a file. So searching for shifted data wastes a great deal of resources.
    I'm not sure exactly what a VM file is, but if VM files are like large multi-media projects or disk images, then they are exactly the kind of incremental backup problem that QRecall was designed to deal with.

    When the file is recaptured, QRecall will read each block (quanta) of data and compare it with those that have already been captured. Only new data blocks are added to the archive.

    In the situation of a disk image, take a 600MB disk image file. If you mount that image and add a single 1MB file, only a little over 1MB worth of data has changed. When QRecall recaptures the file, it will discover 599MB of duplicate data and only adds 1MB of new data to the archive.

    There's a similar situation with large multimedia files like Photoshop or audio files. Open up a 300MB Photoshop file and add an adjustment layer. The original 300MB of data is still in the file, although it might have shifted to a slightly new location. That's where QRecall's shifted-quanta analysis gets to work. It still finds the 300MB of existing data and only adds the new data to the archive. Ditto for changing the MP3 tags on a large audio file. The audio hasn't changed, only the tiny bit of tag data.

    Now this still requires QRecall to re-analyze all of that data, which will be time consuming. So this is definitely something that you'll want to schedule to run at night or when you're not using your system.

    (And I'm glad you like the answers.)
    Forum Index » Profile for James Bucanek » Messages posted by James Bucanek
    Go to:   
    Powered by JForum 2.1.8 © JForum Team