Christian Roth wrote:QRecall's new visuals seem not to be compatible with Screen Sharing:

I've noticed the same thing (Mac OS 10.7 client sharing a headless Xserve running OS X 10.6). I find that if I do anything to change the browser display (click on row to select it, scroll, resize the window, etc.) the window redraws correctly.

I assume this is a consequence of QR using the fastest graphics routines available, hence bypassing some hooks in the OS that it uses to detect changes for sending over Screen Sharing?

Alas, I'm not that clever. Most of the revamped user interface is built using Apple's Core Animation, an API introduced in OS X 10.5.

Is there anything I could do about it (some hidden pref in QR) to disable fast drawing, but enable compatibility with Screen Sharing?

I'm pretty sure this is a bug in Screen Sharing. I'll file a bug report with Apple.

Adrian,

It has occurred to me that there might be a general solution to this problem that could be added to QRecall. Let me know if the hack works. If it does, I might be able to create an equivalent workaround in QRecall.

Adrian Chapman wrote:I am not happy with QRecall running where I can't see it because if I put my Mac to sleep while it is capturing I end up with a corrupted backup.

Here's a hack to try: Create a QRecall folder on your boot volume (anywhere that makes sense). It should be owned by the user that runs QRecall and have read+write+search privileges only for the owner (i.e. disable all group and other access). Copy all of the files in ~/Library/Application Support/QRecall into the new folder. Make sure they all have the correct ownership and permissions. Delete ~/Library/Application Support/QRecall and replace it with a symbolic link to the new folder.

The QRecall code that loads and identifies the support folder in ~/Library uses (for the most part) standard UNIX paths, so the symbolic link should resolve to the executable folder on the startup volume, which should allow the QRecallHelper executable to run with elevated privileges.

Adrian Chapman wrote:I have just reconfigured my Mac so that the system and user directories are on separate disks.

Sadly, this configuration is incompatible with the current version of QRecall.

Security policy changes in OS X now prevent the system from launching executables with elevated privileges from any drive other than the boot volume. QRecall installs its privileged helper process in ~/Library/Application Support/QRecall. Since your home folder is now on another volume, the OS won't launch any executables it contains with elevated privileges.

QRecall is successfully installing its helper process as a privileged executable, but when it goes to launch it the OS runs it as regular process. Sensing this, QRecall assumes that the helper wasn't installed as a privileged executable and asks to install it again. Wash, rinse, repeat.

You could uninstall QRecall, create a new administrator account who's home folder is on the startup volume, and then install and manage QRecall from that account. (If you copy ~/Library/Preferences/com.qrecall.* and ~/Library/Preferences/QRecall to the new account before installing QRecall, it will transfer all of your preferences and actions.)

Steve Mayer wrote:I was attempting to find a file (filename contained spaces if that is of any import) within my archive and QRecall crashed with the following stack:

Steve,

Thanks for the bug report. Could you also send a diagnostic report (Help > Send Report)? The tools I have to decoding stack traces won't work with text pasted into the forum.

2. How is the search supposed to reveal the searched item? I see that there are a bunch of folders that are "greyed out", but nothing that truly points to where the file would be located.

See the release notes for 1.2.0(37).

In short, a dark gray silhouette of a folder means it's in the process of being searched.

Matching items appear normally; non-matching items are dimmed.

In search mode, folders can only be opened if they contain items that match. So a dimmed folder that can be opened means the folder doesn't match but it contains one or more items that do. A dimmed folder that can't be opened means it doesn't match and doesn't contain any items that match.

To quickly see just the matching items, select the "Only Show Matching Items" option in the search menu.

3. There seems to be no method of showing that the search is still occurring (i.e. Progress)

A folder or disk that appears as a dark grey shadow is in the process of being searched. As soon as that folder is searched, the folder and its contents will appear either as normal items (matches) or be dimmed (doesn't match).

pirem71 wrote:1. I'm still not able to tell how the instances captures differs from each other and that creates a sense of lack of control on what's going on. I must probably blame it to my scarce knowledge of Mac environment but I'm still uncomfortable with that (I hope you can understand me).

This problem isn't unique to QRecall, and I'm afraid I don't have a simple answer. What constitutes a significant change in a file is subjective. QRecall's #1 job is to capture all changes to a file over time, and almost by definition that means it's going to capture changes that you're not particularly interested in.

Restoring a specific version of file could became difficult (since usually one doesn't know the modification date of the file and maybe not even which is exactly the version he's looking for).
...

In this particular case, you shouldn't have to look far. QRecall is recapturing the item because metadata (permissions, extended attributes, ...) is changing. But that won't change the modification date of the file, which should only change if the data of the file changes. So you can simply view the most recently captured version of the file and look at its modification date. This will tell you the last time the actual contents of the file changed. All intermediate captures are probably uninteresting.

Is there any possibilities to perform an automatic recall of all versions of a file over a time range defined shading/un-shading layers (e.g. adding incremental suffix to the name)?

Not at this time. (Again, you could write probably write a simple shell script to do this once a command-line version of QRecall was available.) I have a more general solution to this kind of problem on the drawing board, which I hope to attack after 1.2.0 is released.

Adrian Chapman wrote:what, if any, files or folders are excluded by default?

Good question.

As of 1.2.0(50), the list of items the get automatically excluded from a capture are:

The action's archive [1]

/tmp directory

/var/temp directory

VM swap directory [2]

/.vol directory

/.hotfiles.btree directory

/.fseventsd directory

/.dbfseventsd directory

/.MobileBackups directory [3]

/.DocumentRevisions-* directory [4]

Mountpoints (i.e. other volumes) [5]

Named pipes (FIFOs)

Character special files (character generators)

Block special files (device files)

Sockets (communication ports)

Whiteout files

For every user's home folder:

~/.TemporaryItems directory

~/.vol directory

~/.fseventsd directory

~/.dbfseventsd directory

In addition, the capture action may specify other items to be filtered out and has canned filters for the various Trash and Cache folders.

[1] You can't capture an archive to itself.

[2] The location of the VM swap is determined by examining the dynamic pager daemon and is normally at /var/vm. If you've customized your VM swap location, you may need to set the QRVMSwapStorePath, see advanced setting.

[3] Only when running Lion (OS X 10.7) or later

[4] Only when running an OS earlier than Lion. Only Lion can understand the files in this directory; to earlier operating systems, these files are unreadable.

[5] Most of the items in the /Volumes directory are mount points. So the /Volumes directory itself gets captured, but rarely anything inside it. Similarly for the /dev folder, which contains mostly block and character special files.

pirem71 wrote:1. In one case only the capture action didn't create a layer (the log says, among other things, "Nothing captured")

In that case, nothing changed. If a capture can't find anything to capture, it doesn't create a layer.

2. 90% of the times a new layer was created capturing 1 item (log says "Action 2012-01-13 15:00:40 Minutia Capture Decision; capture file .DS_Store because content modification date different; was 2012-01-13 14:52:50 +0100, now 2012-01-13 14:59:55 +0100").

QRecall found that a .DS_Store file in the folder changed and capture it.

If I check the new layer in the main Archive window (I don't know if this is the correct name) I cannot see the modified .DS_Store file (since they are not displayed) but the folder containing the .DS_Store is marked as modified in its Timeline.

.DS_Store files are normally invisible. Choose View > Show Invisible Items in the QRecall browser to see invisible items.

3. 10% of the times a new layer was created capturing 2 item: the .DS_Store and the viewed .pdf file itself.
The log says (for .pdf only) "Action 2012-01-13 15:02:12 Minutia Capture Decision; capture file Decisione_CE_3_maggio_2000,_n._532.pdf because attribute modification date different; was 2011-11-15 21:25:01 +0100, now 2012-01-13 15:01:59 +0100.
Anyway having a look un the Finder the modification date for that file is still 2011-11-15 21:25

The data modification date and the attribute modification date of a file are different things. You normally don't see the attribute modification date, but it's one of the metadata values that QRecall checks. The attribute modification date is changed whenever some process sets the attributes, permissions, extended attributes, or other metadata for a file.

1. A folder is marked as changed if its .DS_Store file is changed even if it's not shown (since .DS_Store files are actually captured).

That's correct. In QRecall. a folder is considered to have changed if its metadata or immediate contents have changed.

Is it possible to avoid this behavior

I can't imagine why one would want to.

or even exclude .DS_Store files from capture?

Not at this time, but arbitrary file filers are on the to-do list.

Are there any advantages in capturing .DS_Store files that I'm not aware of?

.DS_Store files keep all of the display information used by the Finder. It includes the screen position and size of the folder's window, its display mode (icon, column, ...), sorting preference, and a host of other details. The Finder is constantly updating these files as you interact with Finder windows.

2. It would be nice to be able to select a Layer in the main Archive window and choose a command "Show capture log" (that shows the log in a separate window or, as an alternative, open the standard Log window and highlights the searched row) instead of manually search the Log window comparing the time column

Layers and log records do not have a one-to-one correlation.

Note: I made a copy/paste of Log file rows; is there a better way to include portions of the Log in a post?

The forum has a "code" tag that might make them easer to read, but other than that copying and pasting seems like the best way.

pirem71 wrote:I noticed that opening/closing a .pdf file (both in Preview and Acrobat) without making any modifications will anyway result in a capture of the file as if it was modified.

Something is changing.

QRecall looks a variety of information about each item to determine if it should recapture it. This includes the file's dates (creation date, data modification date, attribute modification date), permissions, ownership, extended attributes, finder information, resource forks, access control lists, and so on. If any of that has changed, QRecall will recapture the item. In this case, since the actual data of the items hasn't changed, the only thing QRecall will add to the archive is the modified metadata.

If you want to know exactly what's changing, you can temporarily set the QRLogCaptureDecisions option (see advanced setting). Perform a capture, change this setting to true, open and close a PDF, perform another capture, and then delete the setting (or set it to false). The log will record the first reason that QRecall decided to recapture each item.

pirem71 wrote:When you say you plan the development of a command-line utility do you mean also Apple scriptability? I gave a look and I found no dictionary for QRecall in Script Editor; I must assume it is not scriptable for the time being. It would be nice to have it included into an Applescript.

The two are very different. I have "add AppleScript support" on my list of future features, but it's been a low priority since I consider its utility to be somewhat limited. A command-line version of QRecall, on the other hand, is pretty high on the list. And since you can invoke command-line programs from AppleScript, it would provide limited AppleScript support as well.

As for point #4: I like QRecall ability to keep all data under control and I was imagining a way to use it to protect my files (the ones on the disk, not the archive) against corruption. I had another idea: what if the user can add a flag to files that shouldn't change (let's say an OpenMeta tag like "Frozen" to be communicated to QRecall)?
When QRecall find a file tagged "Frozen" with a changed content, it can not only capture any changes in the new layer but also alert the user that can take corrective measures. If I deliberately want to change a file I have to remove the "Frozen" tag first.
This would really give peace of mind about integrity of data.
You would have for sure a few false positives (forgotten "Frozen" tags when changing a file) but it is better than find an unusable file after 5 or 10 years.

This really isn't QRecall's job. If you want to prohibit a file from being modified, there are a number of file system features at your disposal. You can clear the write permission bit, lock the file in the Get Info window, or (if you want to go completely crazy) set the low-level "immutable" attribute for the file. All of these will prevent the file from being accidentally changed. QRecall doesn't need to be involved at all.

What about the possibility to make "merge" actions at file level instead of merge two entire layers? Is it or will it be possible?

It's not feasible. A merge occurs when the data in two layers are combined. This results in the earlier layer being deleted.

But the earlier layer can't be deleted unless all of the files are merged. So you can't merge one file without deleting that file from the earlier layer, which means that if you rewound the archive to that earlier layer and recalled all of the files, the file you merged would be missing--which I don't think is a good idea.

pirem71 wrote:Hi, I discovered QRecall reading a book by Joe Kissel and I can say it is exactly what I was looking for: a time machine on steroids. Simple and effective, a pure mac style app.

I'm glad you found us.

1. I want to use it to back-up my DevonThink indexed folders. Each top level folder (corresponding to a DT database) includes some 50.000-100.000 files in 10.000-15.000 sub-folders. I back them up creating a separate QRecall archive for each of that folders. Are these dimensions likely to cause any problems to QRecall? Is there a limit (in file number o MB/GB) to be respected for a single QRecall archive?

A QRecall archive (as of version 1.2.0) can be up to 6TB in length and can easily store upwards of several billion file and data records (it's hard to translate QRecall's data records in a number of "files", because it's not a one-to-one correlation). Because of de-duplication and compression, that 6TB of archive data can represent tens of terabytes of actual file data.

2. Partial update of a file (sub-file updates), deduplication, compression, etc are all good features of a "modern" back-up system (as stressed out by Joe Kissel). Anyway I'm a bit concerned about reliability issues that these "structures" may introduce when compared to the old "make a Finder copy of the entire changed file" strategy. E.g. if a few blocks of the HD get corrupted (say the ones storing archive indexes, not quanta chunks), is there any chance that the corruption can destroy the integrity of the whole archive and not only the integrity of a few files? Have you got any good news that can reassure us (paranoid people)?

QRecall's archive structure was specifically designed to be robust and damage resistent. A QRecall archive, quite unlike a file system volume or database, is a collection of small, self-contained, data records. Each record is checksummed to ensure data integrity. The interdependence between records is deliberately kept to a minimum. As an example, in a typical file system a directory (folder) is a record that lists the files and sub-directory it contains. If that directory record is destroyed, all of the files and subfolders it contained become "lost". (Those files might be recoverable, but there's no way to know where they belong in the directory structure.) In a QRecall archive, the complete location of each folder is stored redundantly in every directory record. If a directory record is destroyed, the sub-folders of that directory can reconstruct it's name, location, and most of its content. Similarly, QRecall archives rely on a number of complex index files; however, none of the index files contain any unique information. All index files can be reconstructed from the primary set of archive records (this is one of the things the Repair command does).

3. Working with very large archives, visual instruments lost partially their utility and one must rely on creation of text logs to be then managed/filtered/manipulated with app like TextWrangler.
E.g. Highlights (all items captured in a layer) is a nice feature but would it be possible to export the highlight in a text file instead of simply graphically reaveal it on screen? There could be a command to create highlights file for a given layer, a set of layers or all the layers of the archive (one separate txt file for each layer).

I have no plans for exporting (or importing) visual information in the from of text files from the graphical user interface. However, I do have future plans for a QRecall command-line utility that might satisfy your needs.

4. Great part of my files are closed projects archive files, i.e. they are not supposed to change over time (or change only 1 or 2 times in several years). Is there any suggestions on your side on how to keep these files integrity under control (detect as soon as possible a corruption so to restore on the disk the correct file version using QRecall)?
I was thinking again to text files: e.g. create a file listing all the files that never changed since the first capture (checked using MD5 checksum; I know they're OK), then create a file with all items changed 1 time since the first capture and verify if the change was made by the user or if it is a corruption, than create a file with items changed 2 times, etc. Using this procedure for items changed up to 3-4 times should cover all long term storage critical files.
Primitive strategy, I know; can you think about something better?
Of course, after a positive check, there could be the possibility to merge two "snapshots" of a file to reduce the effort during next time check procedure (e.g. files changed 1 time could be merged and then go to "never changed" category that needs no further control). Is it possible to make this merge on "per file" basis instead of doing for the whole layers?

QRecall doesn't have specific features to enforce any kind of change policy in your files, however I can make a couple of suggestions.

To detect if documents have been inadvertently changed, you can simply review the QRecall archive from time to time and see if any of those old project files have changed (by shading older layers, QRecall will show only changed items in the browser). If they shouldn't have changed, recall the earlier, presumably correct, version. And again, a command-line version of QRecall might let you design something like this that would run automatically.

QRecall can't detect data corruption in your files. Data corruption will happen silently with nothing to clue QRecall to the fact that something has changed. If you're worried about corruption in a set of files that should not be changing you can periodically perform a restore of the files. When QRecall recalls a file from the archive, it first sees if the file already exists. If it does, it performs a comparison of the existing file and the data in the archive. It only rewrites the files if the comparison fails. So, in a way, restoring a set of existing files that shouldn't have changed is really an integrity check. All of the data in the QRecall archive is tested using data checksums and inter-record consistency checks, so QRecall knows that the data in the archive is valid.

As for the data in the QRecall archive getting corrupted, it's highly recommended that you schedule a verify action to run periodically. A verify will test and validate every byte of information in the archive; even a single bit of data out of place will be immediately detected.

I hope that helps.

Peter N Lewis wrote:It becomes a hack when you have three or four rotating backups, and need to create three or four copies of each action on each Mac.

I feel we're getting the "solution" cart in front of the "need" horse. While seme kind of "multi-archive" action might make this kind of backup strategy easier to set up and maintain, it adds a lot of complexity for a very small benefit (in my not-so-humble opinion).

The real need is to have redundant, rotating, cascading, or fall-over backups. For that, I have a couple of new ideas on the drawing board that I think will address those needs much more directly.

Peter N Lewis wrote:Are there plans to make this automatic?

It's an interesting idea, but I don't have any plans to implement actions that target multiple archives, primarily because I don't see any way of implementing such a feature without creating a lot of ambiguities. (Ambiguities and backups aren't a good mix.)

It seems to me, if your Archive field allowed multiple selections like the Items to Capture does, you'd be able to do this within QRecall rather than the hackish solution of duplicating all actions?

You're capturing to two archives, so you should have two sets of actions—one set for each archive. Duplicating the first set of actions isn't so much of a "hack" as it is simply a way of saving time when setting up the second set of actions (as opposed to creating the second set from scratch too).

Adam Horne wrote:I can not seem to locate the Library folder within my Qrecall archive. I'm using Lion 10.7.2.

In OS X 10.7 (Lion), the Library folders are now hidden. When captured by QRecall, they are recorded as "hidden" in the archive.

Choose View > Show Invisible Items to reveal the Library folder in the archive browser.

Christian Roth wrote:What could be the issue here?

Something has mangled the preferences saved by the scheduler that it uses to restore its state when restarting. This is the second report I've had of this, but I still don't know what's going on. The saving and reading of preference values is handled by the operating system, and is usually quite reliable.

First, send me a diagnostic report (Help > Send report...). This will let me look at your preference setting files in hopes of understanding what's happening to them.

Next, open up a Terminal window, paste in the following command, and press Return.

defaults delete com.qrecall.scheduler QRSavedUserLogInState

That should delete the offending value, the scheduler will revert to using a default value, and everything should be running again. If it doesn't restart on its own, quit and relaunch QRecall. If there's still no joy, restart your system. If you encounter a new problem, send another diagnostic report.

Is this intended during beta phase

This assertion is only in the beta version of QRecall (the release version would have just shrugged it off and gone on). The beta versions of QRecall are far less tolerant of these kinds of inconsistencies.