James,
thanks for the clear explanation; I realized that maybe I misunderstood the concept of "deleted files".

If I delete a file in the filesystem between capture N and capture N+1, the file will be shown (from capture N+1 onward) with red stripes and only if I choose to display deleted items.
That is a "deleted file" in my understanding and it is not created by any merging actions but simply derived from a change in filesystem.

According to my understanding, such a file (after a grace period set in archive preference, in my case 200 months) will be completely erased in any case by a compacting action/command even if I don't merge any of the archive layers.
Based on your reply it seems that, if I don't merge layers N and N+1, the files that I deleted in finder between the captures remains forever in the archive (as they belong to layer N that is still there).

The confusion rises from the term deleted used in two different meaning: what will be erased from a compacting action are the "unallocated" items (the layer which they belong to doesn't exist any longer) while "deleted" items from file system are preserved.

May I suggest to use two different terms for the two file states? The Help is not so clear as well about "deleted items".

Hi James,
thanks for the prompt reply (no emergency here so take your time and enjoy the week-end; next time I'll avoid posting on Sunday ).

Back to the issue:
I'm sure my archives are set with the (admittedly) conservative "200 months" preference so I'll send you a report.

As for the recovered space I think I never used Merge command/action on that archives (in which I want to preserve everything has been captured) but not 110% sure.
On the other hand I used for sure the Archive>Delete Items... command; maybe the files are deleted but the space is reclaimed only during subsequent compacting.
Could that be the reason for the "Erased deleted items from layer ..." messages?

Thanks again for the outstanding support,
Pierpaolo

Hi James,
on a few of my archives I set the preference "Keep deleted items for at least" to a very high value (200 months) so to force Qrecall to keep all deleted files and don't erase them during a compact activity.

I launched a Compact command on a couple of that archives and found on Log that:
1. there are a lot of entries "Erased deleted items from layer xx"
2. Qrecall was able to free and recover some space (a few hundreds MB out of 20-30 GB)

How is it possible to free and recover space if no files are to be deleted?
Is there something I'm missing about compacting?

By the way, besides Compacting archives, do you suggest any other maintenance activity on a disk storing Qrecall archives? Will defrag be useless or even dangerous?

Thanks,
Pierpaolo

Hi James,
I have a QR archive that I don't use any more to capture my files (folders structure was re-organized) but I want to keep it as it contains old versions of that files; I'm therefore verifying it using a scheduled action.
The problem is the status window (and menu item) is always colored in red complaining that I haven't captured the archive since weeks.

Is it possible to have a preference to close/lock the archive to correct the situation (and to prevent any capture done by mistake)?

Thanks,
Pierpaolo

Hi James,
I have a few archives backing-up my DevonThink databases (big package files) and I would prefer to capture them (with scheduled actions) only when closed to preserve DB integrity.

My only chioice now is to Hold the action while logged in but it's not handy.
A better solution would be to provide scheduled actions with a preference "Request User Confirmation"; confirmation could be given with a button in the Activity Monitor small box popping up when action starts or, as alternative, with a dedicated bigger pop-up window.

Another (multi-purpose) solution would be to give actions the capability to run a script before execution; user can then display a message box or perform other checks/actions before the package is captured.

Another separate feature request: menu bar drop-down list is now displaying next scheduled action; it would be nice if it could show the scheduled time of that action (as done in Actions window).

Thanks,
Pierpaolo

QRecall is already managing the restoration of a whole archive in the best and easy way (with layer shading) but when it comes to recalling a single file I would like a slightly different approach.

It would be nice if I can select a file in the file browser pane and then launch the command "Recall all versions". The command will create a folder containing all captured versions of that file (identified appending capturing date).

This approach will spare the pain to go through timeline and perform numerous recall actions since, when I'm recalling an old version of a file, I usually don't know in advance which is the version I'm looking for.

It can work for multiple files as well (creating a parent folder containing one sub-folder for each file).

Thanks,
Pierpaolo

Consider the following scenario:
I deleted a file between "capture #6" (that originates Layer #6) and "capture #7".
In the file browser pane I can easily verify that operation making use of timelines (with deleted files visible): timeline of the file in object stops in correspondence of Layer 6.
I also would like to be able to find all the files deleted between two captures (in the example, capture 6 and 7).

When I restrict the view to a single layer, I would like to see ALL changes occurred between that Layer capture and the capture before (in the example, shading all layers except Layer 7 I could see all changes between capture 6 and capture 7).
Qr's already working this way for modified files but not for deleted files.

I know that, "technically" speaking, that's a non sense: if the file is deleted before capture 7, Layer 7 (and above) will know nothing about that file.
QR, analyzing a single Layer, should also taking into consideration the previous layer ONLY to show the deleted file.

Maybe there's a better solution (in this case I'm sure James will be able to find it ).

P.S. I was one of the guys suffering from "extended attributes issue" (I'm using Skim) and release 1.2.0(64) completely solved the problems. Great work!

Thanks,
Pierpaolo

Hi James,
the addition of the new "Status Window" and the colored icon in the menu bar is very useful to keep under control the health status of my archives.

It would be even easier to accomplish this task if:

1. The status windows would includes buttons (one for each archive) to launch an archive verification without opening QR, opening the archive and use the menu item

2. (independent form point 1) It would be nice to have an option that allows QR to perform automatically the verification based on the same logic now used for the "traffic light" (i.e. if an archive was not verified since long time, instead of switch on the red light QR directly performs the verification).

Thanks,
Pierpaolo

Hi James,
good to hear from you that exclusions based on filters are under development.

Back to capture process, with my previous post I didn't want to imply in any way that QRecall is doing something strange or even wrong.
If QR captured a revision of a file, something MUST be different on that file (or attributes, metedata, etc).

My only concerns are from the perspective of low level user:

1. I'm still not able to tell how the instances captures differs from each other and that creates a sense of lack of control on what's going on. I must probably blame it to my scarce knowledge of Mac environment but I'm still uncomfortable with that (I hope you can understand me).

2. Let's consider the moment you have to make a restore (the moment of truth for a back up system). If files accessed for viewing are continuously captured, after a few years I will have 100-200 (or more) instances of a certain file spread along its timeline.
Restoring a specific version of file could became difficult (since usually one doesn't know the modification date of the file and maybe not even which is exactly the version he's looking for).
My approach in that situation would be to restore in a folder all the instances included in a time range and throw them to a program that can highlight differences.

Is it the right strategy or am I misunderstanding the restore process?

Is there any possibilities to perform an automatic recall of all versions of a file over a time range defined shading/un-shading layers (e.g. adding incremental suffix to the name)?

Regards,
Pierpaolo

Hi James,
I set the QRLogCaptureDecisions option and made a few tests (after opening/closing one .pdf file at a time in Preview or Acrobat, sometimes only scrolling the document sometimes doing absolutely nothing).
I ended up with different behaviors but I'm not able to find a correlation between what I did and the result of capture action:

1. In one case only the capture action didn't create a layer (the log says, among other things, "Nothing captured")

2. 90% of the times a new layer was created capturing 1 item (log says "Action 2012-01-13 15:00:40 Minutia Capture Decision; capture file .DS_Store because content modification date different; was 2012-01-13 14:52:50 +0100, now 2012-01-13 14:59:55 +0100").
If I check the new layer in the main Archive window (I don't know if this is the correct name) I cannot see the modified .DS_Store file (since they are not displayed) but the folder containing the .DS_Store is marked as modified in its Timeline.

3. 10% of the times a new layer was created capturing 2 item: the .DS_Store and the viewed .pdf file itself.
The log says (for .pdf only) "Action 2012-01-13 15:02:12 Minutia Capture Decision; capture file Decisione_CE_3_maggio_2000,_n._532.pdf because attribute modification date different; was 2011-11-15 21:25:01 +0100, now 2012-01-13 15:01:59 +0100.
Anyway having a look un the Finder the modification date for that file is still 2011-11-15 21:25

A few thoughts about the results:

1. A folder is marked as changed if its .DS_Store file is changed even if it's not shown (since .DS_Store files are actually captured). Is it possible to avoid this behavior or even exclude .DS_Store files from capture? Are there any advantages in capturing .DS_Store files that I'm not aware of?

2. It would be nice to be able to select a Layer in the main Archive window and choose a command "Show capture log" (that shows the log in a separate window or, as an alternative, open the standard Log window and highlights the searched row) instead of manually search the Log window comparing the time column

3. My guess about modification time issue: maybe Preview makes a "temporary" modification of the time when you open the file and discard it when you close it in case you made no changes. The temporary date could be stored somewhere (.DS_Store?) and it misleads QRecall.


Note: I made a copy/paste of Log file rows; is there a better way to include portions of the Log in a post?

Regards,
Pierpaolo

Hi,
I'm using beta version 1.2.0.55 of QRecall.

I noticed that opening/closing a .pdf file (both in Preview and Acrobat) without making any modifications will anyway result in a capture of the file as if it was modified. QRecall can recognize that the file has not changed (the log says data are 100% duplicate) but the file is anyway captured again.

Is there a way to avoid this behavior?

Thanks,
Pierpaolo

Hi James,
thanks for your quick and exhaustive reply!

I'm glad to hear that data integrity is one of your major concearn when designing QRercall. I'll make use regularly of the verify and repair functions offered by QRecall.

When you say you plan the development of a command-line utility do you mean also Apple scriptability? I gave a look and I found no dictionary for QRecall in Script Editor; I must assume it is not scriptable for the time being. It would be nice to have it included into an Applescript.

As for point #4: I like QRecall ability to keep all data under control and I was imagining a way to use it to protect my files (the ones on the disk, not the archive) against corruption. I had another idea: what if the user can add a flag to files that shouldn't change (let's say an OpenMeta tag like "Frozen" to be communicated to QRecall)?
When QRecall find a file tagged "Frozen" with a changed content, it can not only capture any changes in the new layer but also alert the user that can take corrective measures. If I deliberately want to change a file I have to remove the "Frozen" tag first.
This would really give peace of mind about integrity of data.
You would have for sure a few false positives (forgotten "Frozen" tags when changing a file) but it is better than find an unusable file after 5 or 10 years.

What about the possibility to make "merge" actions at file level instead of merge two entire layers? Is it or will it be possible?

Thanks again,
Pierpaolo

Hi, I discovered QRecall reading a book by Joe Kissel and I can say it is exactly what I was looking for: a time machine on steroids. Simple and effective, a pure mac style app.

I only have a few questions about your app:

1. I want to use it to back-up my DevonThink indexed folders. Each top level folder (corresponding to a DT database) includes some 50.000-100.000 files in 10.000-15.000 sub-folders. I back them up creating a separate QRecall archive for each of that folders. Are these dimensions likely to cause any problems to QRecall? Is there a limit (in file number o MB/GB) to be respected for a single QRecall archive?

2. Partial update of a file (sub-file updates), deduplication, compression, etc are all good features of a "modern" back-up system (as stressed out by Joe Kissel). Anyway I'm a bit concerned about reliability issues that these "structures" may introduce when compared to the old "make a Finder copy of the entire changed file" strategy. E.g. if a few blocks of the HD get corrupted (say the ones storing archive indexes, not quanta chunks), is there any chance that the corruption can destroy the integrity of the whole archive and not only the integrity of a few files? Have you got any good news that can reassure us (paranoid people)?

3. Working with very large archives, visual instruments lost partially their utility and one must rely on creation of text logs to be then managed/filtered/manipulated with app like TextWrangler.
E.g. Highlights (all items captured in a layer) is a nice feature but would it be possible to export the highlight in a text file instead of simply graphically reaveal it on screen? There could be a command to create highlights file for a given layer, a set of layers or all the layers of the archive (one separate txt file for each layer).

4. Great part of my files are closed projects archive files, i.e. they are not supposed to change over time (or change only 1 or 2 times in several years). Is there any suggestions on your side on how to keep these files integrity under control (detect as soon as possible a corruption so to restore on the disk the correct file version using QRecall)?
I was thinking again to text files: e.g. create a file listing all the files that never changed since the first capture (checked using MD5 checksum; I know they're OK), then create a file with all items changed 1 time since the first capture and verify if the change was made by the user or if it is a corruption, than create a file with items changed 2 times, etc. Using this procedure for items changed up to 3-4 times should cover all long term storage critical files.
Primitive strategy, I know; can you think about something better?
Of course, after a positive check, there could be the possibility to merge two "snapshots" of a file to reduce the effort during next time check procedure (e.g. files changed 1 time could be merged and then go to "never changed" category that needs no further control). Is it possible to make this merge on "per file" basis instead of doing for the whole layers?

Best regards,
Pierpaolo