I am trying to understand the concept behind the stacks.

Too me it looks that it offers mainly an other level of redundancy.

I have two scenarios where this might be of use, but I am not sure:

1) Instead of a file system backup of an archive to an other disk, I can now use a stack. Here the advantage seems clear: Instead of copying the whole file every time, stacks can update incrementally. Very handy if the backup location is online and bandwidth limited. Right?

2) Instead of two independent archives on two disks I can now have an archive on one disk and the stack on an other. What's the advantage of stacks here?

Would be nice, if you could shed some light on my thoughts.

And one more question:
Is it planned to support stacks via FTP?

Thanks,
Johannes

Switching to a new machine (M1 Max) I thought I start with QRecall 3
Nice to see my favorite backup tool advancing further.

I have not yet looked into stacks and other new features but found a bug: I can't edit actions once they are created. When I double click an action (or use the context menu) a window is opened but it is blank. This happens in the action window and in the archive window sidebar. Using the triangle in the sidebar does not reveal anything.
The archive and the capture action were freshly created (not an upgraded old archive). The action captures as intended.

My new machine is on macOS 12.0.1
I have not tried it on my other machine (Mojave) because I don't want to mess with the old archives.

Let me know, if you need more information.

Johannes

James,
it seems your theory is correct. After 2 weeks both archives have kept their size. Everything works again as expected with the hacked version.
Thanks for the quick support.
Johannes

I compacted the Archive 1 (run surprisingly fast). Qrecall size is back to Finder size.
I installed the special version of qrecall, opened the archive: Size in Qrecall is back to 80 something GB (strange)
Again compacting (this time it took as long as usual). Qrecall Size is back to finder size.
Now let's see what happens the next days ...

Johannes

James,

thanks for looking into this.

Here is part 1 of the answers:

Archiv 1:
Finder size: 42,94 GB
Qrecal size: 82,62 GB
Output of ls -lskn:

total 80155512

       60 -rw-r--r--  1 501  0        59224 10 Jul 18:06 displayname.index

    15920 -rw-r--r--  1 501  0     16301912 10 Jul 18:06 filename.index

        4 -rw-r--r--  1 501  0           56 10 Jul 18:06 fill.index

   196612 -rw-r--r--  1 501  0    201326648 10 Jul 18:06 hash.index

      576 -rw-r--r--  1 501  0       588449 10 Jul 18:06 hash_adjunct.index

     1068 -rw-r--r--  1 501  0      1088080 10 Jul 18:06 layer.index

    16388 -rw-r--r--  1 501  0     16777220 10 Jul 18:06 negative.index

        4 -rw-r--r--  1 501  0         1489 10 Jul 18:06 outline.index

   485932 -rw-r--r--  1 501  0     30081072 10 Jul 18:06 package.index

      756 -rw-r--r--  1 501  0       770428 10 Jul 18:06 package_adjunct.index

 76945184 -rw-r--r--  1 501  0  40124391024 10 Jul 18:06 repository.data

    38792 -rw-r--r--  1 501  0     39183976 10 Jul 18:06 repository_4k.checksum32

  1224892 -rw-r--r--  1 501  0   1253888000 10 Jul 18:06 repository_p4w8k32m2.0.anvin_reed_sol

     2208 -rw-r--r--  1 501  0      1224500 10 Jul 18:06 repository_p4w8k32m2.0_4k.checksum32

  1224892 -rw-r--r--  1 501  0   1253888000 10 Jul 18:06 repository_p4w8k32m2.1.anvin_reed_sol

     2208 -rw-r--r--  1 501  0      1224500 10 Jul 18:06 repository_p4w8k32m2.1_4k.checksum32

        4 -rw-r--r--  1 501  0          122 10 Jul 18:06 sequence.index

        4 -rw-r--r--  1 501  0         1814  5 Jul 09:33 settings.plist

        4 -rw-r--r--  1 501  0          874 10 Jul 18:06 status.plist

        4 -rw-r--r--  1 501  0         3872 30 Jun  2018 view.plist

Archiv 2:
Finder Size: 491,5 MB
Qrecall Size: not available (opening the Archive presents an error as reported), probably 100 GB or more

total 151814832

        4 -rw-r--r--  1 501  20        240  8 Jul 17:00 displayname.index

       36 -rw-r--r--  1 501  20      33144  8 Jul 17:00 filename.index

        4 -rw-r--r--  1 501  20        520  8 Jul 17:00 fill.index

    24580 -rw-r--r--  1 501  20   25165880  8 Jul 17:00 hash.index

        4 -rw-r--r--  1 501  20         82 24 Mai 09:00 hash_adjunct.index

        4 -rw-r--r--  1 501  20       3152  8 Jul 17:00 layer.index

        4 -rw-r--r--  1 501  20        770 11 Jun 23:00 outline.index

   339172 -rw-r--r--  1 501  20    1278000  8 Jul 17:00 package.index

      148 -rw-r--r--  1 501  20     147532  8 Jul 17:00 package_adjunct.index

151398188 -rw-r--r--  1 501  20  412997368  8 Jul 18:00 repository.data

      200 -rw-r--r--  1 501  20     201660  8 Jul 18:00 repository_8k.checksum32

    26224 -rw-r--r--  1 501  20   25812992  8 Jul 18:00 repository_p8w8k16m2.0.anvin_reed_sol

       16 -rw-r--r--  1 501  20      12604  8 Jul 18:00 repository_p8w8k16m2.0_8k.checksum32

    26224 -rw-r--r--  1 501  20   25812992  8 Jul 18:00 repository_p8w8k16m2.1.anvin_reed_sol

       16 -rw-r--r--  1 501  20      12604  8 Jul 18:00 repository_p8w8k16m2.1_8k.checksum32

        0 -rw-r--r--  1 501  20          0  8 Jul 18:00 sequence.index

        4 -rw-r--r--  1 501  20       2038 10 Jun 18:22 settings.plist

        4 -rw-r--r--  1 501  20       1016 10 Jul 18:06 status.plist

No snapshots on both Volumes.

I'll try Disk repair later (have to save boot from an other account)

Johannes

(Not sure whether this is related with this topic: http://forums.qrecall.com/posts/list/913.page)

Since changing two Volumes from hfs+ to apfs I have two archives that frequently fill up my hard disk for apparently no reason.
After some investigation here are my findings:

Both archives report significant lower size in Finder than within QRecall (about 40 GB vs 120 GB).

Free Space of the disk indicates that Finder is wrong (like in the other thread)

Compacting the archive brings the archive back to the small size given in Finder

Trying to automate the solution with a compact action (condition: ignore if archive size is <60GB) failed. The action is not triggered (Finder size is 40 GB, Qrecall reports 72 GB today).I would assume that the condition is looking at the Finder size.

this did not happen when the Volmes were hfs+ formatted. Source volume and archive volume are both apfs.

Both archives are triggered several times daily, have regular merge actions and mostly have small layer sizes.

I'll send a diagnostic report in a few minutes.

Thanks for looking onto the issue.

I capture the entire folder "Sound room" (but some of the folders within have their capture preferences that to "do not capture")

Thanks, James, for getting back to me.

I tried all your suggestions ? the result remains the same. QRecall keeps on capturing mp3 files.
I used the patterns in the Archive settings (I did not know that this is possible with capture preferences as well)

What worked was the .qrecall_exclude_patterns.txt file. This is a good solution for me. So the issue is solved for me.

Let me know if you want me to do further testings ...

Thanks,
Johannes

I was very happy to see the feature "exclude by expression" finally appear in the archive settings

But I can't get it to work. Perhaps I'm doing something wrong ...

I have an Archive backing up a certain Folder and I want to exclude mp3 files.
So I set up a a glob pattern like this
path: /Volumes/WorkingMedia/Sound room
Pattern: *.mp3
repetition: check

But when I capture the folder mp3 files are still included in the Archve.

What am I doing wrong?
Thanks in advance,
Johannes

Thanks for your detailed response. After moving the old symbolic link the Services now work as expected. I also found at least one "culprit" for the difference in sizes: I had not shown invisible items (and the hidden ~/Library accounts for quite some data).

I have one issue though:
It seems that the scheduler does not start at system startup. The Monitor says "waiting for scheduler" (or something like that). When I launch Qrecall manually everything starts working fine.
Any hints?

Johannes

Thank you for the work around code lines. I was thinking in that direction myself. Your examples saved me half an hour digging into man pages and pipe procedures. Seems that the command line tool will provide means to deal with my problem. But I hope there will be a more reliable solution (it's easy to forget to run the exclude script when your head is in creative mood when starting new projects).

Johannes

As promissed, here comes my report, what I have encountered so far (I'm on OS 10.10.3):

When updating QR2 did not carry my excluded folders from the actions to the archive. I was under the impression it should ... I should have checked it, my bad. My archive exploded because it stuffed my whole iTunes library in.

I excluded three folders for the archive manually. When I realised that I should have excluded the iTunes library too, it won't stick. I could add it to the list, but the next capture will start again archiving iTunes. I canceled, opened the archive, checked the settings: iTunes is not in the exclude list. Tried it again, it was added to the list. Closed the archive, re-opened it, checked settings: iTunes no longer in the list. Strange. I had to exclude the folder by setting its preferences via command line tool.

While browsing to find out why my archive has started exploding, I noticed that the file sizes in various places do not "match". Folder or Layer sizes where much bigger or smaller than the content. I did not check every item of course but it was strange to find only one GB of data in al Layer that claimed 6 GB. But maybe that is a lack of my understanding of QR's magic.

QR2 suddenly started capturing the system folder. Don't know why. The system has never been part of any QR archive. I checked the action and there was "/" in the list of items to capture. I have no idea how it was added to the list (previous captures of this action behaved well). I deleted the entry. Seems fine for now.

When I browse my archive I see two disks under my user with all the folders like it should be. But the inspector for both disks looks different. One shows the properties like kind, size, owner etc - as it should. The other says: Kind: Placeholder and no further infos beside the list of the layers.

QRecall Capture Preferences is not in the service menu. I searched in the system preferences>keyboard, no luck either. Only the two "old" entries capture to/reveal in Qrecall archive are there. Tried restarting. No luck.

Hope I described everything clearly enough. Otherwise please ask.

Johannes

I made the jump and started testing QRecall 2 (b13).

While there are many nice things to discover, I encountered quite a few issues that I will report later. For the moment my biggest concern is how to deal with databases and big but unimportant data on my disks. While it sounded like QR2 would have some solutions, it seems they are not enough in my case. But maybe I overlooked something. So no complains but the question: How would you advice to deal with the following situations in QR2:

1) Excluding the data hogs
I do audio and film productions which generate a lot (>100GB) of intermediate data, proxy files, analysis data, caches etc. that do not belong in a backup. Unfortunately they are scattered all over the disks and with every new project new folders would have to be excluded. So the manual route seems no work around.

2) Taming the databases
Many of my applications work with database. Preventing backups while any of those are running would effectively stop all backups. And again there are too many files involved to exclude all of them from the hourly backup and create a manual action to cover them before shutdown.

A solution for both issues would be simple filters on file/folder names (exclude/include *.dtBase2, peaks/*, analysis/* export/* ect). Any other suggestions?

I hope there will be a solution in RQ2, cause otherwise QR is (and has been for years) exactly what I need (and the alternatives are not convincing either).
Thanks,
Johannes

Hi James,

Glad to finally see 2.0 beta
Looks promising.

I'd like to start using the beta for some of my archives and keep other archives on 1.2 for a while in case something goes haywire. All actions (weekly captures in essence) for those legacy archives would be started manually form the old version, while all scheduled actions would run with updated 2.0 archives. Is this possible? From what I've understood so far, the preferences might be a problem.

What would be your advice to make a save transition? I hesitate throwing everything on a beta at once.

Johannes

When I read Norbert's post, it felt quite familiar. I too have some trouble to get "many sources to many archives at the right time" working. I really tried to get there with the many clever schedule options of QRecall (and James' help) ? it did not work out.

Life would be so much easier if QRecall only had one simple scripting command: "run Action #NameOfAction# now". I could easily write a batch that would take care of everything else (mounting/unmounting disks, stopping Time Machine, closing databases, not interfering with my work or the SuperDuper cloning of the same disk at the same time, not running a time consuming verify action for the local archive when I need to capture to the offsite archive first etc.).

Otherwise QRecall is really the best backup tool I ever used (and I used a lot).

Johannes