James Bucanek wrote:I'm assuming you didn't see my email about your earlier problems.

No, sorry, I missed it. I was sick for about a week (worst case of food poisoning EVER!) and I'm still not caught up on everything. But I found your re-send, and I'll go get b79 now and let you know what happens. I still have copies of my version 2 archives I can start with.

Well, this update is a bit delayed, for reasons that have nothing to do with QRecall -- my apologies.

I did get QRecall 3 beta 78 running on my Mac at work. QR 3 wouldn't read my QR 2 archives there either, until I did a repair. After that, they open just fine. The beta mostly works, but I'm still getting a lot of odd errors in the logs. Today, I tried to run a Verify on my main archive, and it found errors and indicated I needed to repair. I've tried twice today to repair, both unsuccessful. The first time was a "lost connection with helper". I couldn't figure out if it actually finished or not, so after a couple of hours, I rebooted. After the reboot, QR 3 indicated there were still problems with the archive, so I tried to repair again. After some period of time, I noticed the numbers in the status dialog were no longer changing, although I did not get a "lost connection" error. Eventually, the entire Mac locked up -- even command-option-escape wasn't allowing me to kill any tasks, so I did a hard reboot by holding down the power button. After a reboot, I did a Disk Utility scan of the boot drive, and it found no problems. I don't know that QRecall was the cause of the lockup, but I rarely have a Mac lock up like that.

I'll send a report from the work Mac.

James Bucanek wrote:Bruce, that is quite the story of woe! Sorry to hear you're having such a rough time.

That's OK. I pretty much expect that when I'm beta testing. All part of making the software perfect.

First, please send a diagnostic report. I'm particularly interested in your crash logs, because you're getting a lot of crashes for some reason (both versions!) and I think we should start there.

I just sent it. This first report is from my home Mac. I briefly tried it on my office Mac last week, and had some of the same problems. I'll be back in the office again tomorrow, so I'll try it again and send a report from there as well.

QRecall 3.0.0(76) is my first experience with any of the 3.0 betas. I started by compressing (.zip file) QRecall 2.2.12(1) so that I could recover it if need be. Also made copies of my two existing archive files, and will use the copies with the QRecall 3 betas. Initially, QRecall 3 crashed every time I launched it. After several attempts, all of which failed, I re-installed QRecall 2, and then held down shift-option to do a Quit and Uninstall, which appeared to be successful. Then I launched QRecall again, and this time it launched without crashing -- until I clicked on something in the Quick Start window, and then it crashed. I'm not sure what I clicked, because it happened so fast, but I was aiming for the Open Archive button.

So, I launched QR 3 again and this time tried File > Open. No crash, but it refused to open either of my archives from the QR 2.2.12(1). Log file said "can only decode version 3 and its immediate predicessor". Which I thought was what I had. By the way, note misspelling of "predecessor". Although File > Open didn't work, File > Repair did work. I was able to repair both archives, even though they had not needed repair under QR 2. After repairing both archives, I tried opening them again. This time, one of them, which only had 2 or 3 layers, did open after upgrading itself. I was able to run a backup using QR 3 on that archive. For the second archive, which had 75 layers (been using it for several years), I could open it, but as soon as I tried to do anything with it, it would start to upgrade itself, and after some period of time QRecall 3 would lose its connection with the helper, and the upgrade would never complete. I had noted that the archive had six instances of "Macintosh HD" at the top level, so I thought maybe that was the problem. I tried multiple times to select them and combine them in QR 3, but that always failed as well (lost connection with helper). Eventually, I reinstalled QR 2.2.12.1 long enough to open that archive and combine all the hard drive instances. Then I tried to open the newly-combined archive in QR 3. Once again, it refused to open at all, but it did allow to me Repair it. After that, I could open it in QR 3, but again, as soon as I started to try to do anything with it, it would start the upgrade of the archive format, then a while later 'Lost connection with helper" and "could not send message to helper process".

So, the archive that won't upgrade is my main archive, and I haven't really been able to do anything useful with it in QR 3 yet. When I get more time, I may try a new archive and see how that goes.

Several of my backup drives are getting old, and I'm going to need to replace them soon. James, it has occurred to me that you must have a lot of experience with backup drives. Do you have any recommendations for what drives work best with QRecall? Are there any particular manufacturers or models you recommend, and are there ones to avoid? How about spinning hard drives versus solid state drives? What about shingled magnetic recording (SMR) drives? Do you prefer ready-to-use external drives (includes drive and case as a unit), or do you prefer to buy a bare drive and put it in an external case yourself? What about NAS drives, RAID, or Drobo units? Personally, I'm looking for something in the 2 terabyte range and I would prioritize reliability and speed over cost.

-- Bruce

I'm still not sure why the bookmarks would have changed. I haven't renamed or moved the folders in question, nor have I done anything to the volume itself (as far as I know) that would have caused bookmarks to break. But I tried your solution of using a glob pattern, and it seems to be working. The folder itself gets backed up, but nothing in the folder does, so that's fine. Thanks!

-- Bruce

I have an archive file called "Main Backup.quanta" which I use for backing up my entire hard drive (and home folder). I have a separate archive file called "Virtual Machines.quanta", which I use to backup my Virtual Machines folder (from VMware Fusion). Since I only want the VM files to get backed up to the Virtual Machines archive, I've gone into the archive settings for Main Backup, and excluded the Virtual Machines folder. Specifically, in the Exclude section, under Specific Items, I have "bgiles > Documents > Virtual Machines".

This works for a while, and then at some point it stops working. I noticed because my home folder backups seemed to be taking a LOT longer, and then I noticed that the VM folder was appearing in the Activity Monitor window. I checked the log file for the capture that was currently running, and found this:

Filter cannot make reference to item Virtual Machines
path: /Users/bgiles/Documents/Virtual Machines

And sure enough, the folder was now being archived in the Main Backup file. The first time this happened, it was just after upgrading to Catalina and the Catalina version of QRecall, so I deleted the exclude for the folder, and then re-added it, and then checked the next backup session to confirm that the folder was indeed being excluded. But sometime later, I'm not sure how long it took, the exclude quit working again, with the same message in the log. What's going on, and how do I fix it? I'm sending a report as well, in case that provides some additional information you need. Thanks!

-- Bruce

I just upgraded my iMac to Catalina, and the next time I ran QRecall, it prompted me to download the Catalina-compatible version, so that was good. It seems to be working, but I did run into one issue I didn't expect. The release notes did warn that after a Catalina upgrade, with APFS volumes, QRecall would see your drive as a new volume with the same name, and you could use the "Combine Items" function to fix that. When I ran my first home folder recapture action after upgrading, I could quickly see that it was recapturing EVERYTHING, so I decided to let it run and I went to bed. I checked it the next morning, and it hadn't finished. There was a dialog on the screen saying '"QRecallScheduler" would like to access files in your Documents folder.' with buttons for "Don't Allow" and "OK". The recapture had apparently been paused for hours waiting for me to answer the dialog. I clicked the OK button and left the Mac to finish the recapture. About 20 hours after I started the action, I finally got around to looking at it again, and it had once again paused, with a similar dialog. I think this time it was asking about the Desktop folder, but I'm no longer certain. I hit the OK button again, and the next time I checked, about 4 hours later, the recapture had completed with no errors.

So those two requests for access were unexpected, and caused a significant delay. I'm hoping this was a one-time thing, and the problem won't recur.

I had one other small issue, which may affect some people. My backup drive is bootable, and I was storing my archive files in a folder called "QRecall Archives" at the top level of the backup drive. I also upgraded my backup drive (which was also an APFS volume) to Catalina, and when I did so, it didn't like where my archives were stored. They got moved to the "Relocated Items" folder on my desktop. I moved them to the /Users/Shared folder, and adjusted my actions to match, and now everyone is happy again.

By the way, when I moved the archives, they appeared to copy rather than move. I had a copy in /Users/Shared, and I had another copy in the "Relocated Items" folder. My archives were large enough that this shouldn't have been possible -- I didn't have enough free space on the drive to maintain two copies of the archive files. When I deleted the copies in the Relocated Items folder, I checked the free space on the disk and noticed that it didn't change. Apparently the duplicate copies never occupied any space. I'm guessing this is one of the new tricks that APFS provides. Snapshotting maybe? It may have been possible in Mojave and earlier, but if so, I hadn't noticed until now.

-- Bruce

James Bucanek wrote:Well, color me dumbfounded.

Bruce, you were absolutely right. This is not a snapshot issue.

Never doubt me. My gut feelings are better than most people's facts.

It worked! I started both archives on a reindex last night (they each took about 3 hours), and tried a capture with both this morning. They each took about 15 or 20 seconds to open the archive, and then the capture started.

I could almost swear that I tried to repair at least one of the archives a couple of weeks ago, and it didn't help, but maybe I just thought about it, without actually doing it. Or does Reindex do something that Repair doesn't?

I'm not sure why there would be a VM issue, or if there was, why it would slow things down so much. This Mac is a Late 2015 iMac, with a 4 GHz i7 processor, 16 GB RAM, and a 2 TB hard drive with only about 640 GB in use. It's not the current state-of-the-art, but it's no slouch either. I'm usually running a Windows 10 VM (VMware Fusion), which at various times has been allocated from 2 GB to 8 GB, but even when it's running at 8 GB, there's not that much going on in the rest of the iMac.

I'll try to keep a closer eye on the captures, and if they start slowing down again, I'll let you know. Thanks!

-- Bruce

Just now sent one.

-- Bruce

OK, I still don't know what's going on. Before experimenting with tmutil, I decided I ought to check my boot drive. I rebooted into single-user mode, and ran /sbin/fsck -fy. During the "Checking the fsroot tree" phase, it found four errors:

error: directory valence check: directory (old 0xb0069): nchildren (25) does not match drec count (0)
error: directory valence check: directory (old 0x150050): nchildren (25) does not match drec count (0)
error: directory valence check: directory (old 0x180042): nchildren (25) does not match drec count (0)
error: directory valence check: directory (old 0x290064): nchildren (25) does not match drec count (0)

I don't know what that was about, but the errors seem to have been fixed, as they did not appear again when I re-ran fsck. I also booted into Recovery Mode and then used Disk Utility to check both the boot disk and the backup disk. It didn't find problems with either of them.

So I rebooted, and ran QRecall and tried my capture action. The delay still occurred. While I was waiting for something to happen in QRecall, I tried tmutil localsnapshot. It created a snapshot in about a second. I was then able to use listlocalsnapshots to list it, and deletelocalsnapshots to delete it. It took about 30 seconds to delete. QRecall was still waiting for the archive to open, so I created another snapshot, and listed it. It showed up as:

com.apple.TimeMachine.2019-09-13-194448

I tried the list command several more times while I was waiting, and got the same result every time. Finally, after about 10 minutes, the archive opened, and QRecall started capturing. I immediately ran the list command again, and this time I got this:

com.apple.TimeMachine.2019-09-13-194448
com.qrecall.capture.596.590111295

Finally, the QRecall snapshot is listed. From this, it would appear that the delay is indeed occurring in the time between when the action starts and when the snapshot is created, but I don't know why, since I was able to quickly manually create and delete snapshots during the period when QRecall appeared to be hanging. When the QRecall capture finished, I verified that the QRecall snapshot was no longer listed, and then I deleted the one I had manually created.

Next, I went into QRecall's advanced preferences and set "Capture a snapshot" to false. Tried another capture, and the 10-minute delay still occurred. Again I used tmutil to list snapshots, and confirmed that none were created by QRecall this time. The capture finished with 4 capture errors. All were "Unable to read extended attribute data". I suspect these were due to the snapshots being turned off. I went back to QRecall's preferences, and set "Capture a snapshot" back to true. Did another capture, and discovered that the snapshot still wasn't being created, even though the preference was now set. Maybe it needs a reboot? So I rebooted, and did another capture and confirmed that the delay still occurred, but QRecall is again creating snapshots, and the capture errors did not appear this time.

From all of the above, it seems pretty evident that snapshots are not the problem. The delay occurs whether QRecall creates snapshots or not. I just now had QRecall send you a report. Maybe that'll provide a clue.

-- Bruce

Hi James,

It's really not a problem -- it's just something I notice frequently because it shows up in the Activity window. But I'm not convinced that "other changes being made to the volume at the same time" is the culprit. This happens, as far as I can tell, EVERY time the action runs, whether scheduled or manually initiated, even when the computer is essentially idle. I also have a second archive, which just backs up my Virtual Machines folder, and I have the same issue with that one as well, although the delay period is shorter.

I will try disabling the snapshots just to see if it makes a difference, and I'll report back after I try that. If I tried manually snapshotting a volume from the command line, would I likely see the same delay? (I don't know how to do that, but presume, between the MAN file and Google, I could figure it out.)

-- Bruce

When I open my QRecall archive (about 500 GB in size) from the Open menu, or the QuickStart window, it takes about 10 SECONDS to open. But when I run an action that uses this archive, the Activity window sticks at "Opening archive" for about 10 MINUTES. Any idea why this is happening?

Here's a brief excerpt from the log file that shows what's happening:

Action 2019-09-12 23:00:07 ------- Capture Home Folder
Action 2019-09-12 23:00:07 archive: /Volumes/Seagate 2GB Backup/QRecall Backups/Macintosh HD.quanta
Action 2019-09-12 23:10:40 Minutia Took snapshot of volume Macintosh HD
Action 2019-09-12 23:10:40 Capture bgiles

Thanks for the clarification. Now I've got a better understanding of what's going on.

However, scheduling a capture action to start as soon as the VM app quits doesn't quite work for me. The problem is that I don't usually quit it until I'm done for the day (this is a work computer) and ready to head home. And it can take 30 minutes or longer for the capture to complete. So I'm thinking what I should do is schedule the capture to occur very early in the morning, at a time before I normally arrive at work. (The computer is shut down when I'm not there.) Then when I arrive at work and boot up the computer, the action would run immediately. I wait a few seconds to give the capture action time to generate the snapshot, then I can start the VM. Does that sound reasonable?

I use QRecall to capture my VMware virtual machine file. It was my understanding (from knowledge I picked up years ago), that virtual machine files should not be backed up while in use. The problem, as I understood it (which means I may be wrong) was that the backups could be corrupted in some way, so that if you ever had to restore the VM file from a backup, you might find out that the restored VM no longer worked. I think the general idea was that it takes a long time to backup a VM file, so the state of the file is constantly changing while you're backing it up. Thus, I've always had an archive setting in my archive file to not capture the folder containing my virtual machine files, so I don't have to worry about a capture action running and capturing up a VM file while it's in use. Then I have a separate archive file just for capturing the VM files. I only capture the VM files after I've exited all VMs, and then I manually run my action that captures the VM folder to the second archive.

Recently, I've upgraded all my volumes to APFS volumes, and I noticed that the release notes for one of the recent updates mentioned that QRecall now takes a snapshot of volumes before capturing them. I don't yet fully understand what APFS snapshots are and how they work, but I'm wondering if this means I can simplify the process I mentioned in the first paragraph. Is it now safe to capture a VM while it's in use? Or was that never really a problem to start with? Can I dispense with my second archive and second action, and just capture the VM folder normally to the primary archive, without having to worry about whether the VM is running or not? What's the "best practice" here?

-- Bruce