QRecallDownloadIdentity KeysForumsSupport
  [Search] Search   [Recent Topics] Recent Topics   [Hottest Topics] Hottest Topics   [Groups] Back to home page 
Messages posted by: Bruce Giles
Forum Index » Profile for Bruce Giles » Messages posted by Bruce Giles
Author Message
Several of my backup drives are getting old, and I'm going to need to replace them soon. James, it has occurred to me that you must have a lot of experience with backup drives. Do you have any recommendations for what drives work best with QRecall? Are there any particular manufacturers or models you recommend, and are there ones to avoid? How about spinning hard drives versus solid state drives? What about shingled magnetic recording (SMR) drives? Do you prefer ready-to-use external drives (includes drive and case as a unit), or do you prefer to buy a bare drive and put it in an external case yourself? What about NAS drives, RAID, or Drobo units? Personally, I'm looking for something in the 2 terabyte range and I would prioritize reliability and speed over cost.

-- Bruce
I'm still not sure why the bookmarks would have changed. I haven't renamed or moved the folders in question, nor have I done anything to the volume itself (as far as I know) that would have caused bookmarks to break. But I tried your solution of using a glob pattern, and it seems to be working. The folder itself gets backed up, but nothing in the folder does, so that's fine. Thanks!

-- Bruce
I have an archive file called "Main Backup.quanta" which I use for backing up my entire hard drive (and home folder). I have a separate archive file called "Virtual Machines.quanta", which I use to backup my Virtual Machines folder (from VMware Fusion). Since I only want the VM files to get backed up to the Virtual Machines archive, I've gone into the archive settings for Main Backup, and excluded the Virtual Machines folder. Specifically, in the Exclude section, under Specific Items, I have "bgiles > Documents > Virtual Machines".

This works for a while, and then at some point it stops working. I noticed because my home folder backups seemed to be taking a LOT longer, and then I noticed that the VM folder was appearing in the Activity Monitor window. I checked the log file for the capture that was currently running, and found this:

Filter cannot make reference to item Virtual Machines
path: /Users/bgiles/Documents/Virtual Machines

And sure enough, the folder was now being archived in the Main Backup file. The first time this happened, it was just after upgrading to Catalina and the Catalina version of QRecall, so I deleted the exclude for the folder, and then re-added it, and then checked the next backup session to confirm that the folder was indeed being excluded. But sometime later, I'm not sure how long it took, the exclude quit working again, with the same message in the log. What's going on, and how do I fix it? I'm sending a report as well, in case that provides some additional information you need. Thanks!

-- Bruce
I just upgraded my iMac to Catalina, and the next time I ran QRecall, it prompted me to download the Catalina-compatible version, so that was good. It seems to be working, but I did run into one issue I didn't expect. The release notes did warn that after a Catalina upgrade, with APFS volumes, QRecall would see your drive as a new volume with the same name, and you could use the "Combine Items" function to fix that. When I ran my first home folder recapture action after upgrading, I could quickly see that it was recapturing EVERYTHING, so I decided to let it run and I went to bed. I checked it the next morning, and it hadn't finished. There was a dialog on the screen saying '"QRecallScheduler" would like to access files in your Documents folder.' with buttons for "Don't Allow" and "OK". The recapture had apparently been paused for hours waiting for me to answer the dialog. I clicked the OK button and left the Mac to finish the recapture. About 20 hours after I started the action, I finally got around to looking at it again, and it had once again paused, with a similar dialog. I think this time it was asking about the Desktop folder, but I'm no longer certain. I hit the OK button again, and the next time I checked, about 4 hours later, the recapture had completed with no errors.

So those two requests for access were unexpected, and caused a significant delay. I'm hoping this was a one-time thing, and the problem won't recur.

I had one other small issue, which may affect some people. My backup drive is bootable, and I was storing my archive files in a folder called "QRecall Archives" at the top level of the backup drive. I also upgraded my backup drive (which was also an APFS volume) to Catalina, and when I did so, it didn't like where my archives were stored. They got moved to the "Relocated Items" folder on my desktop. I moved them to the /Users/Shared folder, and adjusted my actions to match, and now everyone is happy again.

By the way, when I moved the archives, they appeared to copy rather than move. I had a copy in /Users/Shared, and I had another copy in the "Relocated Items" folder. My archives were large enough that this shouldn't have been possible -- I didn't have enough free space on the drive to maintain two copies of the archive files. When I deleted the copies in the Relocated Items folder, I checked the free space on the disk and noticed that it didn't change. Apparently the duplicate copies never occupied any space. I'm guessing this is one of the new tricks that APFS provides. Snapshotting maybe? It may have been possible in Mojave and earlier, but if so, I hadn't noticed until now.

-- Bruce
James Bucanek wrote:Well, color me dumbfounded.

Bruce, you were absolutely right. This is not a snapshot issue.


Never doubt me. My gut feelings are better than most people's facts.

It worked! I started both archives on a reindex last night (they each took about 3 hours), and tried a capture with both this morning. They each took about 15 or 20 seconds to open the archive, and then the capture started.

I could almost swear that I tried to repair at least one of the archives a couple of weeks ago, and it didn't help, but maybe I just thought about it, without actually doing it. Or does Reindex do something that Repair doesn't?

I'm not sure why there would be a VM issue, or if there was, why it would slow things down so much. This Mac is a Late 2015 iMac, with a 4 GHz i7 processor, 16 GB RAM, and a 2 TB hard drive with only about 640 GB in use. It's not the current state-of-the-art, but it's no slouch either. I'm usually running a Windows 10 VM (VMware Fusion), which at various times has been allocated from 2 GB to 8 GB, but even when it's running at 8 GB, there's not that much going on in the rest of the iMac.

I'll try to keep a closer eye on the captures, and if they start slowing down again, I'll let you know. Thanks!

-- Bruce
Just now sent one.

-- Bruce
OK, I still don't know what's going on. Before experimenting with tmutil, I decided I ought to check my boot drive. I rebooted into single-user mode, and ran /sbin/fsck -fy. During the "Checking the fsroot tree" phase, it found four errors:

error: directory valence check: directory (old 0xb0069): nchildren (25) does not match drec count (0)
error: directory valence check: directory (old 0x150050): nchildren (25) does not match drec count (0)
error: directory valence check: directory (old 0x180042): nchildren (25) does not match drec count (0)
error: directory valence check: directory (old 0x290064): nchildren (25) does not match drec count (0)

I don't know what that was about, but the errors seem to have been fixed, as they did not appear again when I re-ran fsck. I also booted into Recovery Mode and then used Disk Utility to check both the boot disk and the backup disk. It didn't find problems with either of them.

So I rebooted, and ran QRecall and tried my capture action. The delay still occurred. While I was waiting for something to happen in QRecall, I tried tmutil localsnapshot. It created a snapshot in about a second. I was then able to use listlocalsnapshots to list it, and deletelocalsnapshots to delete it. It took about 30 seconds to delete. QRecall was still waiting for the archive to open, so I created another snapshot, and listed it. It showed up as:

com.apple.TimeMachine.2019-09-13-194448

I tried the list command several more times while I was waiting, and got the same result every time. Finally, after about 10 minutes, the archive opened, and QRecall started capturing. I immediately ran the list command again, and this time I got this:

com.apple.TimeMachine.2019-09-13-194448
com.qrecall.capture.596.590111295

Finally, the QRecall snapshot is listed. From this, it would appear that the delay is indeed occurring in the time between when the action starts and when the snapshot is created, but I don't know why, since I was able to quickly manually create and delete snapshots during the period when QRecall appeared to be hanging. When the QRecall capture finished, I verified that the QRecall snapshot was no longer listed, and then I deleted the one I had manually created.

Next, I went into QRecall's advanced preferences and set "Capture a snapshot" to false. Tried another capture, and the 10-minute delay still occurred. Again I used tmutil to list snapshots, and confirmed that none were created by QRecall this time. The capture finished with 4 capture errors. All were "Unable to read extended attribute data". I suspect these were due to the snapshots being turned off. I went back to QRecall's preferences, and set "Capture a snapshot" back to true. Did another capture, and discovered that the snapshot still wasn't being created, even though the preference was now set. Maybe it needs a reboot? So I rebooted, and did another capture and confirmed that the delay still occurred, but QRecall is again creating snapshots, and the capture errors did not appear this time.

From all of the above, it seems pretty evident that snapshots are not the problem. The delay occurs whether QRecall creates snapshots or not. I just now had QRecall send you a report. Maybe that'll provide a clue.

-- Bruce
Hi James,

It's really not a problem -- it's just something I notice frequently because it shows up in the Activity window. But I'm not convinced that "other changes being made to the volume at the same time" is the culprit. This happens, as far as I can tell, EVERY time the action runs, whether scheduled or manually initiated, even when the computer is essentially idle. I also have a second archive, which just backs up my Virtual Machines folder, and I have the same issue with that one as well, although the delay period is shorter.

I will try disabling the snapshots just to see if it makes a difference, and I'll report back after I try that. If I tried manually snapshotting a volume from the command line, would I likely see the same delay? (I don't know how to do that, but presume, between the MAN file and Google, I could figure it out.)

-- Bruce
When I open my QRecall archive (about 500 GB in size) from the Open menu, or the QuickStart window, it takes about 10 SECONDS to open. But when I run an action that uses this archive, the Activity window sticks at "Opening archive" for about 10 MINUTES. Any idea why this is happening?

Here's a brief excerpt from the log file that shows what's happening:

Action 2019-09-12 23:00:07 ------- Capture Home Folder
Action 2019-09-12 23:00:07 archive: /Volumes/Seagate 2GB Backup/QRecall Backups/Macintosh HD.quanta
Action 2019-09-12 23:10:40 Minutia Took snapshot of volume Macintosh HD
Action 2019-09-12 23:10:40 Capture bgiles
Thanks for the clarification. Now I've got a better understanding of what's going on.

However, scheduling a capture action to start as soon as the VM app quits doesn't quite work for me. The problem is that I don't usually quit it until I'm done for the day (this is a work computer) and ready to head home. And it can take 30 minutes or longer for the capture to complete. So I'm thinking what I should do is schedule the capture to occur very early in the morning, at a time before I normally arrive at work. (The computer is shut down when I'm not there.) Then when I arrive at work and boot up the computer, the action would run immediately. I wait a few seconds to give the capture action time to generate the snapshot, then I can start the VM. Does that sound reasonable?
I use QRecall to capture my VMware virtual machine file. It was my understanding (from knowledge I picked up years ago), that virtual machine files should not be backed up while in use. The problem, as I understood it (which means I may be wrong) was that the backups could be corrupted in some way, so that if you ever had to restore the VM file from a backup, you might find out that the restored VM no longer worked. I think the general idea was that it takes a long time to backup a VM file, so the state of the file is constantly changing while you're backing it up. Thus, I've always had an archive setting in my archive file to not capture the folder containing my virtual machine files, so I don't have to worry about a capture action running and capturing up a VM file while it's in use. Then I have a separate archive file just for capturing the VM files. I only capture the VM files after I've exited all VMs, and then I manually run my action that captures the VM folder to the second archive.

Recently, I've upgraded all my volumes to APFS volumes, and I noticed that the release notes for one of the recent updates mentioned that QRecall now takes a snapshot of volumes before capturing them. I don't yet fully understand what APFS snapshots are and how they work, but I'm wondering if this means I can simplify the process I mentioned in the first paragraph. Is it now safe to capture a VM while it's in use? Or was that never really a problem to start with? Can I dispense with my second archive and second action, and just capture the VM folder normally to the primary archive, without having to worry about whether the VM is running or not? What's the "best practice" here?

-- Bruce
James Bucanek wrote:My concern is that there's a lot of stuff in /Library and /var/db that you might want and just migrating /Users and /Groups is not going to include.


Ah, now I understand. For this server, I don't think it's going to be a problem. A few years ago, we used it for a lot of things -- it was our network firewall, it was a license server, it was our DHCP server, and our DNS server, etc. But over time we moved all those functions over to other hardware. The only thing the Xserve does any more is act as a file server and run OpenDirectory. The users are all running Windows machines, and they don't even know it's an Apple server running Mac OS. They store only data files on it. We no longer run any third-party apps on it other than QRecall, or even any Apple apps other than OS utilities. So as long as we can get all the data transferred to the mini, I don't think there's anything else we'll need from the old server. But like you said, if that turns out to be wrong, there's always Plan B.

-- Bruce
I'm not going to transfer over ANY of the old operating system or server applications. In fact, I don't think that would work, even if I wanted to. The Mac mini is much newer than the Xserve, and while I might be able to use QRecall to copy Snow Leopard Server onto it, I doubt it would boot. The mini already has the latest High Sierra OS and the latest Server app, and I intend to stick with those.

What I do want to transfer is all the user (/Users) and group (/Groups) folders, along with the OpenDirectory database. (I'm somewhat concerned that the OpenDirectory database format may have changed enough I can't import the old database into OpenDirectory in the current Server app, but that's my problem.) If I can get all the User and Group folders transferred, that will accomplish most of what I need to do.

So you're saying that if I install the latest QRecall on the mini, and then connect the external drive that contains the backup created by QRecall 1.2.3.8, I should be able to restore those files and folders to the mini?

Also, will it make any difference if the server drives are APFS?

-- Bruce
James,

I have an old Xserve running Snow Leopard Server 10.6.8. It's been successfully backed up for years with QRecall, and it's currently running QRecall version 1.2.3.8. I'm planning to replace the Xserve with a newer Mac mini, running the latest version of High Sierra. Somehow, I need to transfer the top-level /Groups and /Users folders from the Xserve to the mini, along with the OpenDirectory database (I have the Server app on the mini.) I do not need to transfer system files, applications, or anything else. The backup archive is on an external drive.

The OpenDirectory database is small, and I have an export of it that I'm hoping will load into the Server app on the mini. I can transfer the database on a USB stick. The tricky part, I think, will be moving the Groups and Users folders, which have about 750 GB of data. Ideally, all files and folders need to retain their group and user IDs and protections, so that I don't have to reset all that after I get them moved.

Can I do that with QRecall? If so, do you have any recommendations? For example, should I load the old version of QRecall on the mini (if that will even work) and use that to restore the files to the mini? Or can I put the latest QRecall on the mini and then use that to restore the old archive? Or is there a better way?

-- Bruce
I see you've implemented it now, and it's working. Thanks!

-- Bruce
 
Forum Index » Profile for Bruce Giles » Messages posted by Bruce Giles
Go to:   
Powered by JForum 2.1.8 © JForum Team