I seem to have gotten myself into a box that I can't see how to get out of. I decided to encrypt a 700gb archive on a 2tb drive with about 1.2gb free space, which seemed like plenty to allow the encryption. The process quit on me a couple of times, but I was able to restart it without difficulty. When I was ready to go to bed at about 11pm it still had a ways to go, so I triggered a backup action thinking that it would start after the encryption completed. I got up this morning to find that the backup had apparently run immediately, stopping the encryption in the process. When I tried to restart the encryption I got an "Insufficient free space to duplicate archive" error, because the partially encrypted archive size is now 1.2tb and I have less than 600gb free space.

What's my best course forward at this point? I assume that the archive is as big as it is because it now contains both unencrypted and encrypted data. Would merging layers or compacting the archive clean it out to the point that I could continue the encryption?

Ralph

Ralph Strauch wrote:What's my best course forward at this point? I assume that the archive is as big as it is because it now contains both unencrypted and encrypted data.

Ralph,

Thanks for sending a diagnostic report. From the logs, I can see that your encryption action is failing because the macOS encryption services are crashing. Some systems?and I don't yet have a sense for which ones?have a great deal of difficulty executing multiple encryption/description operations simultaneously. Sometimes the operations fail (which QRecall can deal with), but for some users the operations crash.

So what's happening is that your encryption action is crashing. For safety, Qrecall first makes an encrypted copy of the archive data. So if there is a problem, you won't be left with a half-encrypted archive. That intermediate copy is what's using up your disk space.

First, let's fix the disk space problem

Control/Right+click the archive in the Finder and choose Show Package Contents

Trash any files that begin with the name repository_scribble

Close the window and empty the trash

Now you should be able to start a new encryption task. But to keep that from crashing again, you'll want to limit the number of concurrent encryption operations QRecall will perform.

Make sure you've upgraded to QRecall 2.0.7

Go to QRecall > Preferences > Advanced

Find the Concurrent Encryption Limit setting and set it to 1 or 2

Now try the encrypt command again.

You can try adjusting the concurrent limit to a higher value, just realize that if it's too high some QRecall actions will randomly crash. One of the best ways to test a higher setting is using a verify command; verify performs a lot of concurrent encryption operations and there won't be any data loss if it crashes.

As always, your advice is spot on. My archive is now encrypted and I'm back in business. I'm switching over to archives encrypted by qrecall from unencrypted archives stored on FileVault encrypted drives so that I can share the backup drive through a new router that will hopefully resolve the intermittent problems I've been having with network backups for the past couple of years.

After my encryption completed I compacted the archive, which created a problem requiring a repair. The repair logged several additional problems, but eventually completed successfully. I'm back in business now and everything is fine, but I'm sending this report in case any of the diagnostic info is of value to you. I don't need any response.

After repairing my archive I copied the archive back to my reformatted drive and mounted it on my new router. I can now see the drive in the Finder, but Qrecall doesn't seem to be able to connect to the archive. When I try to launch a backup from the action menu, the Monitor Window show "opening archive" or "waiting for archive" and goes no further. If I just to open the archive from the File menu or by clicking on the archive I get a "waiting for archive" window or sometimes just a spinning beachball requiring a force-quit.

Apparently this is a result of the way router presents the file to Qrecall. The disk shared through the router doesn't show up as a device is Finder, but is accessed through Go/Connect to Finder where it shows up as an smb:// connection to my ISP's router. That connection shows up in Finder as a Volume, listing the contents of the drive. I can open most of the files listed, but not the archive.

Is there a way to connect to this router that will work with qrecall, or do I just have to go back to my previous setup of mounting the archive directly on my iMac? The router, if I can make it work, will give me a USB3 connection instead of the USB2 connection the iMac provides, and hopefully be more reliable than my previous router connection has been.

Ralph,

I don't think that the problem is how the volume is mounted. Basically, QRecall (and most software) doesn't care how the volume appears in the Finder or what its mountpoint is. Software (QRecall) gets a path to the document, and it uses that path to manipulate the files. The rest is unimportant details (to the software).

I suspect you have a permissions problem or your SMB server doesn't support the necessary file locking features.

QRecall is getting stuck trying to obtain exclusive access to the archive. It uses several techniques to do this, because not all filesystem support the same file locking features. In your case, it's getting stuck trying to obtain a "distributed lock". The exact mechanics of a distributed lock vary from one filesystem to the next, but most involve create a "lock" file used to coordinate access from multiple clients. This what I found in your log:

2016-12-17 21:32:43.883 breaking distLock after 150 tries

2016-12-17 21:33:51.114 breaking distLock after 150 tries

2016-12-17 21:35:00.437 breaking distLock after 150 tries

2016-12-17 21:36:11.570 breaking distLock after 150 tries

This should never happen, and if it does it should only happen once; once a stale distLock is broken, it should start working again immediately, but clearly this one still isn't.

When a distributed lock is implemented as a file, it must have read and write access to that directory. You can test this in the Terminal. With the archive mounted on your SMB volume, issue these commands:

user$ cd <Drag and drop your archive icon here>

user$ touch .semaphore

user$ rm .semaphore

The .semaphore file is the filesystem object used to coordinate the lock. If you can modify it and delete it as a user, then it should work. If any of these commands report errors, then I suspect you have a permissions or access issue. QRecall documents are just like any other file; your user account must have read, write, and search permission on the contents of the archive package in order to update it.

Now if these commands all work just fine, and the rest of the archive package has the correct ownership and permissions, then I'm stumped. I would suspect that something about the file locking features implemented by your SMB server doesn't line up with what macOS is expecting.

I ran the commands you suggested and get no error messages, but still cannot access the archive

[cpe-172-248-32-105:~] ralph% cd /Volumes/volume2
[cpe-172-248-32-105:/Volumes/volume2] ralph% touch .semaphore
[cpe-172-248-32-105:/Volumes/volume2] ralph% rm .semaphore
[cpe-172-248-32-105:/Volumes/volume2] ralph%

The permissions for both the archive and the volume containing it show up in Finder as "custom access," and in terminal as

[cpe-172-248-32-105:/volumes/volume2] ralph% ls -l
total 4160
drwx------ 1 ralph staff 16384 Dec 18 03:10 3rd backup.quanta

but I don't understand Unix permissions well enough to know If that's something I should try to change that or not. I also notice that I'm apparently running a shell provided by the router, rather than one from my computer. When I run terminal with the router disconnected I get "[Ralphs-rMBP:~] ralph%" and as soon as I connect to the router it changes to the one above.

Maybe it's time for me to take this to the router's support group. Is there anything else I should tell them about the problem that might help solve it from their end?

Ralph,

Close (but no cigar)

You need to test the access to the directory inside the archive package. Try these commands:

cd '/Volumes/volume2/3rd backup.quanta'

touch .semaphore

rm .semaphore

However, since it appears that the archive belongs to you, and I assume that you're performing the capture from the same computer using your account, it should work. Which might mean we're back to square one.

OK, this time I got a "permission denied."

[cpe-172-248-32-105:/Volumes/volume2] ralph% cd '/Volumes/volume2/3rd backup.quanta'
[cpe-172-248-32-105:/Volumes/volume2/3rd backup.quanta] ralph% touch .semaphore touch: .semaphore: Permission denied
[cpe-172-248-32-105:/Volumes/volume2/3rd backup.quanta] ralph% rm .semaphore
rm: .semaphore: No such file or directory

Is changing that permission something I can do in a way that will stick, or do I need to ask the router people for a change? And if so, what specifically should I ask them for?

Ralph Strauch wrote:OK, this time I got a "permission denied."

Then that's the problem. From the perspective of the router's file server, you do not have permission to modify files inside the archive package directory. This pretty much rules out capturing data to it.

Soooooo

If you are the only computer & users capturing to this archive, then there might be a simple solution. Let's try this:

chown -R $(id -u) '/Volumes/volume2/3rd backup.quanta'

This will change ownership of the archive package directory, and all of the files it contains, to the user account you are currently logged in as. After that, try to accessing/capturing to the archive.

Again, if you're the only user/system accessing this archive that might fix the problem. If you share this archive with other users, then that complicates things a bit.

I don't think that solution will work for me. because I also capture my wife's iMac to this same archive from her user account, I think it's more than a permissions problem, though, because I can open and change other files on the same drive. The Qrecall archive is the only file that won't open, so perhaps the problem is somehow with the archive's structure.

The router apparently does not support MacOS file versions, because when I edit a text file in TextEdit and save it, I get an error message that says:

The document ?focusing_your_touch1.txt? is on a volume that does not support permanent version storage.
You will not be able to access older versions of this document once you close it.

The file does save, though.

I'll give TP-link support a try and see if they have any ideas. Is there anything in particular that it might be useful to tell them about the archive?

Ralph

Ralph,

Version support is a red herring. This is only supported on macOS formatted volumes because it's a really complex feature. But it won't interfere with QRecall. QRecall tries to use only the very basic filesystem features so it can be compatible with a wide range of filesystems, servers, and volumes formats. When it does use a fancy filesystem features (like file range locking or atomic swaps), it implements fallback methods to accomplish the same thing when it finds itself using a filesystems that don't support them.

I'll assert that your problems are entirely ones of ownership and permissions, so here's the short course.

On a volume the honors ownership and permissions, you can access the files you own or have been granted access to by its owner. (There's a lot of exceptions and caveats to that rule, but this is the simple explanation.) When you create a file, you own it by default.

So Amy creates a file, but Bob can't read or write it because Amy didn't grant Bob access. This is the default/typical file ownership rule at work, and what keeps users of your "Guest" account from reading your email. The same rule applies to QRecall archives.

So Amy creates an archive, but Bob can't modify it.

Now if both Amy and Bob need to capture to the same archive, it's typically because the archive is on an external drive or a shared volume. An external drive is easy to fix by setting the drive to "Ignore ownership and permissions" on both systems. Now both computers can freely access the archive modified by the other.

On shared volumes things get a little more complicated. If both Amy and Bob both have their own accounts on the server, they have the same problem as before; Amy's archive will belong to Amy. Permissions, however, are managed by the server and there will be no option to ignore them.

The simplest solution is for Bob to sign onto the server as Amy, vice versa, or create a neutral account (Pat) that both users can connect to the server. When connected to a file server, the files you create belong to the account you authenticated as, not your user account. This is easy to forget as most people use the same account name on the server as they do on their computer. The complication here is if both Amy and Bob need to connect to the server using their account for other purposes. Most file servers either do not support, or make it difficult to, open multiple connections to the same server using different accounts.

Another possible solution is to change the umask of your account. The umask adjusts the default permissions granted to new files. You can set it up so that other users in your group (or everyone) can share files you create by default. Unfortunately, this is a global setting that also affects the files you create locally on your hard drive, so it might not be what you want and has obvious security implications.

If you're only using your router/server for QRecall backups, the most manageable solution is to have all clients connect to the server using the same account. (You'll wan to save that account name and password in your keychain too.) Now change the ownership of the archive to the account you all share, and you should have trouble-free access to that archive from all of your systems. How you accomplish that with your particular NAS/Server is an exercise left to the reader.

I've now mounted the backup drive directly on the MBP and run a backup there, which took almost 11 hours and created a new layer containing 204gb, which is about 10x what I would have expected. The Qrecall log did not record the completion of that backup. The drive being backed up contains 360GB and the archive now shows in Finder as 713GB. This is about what it was before the backup, so I'm not sure where the new layer went. It's somewhere, though, because I can see it when I open the archive, and can restore files from it, (I should have been making notes during this process, but wasn't.) The new layer contains files that haven't changed in years, so I don't know what criteria Qrecall was using for changed files.

At this point my best solution may be to return my new router and go back to my old system, so my only questions now are where the 204gb backup went and if the fact that it doesn't show as completed in the log is anything I should worry about?

When you look at my report you'll find a number of failed "waiting for archive" backup attempts. I caused those by trying to run a "backup to router" action while the drive was mounted on my computer, and not realizing why it wasn't going through.

Ralph Strauch wrote:I've now mounted the backup drive directly on the MBP and run a backup there, which took almost 11 hours and created a new layer containing 204gb, which is about 10x what I would have expected.
<clip>
The drive being backed up contains 360GB and the archive now shows in Finder as 713GB. This is about what it was before the backup, so I'm not sure where the new layer went. It's somewhere, though, because I can see it when I open the archive, and can restore files from it, (I should have been making notes during this process, but wasn't.) The new layer contains files that haven't changed in years, so I don't know what criteria Qrecall was using for changed files.

What's happening is QRecall is recapturing every file on your hard drive because the identity key is different. An archive is organized by owners (define by identity keys), which contain volumes, which contain files and folders. If your identity key changes QRecall treats your files as if they were from a completely different computer and recaptures them all. These new files will be contained within the new owner and volume at the root of the archive.

Use the browser bar at the bottom to navigate to the root of the archive to see the new owner and volumes. Use the Combine Items command to stitch two owners, or two volumes, that are actually the same into a single history.

Alternatively, you can reinstall the original identity key and QRecall will resume updating the history you already have. If you're wondering which key you have installed, go to QRecall > Preferences > Identity key, hold down the Option key while clicking on the Enter Key button. In the "key" field, you'll see a light grey number. That is your key's serial number. Now use this Magic Account URL on the QRecall website, and it will list the serial number of each of your identity keys.

The reason the archive isn't (much) bigger is because it's all duplicate data. (Duplicates of that "other" computer's files already in the archive.)

The Qrecall log did not record the completion of that backup.

That's because the capture isn't finishing. The capture action is crashing trying to encrypt data. Try setting your Concurrent Encryption Limit setting to 1 and run the capture again. Meanwhile, I'll fire off another blistering bug report to Apple.

At this point my best solution may be to return my new router and go back to my old system, so my only questions now are where the 204gb backup went and if the fact that it doesn't show as completed in the log is anything I should worry about?

Well, we should be worried about the encryption framework crashing on you. That's not productive at all.

If you didn't mean to change identity keys, that's something that you should look into. Aside from the annoyance of recapturing your entire hard drive and having separate history, or taking the trouble to combine them, it doesn't bother QRecall?but it might bother you.

Honestly, I would hope that you can get this working on your router. If it's possible for all of the QRecall clients to connect to the router with the same authentication, they should be able to peacefully share the single archive.

An archive is organized by owners (define by identity keys), which contain volumes, which contain files and folders. If your identity key changes QRecall treats your files as if they were from a completely different computer and recaptures them all. These new files will be contained within the new owner and volume at the root of the archive.

I originally purchased two identity keys to use one for each computer, but didn't keep track of which was which. As some point in my wrestling with the new router, Qrecall said I didn't have an identity key and needed to reinstall one. I picked the wrong one so ended up with the same key on both machines. I think I've that sorted out now, and am back to one key per computer. I assume I can just delete the extra layer that I created unintentionally without affecting the rest of the backup

The capture action is crashing trying to encrypt data. Try setting your Concurrent Encryption Limit setting to 1 and run the capture again. Meanwhile, I'll fire off another blistering bug report to Apple.

I had set the Current Encryption Limit to 2 after you suggested 1 or 2, but I'll take it down to 1 now.

Honestly, I would hope that you can get this working on your router. If it's possible for all of the QRecall clients to connect to the router with the same authentication, they should be able to peacefully share the single archive.

I'm backing up two computers to the same archive, a MBP and an iMac. I'm the sole user of the MBP and back it up from my account. My wife is the primary user of the iMac and I've been running that backup from her account. I seldom even log onto the iMac, but it looks like Qrecall could run scheduled backups of the whole iMac from my account without my being logged in. Is that correct, and should that satisfy the router's desire for a single common user? (I'm uid 501 on both machines.)

One final item -- my archive now won't open from my wife's account on the iMac, even though it opens fine in my iMac account and on my MBP.

I'm off to a week in the desert tomorrow or Sunday, so may not get back to this till next year. Thanks again for all the hand-holding you provide. I really appreciate it.

Ralph