Ralph,

Thanks for the update. I've got this on the "to be looked into" list, but haven't gotten to it yet.

And sorry about the thread: the beta announcement topic wasn't supposed to be open for comments, so I locked it after your post.

Bruce Giles wrote:I'm less sure I like that behavior with invisible folders. I may be misunderstanding your intent here, so let me ask a direct question. If I'm backing up my entire hard drive, then am I just going to see "Private Folder" for the entire time while it backs up /bin, /cores, /dev, /etc, /home, /mach, /net, /opt, /private, /sbin, /tmp, /usr, and /var? If that's correct, that would NOT match my expectations. If I saw "Private Folder" listed in the activity window while it did all that, I'd probably be wondering what was wrong.

Let me clarify. The "Private Folder" was just a hypothetical example; the status will display the actual name of whatever invisible or unreadable folder is being captured. When the capture encounters an invisible (/private) or unreadable (/Users/somebodynotyou/Desktop) directory, it will display that directory's name ("private" or "Desktop") until it's finished with that directory subtree.

Now (well, soon) the idea is that the items displayed in the capture progress are whatever you could normally encounter by browsing in the Finder (without using any hacks or special Finder commands like Option+Command+L or "Show Package Contents").

Examples would be a .git folder (all dot named directories are implicitly invisible) and ~/Library, because that folder is explicitly made invisible.

I like this because the default will be to honor the access restrictions for the regular user. So even though your QRecall account might, technically, have permission the access all of the documents for other users, it seems polite not to have those filename flashed on the screen.

Anyway, you're welcome to set QRMonitorInvisibleFolderProgress if you don't like it.

Bruce Giles wrote:At first I thought it might be because I'm backing up a folder that contains only a single enormous file -- the VMware virtual machine file. But that file is actually a package containing many files, only one of which is screenshot_0000.jpg.

Ah yes, packages.

This reminded me that I've had a bug report on the books for eons that the file progress doesn't update correctly inside of packages (and I suspect invisible folders).

I started looking into this, and it was like pulling a thread on a sweater; the more I pulled, the more the logic didn't make any sense.

So I rewrote it all today. Here's how it work (in the next beta). This is a little on the experimental side, so I'm looking for feedback.

QRecall no longer reports the progress of individual items inside a directory if that directory is either opaque (package, application, bundle, etc.) or if it is either invisible or inaccessible as a regular user. Instead, it will simply report that it is capturing "Private Folder" or "Adobe Photoshop.app".

Also—and this is the experimental part—it will no longer collect icon or localized display information about these files. The advantage is a (marginal) capture performance improvement. But it also means that if you enable the viewing of invisible items or packages in the archive browser, you won't see pretty icons or localized names for those items.

Finally, if you like this kind of detail there are two new advanced settings: QRMonitorOpaqueFolderProgress and QRMonitorInvisibleFolderProgress. Set them to true to enable the progress reporting of those items (although this doesn't change the display metadata collection).

Bruce Giles wrote:1) The indeterminate progress bar in the Activity window is still nearly impossible to tell from a 100% full progress bar. I realize that's an OS issue, not really a QRecall issue, but Apple has apparently decided this isn't enough of a problem for them to do anything about it.

While I can't do anything about Apple's control design, you'll notice that the collapsed version of action (click to toggle between the two) has a new-and-improved background progress indicator of my own design, which I hope clearly distinguishes between these two states.

2) I'm doing a new backup, to a new archive, of my VMWare Fusion virtual machine file. It's been running for about 42 minutes so far, and backed up about 130 GB, and for the entire time (except for perhaps the first few seconds), the Activity window says "Capturing screenshot_0000.png", which is only a 1.9 MB file. I'm sure it's backed up way more than that, but it just seems a little odd. Is there anything you can do in the new version of QRecall to help with that?

I've looked that this code a lot and don't see why it should be doing that. Having said that, I've recently made performance improvements in the code that updates the activity monitor with the status of the running action; one side effect is, I hope, more timely updating of the actual status. See if the next (post 2.1.0b32) version improves anything.

Ming-Li Wang wrote:Very happy to see v2.1 in beta. I'm you'll beta 31 on macos 10.13.4.

Welcome aboard!

First bug to report: QRecall sometimes has troubles deleting items from an archive, taking a long time waiting for the archive to close. (screenshot 1)

Actually, it's waiting to open the archive. QRecall thinks that some other process might be using the archive and is waiting for it to stop. Sometimes, particularly if a process is on another computer, it can be hard to tell and QRecall will cautiously wait about 10 minutes before proceeding. If everything is working perfectly, this shouldn't happen as the last process should mark the archive as "not busy." Hopefully the report will have some clues.

The dialog would say "complete" if I wait long enough, but it won't go away, and the "Stop" button would be grayed out and unclickable. (screenshot 2)

This is a known bug that I'm still investigating. Basically, the QRecall application loses its connection with the command process at the last moment and gets stuck in a dialog because (a) it can't tell the command to stop and (b) it hasn't gotten a message from the command that says it's finished. Again, your report (and reports from others) should help figure this out.

I've run into the problem numerous times with two different archives, so it doesn't seem to be archive-specific. I've deleted items from both archives successfully as well, so it seems random to me so far.

You are correct, it is not archive specific, or even command specific; this can happen with any command.

A report has been sent.

Thanks.

Ralph,

I'll have to spend some time researching this.

I suspect the iCloud folder is actually a mountpoint, or a collection of mountpoints, for a virtual filesystem who's content resides on different logical device, but it will take a bit of investigation to confirm this.

Update July 23, 2018: The QRecall 2.1 beta is closed. If you are running the beta, use QRecall > Check for Update… to upgrade to the release version. Everyone else can download QRecall 2.1 on the Download page.

*** Thanks to everyone who participated ***

Dawn to Dusk Software is pleased to announce the beginning of the QRecall 2.1 beta test program.

To get started, go to the QRecall Download page.

If you already have a permanent or trial identity key, you can continue to use that. If don't have a permanent key, or your trial key has expired, click the button on the download page to obtain a free Beta Identity Key that will be valid for the entire beta test period.

The theme of QRecall 2.1 is "new technology." The user interface has been redesigned with a modern look, feel, and features (like sidebars, Quick Look, and full-screen mode). Underneath the hood, a massive rewrite of the code adopts modern technologies like automatic memory management, XPC interprocess communications, modern compilers, and much more.

As always, feedback is welcome and encouraged.

David Ramsey wrote:Deleting the files has cured the "runs multiple times per day at odd times" problem.

However, I think I am now noticing another problem.

Now, it seems that every backup takes a LONG time. For example, today's backup has been running for 39 minutes as I type this and has copied a mere 1.28GB of data (didn't do much today).

Performance problems.

During a QRecall capture there are about a thousand different things going on, which means there's a million ways it could be getting slowed down.

While the performance of the source volume could be issue, it's the least likely. One issue could be that QRecall can't use the volume change history for some reason, which will force it to perform an exhaustive scan of the directory structure. But for the most part it just reads directories and files, and unless there's something wrong with the volume (and it doesn't sound like it), this shouldn't be an issue.

It's much more likely that the size of the archive or the performance of the volume it is on will have more of an impact. Due to the nature of QRecall's data de-duplication logic, the amount of work required to add new data to an archive increases exponentially as the size of the archive grows. So if it takes 1 second to add X amount of data to a 100GB archive, it might take 8 seconds to insert that same amount of data into a 400GB archive.

Other factors include the archive volume's speed, bus latency, how full the volume is, and how fragmented the volume has become.

Now, that's not to say nothing is wrong, but it's really hard to tell from this distance given the number of things that could be affecting performance. If you like, send a series of diagnostic reports while one of these long-running captures are executing. The Send Report command samples all running QRecall processes and include that in the uploaded report. That could provide some clues as to what's eating up its time.

David Ramsey wrote:Can I delete these?

Yes. There's ever only one active lock file. The rest are orphans.

David,

Well this is mighty peculiar. You're running a regular scheduler, so when you log out and back in again, the schedule gets shut down and restarted.

When it starts up again it thinks the previous actions didn't run on time and starts them again, even though the log clearly shows that the scheduler started the action and saw that it finished.

It's possible the scheduler is having problems remembering what has run and what hasn't, so let's try this: in your ~/Library/Preferences folder, locate the com.qrecall.scheduler.plist file and trash it (along with any semaphore files that start with com.qrecall.scheduler.plist.). Restart your system and see what happens.

Curious indeed!

I could ask you a million questions, but I think it would be simpler if you would send a diagnostic report (Help > Send Report) from that computer. That will allow me to investigate/eliminate a whole bunch of possibilities.

Steven J Gold wrote:Curious minds want to know...

This is why I love my customers.

I have a 5 GB disk image file I'm backing up.
Actually, it's a "Sparse Disk Image Bundle" consisting of 560+ "bands" each of which is an 8.4 MB file.

I notice when I write to/modify the mounted disk image, the Date Modified for some -- but not all -- of the individual band files change.

Correct. A disk image is a file that pretends the be the blocks of a volume. If the package is broken up into individual files, only the files that correspond to blocks you changed on the "volume" get modified.

When backing up, does QRecall treat each of the bands as individual files even though at the higher level the OS (like Finder) treats the conglomeration as a single file?

Technically, it depends on how you backup up the disk image. If you capture the disk image file/package then QRecall sees them as regular files. If you capture the volume of the mounted disk image, QRecall capture those individual files as it would the files on any volume.

But it doesn't matter (at least to the size of the archive).

QRecall performs block-level de-duplication of the data, so it actually doesn't matter if the disk image is a single massive file, a collection of "bands," or the individual files stored on that disk image. Every file will be broken down into its individual data blocks and analyzed one at a time.

If I write to the mounted disk image resulting in say 100 out of 560 of the band files being changed, will QRecall only recapture the changed band files as opposed to the entire bundle?

Also correct. The advantage of using a disk image with individual bands is that QRecall can skip the analysis of the bands it knows have not changed. But in the end, it doesn't matter how many "bands" you have or which ones were modified, only the modified data blocks get captured.

You/I/we probably don't need that log file, as you seem to have a handle on the issue.

Feel free to toss your QRecall.log file(s) and restart; a new will be automatically created.

Ralph Strauch wrote:Lastly, as I'm writing this, the MBP just popped up another "Drive not Ejected Properly" message, while the drive is still sitting there, connected but not being accessed afaik.

I suspect this is happening more that you realize. The pattern I typically see is an intermittent hardware problem (external bus, cabling, power supply, drive controller, and sometime software) that causes a drive to spontaneously disconnect. It will then, almost immediately, reconnect so by the time you turn around it's already reappeared on the desktop.

This can occur repeatedly, without notice or harm. But when it occurs during a long-running process (like a QRecall capture or repair), then it can cause problems.

I looked at your logs. The QRecallScheduler process monitor and logs mount events. So I can tell that on your iMac, for example, your external volume BUD3 has unmounted and then immediately remounted itself over 60 times in the past month. Here are some recent examples:

2017-12-30 13:32:20.434 -0800 unmounted volume /Volumes/BUD3
2017-12-30 13:32:50.260 -0800 mounted volume /Volumes/BUD3

2017-12-30 13:32:50.983 -0800 unmounted volume /Volumes/BUD3
2017-12-30 13:32:54.054 -0800 mounted volume /Volumes/BUD3

2017-12-30 21:13:12.937 -0800 unmounted volume /Volumes/BUD3
2017-12-30 21:13:42.762 -0800 mounted volume /Volumes/BUD3
2017-12-30 21:13:43.982 -0800 unmounted volume /Volumes/BUD3
2017-12-30 21:13:47.059 -0800 mounted volume /Volumes/BUD3

2017-12-30 21:15:31.133 -0800 unmounted volume /Volumes/BUD3
2017-12-30 21:16:00.896 -0800 mounted volume /Volumes/BUD3
2017-12-30 21:16:02.297 -0800 unmounted volume /Volumes/BUD3
2017-12-30 21:16:05.435 -0800 mounted volume /Volumes/BUD3

2018-01-02 15:25:25.290 -0800 unmounted volume /Volumes/BUD3
2018-01-02 15:25:54.687 -0800 mounted volume /Volumes/BUD3
2018-01-02 15:25:55.759 -0800 unmounted volume /Volumes/BUD3
2018-01-02 15:25:58.805 -0800 mounted volume /Volumes/BUD3

2018-01-06 08:49:29.136 -0800 unmounted volume /Volumes/BUD3
2018-01-06 08:49:58.800 -0800 mounted volume /Volumes/BUD3
2018-01-06 08:50:00.213 -0800 unmounted volume /Volumes/BUD3
2018-01-06 08:50:03.285 -0800 mounted volume /Volumes/BUD3

2018-01-06 12:23:50.293 -0800 unmounted volume /Volumes/BUD3
2018-01-06 12:24:19.997 -0800 mounted volume /Volumes/BUD3
2018-01-06 12:24:21.318 -0800 unmounted volume /Volumes/BUD3
2018-01-06 12:24:24.404 -0800 mounted volume /Volumes/BUD3

As you can see, the volume occasionally unmounts, and then a few seconds later is mounted again. In most cases it does this again within a few seconds.

Honestly, the cause of this kind of thing can be difficult to track down, but swapping out components (try a different cable, swap power supplies, switch from eSATA to USB or vice versa) might narrow it down.

Ralph,

Here's what I found going through your logs.

First, I apologize for the "lost communications" errors you're getting. These are random interprocess communications failures caused by the legacy Mach Ports service. The only way to fix this is to completely rewrite QRecall to use the modern XPC communications. This work has already been done, but won't appear in QRecall until the next major version.

QRecall seems to be working as expected, although I can see why you might think there are problems.

The first set of actions are exactly as you described. You created a new archive (7th bu), started a capture, stopped it, started it again and let it finish.

Except for two issues, this all ran flawlessly. The first issue is that at some point (probably when uninstalling and reinstalling) QRecall is no longer pre-authorized. All of the capture actions are running with user-only privileges. This resulted in vast swaths of system files not being captured and reporting "Could not capture file/Insufficient privileges to read file/Permission denied" errors. It also explains the "Capture encountered problems" warning you see at the end.

The second issue is, of course, the lost communications link between the helper process performing the capture the the monitor app displaying its progress. My suggestion, for QRecall 2.0 and any macOS after 10.10, is to take any loss of communications error with some healthy skepticism. Instead of assuming the process has crashed (the original intent of that error), check Activity Monitor and see if the QRecallHelper process is still running. If it is, ignore the error and check back later to see what the log says. If the process completed successfully, it will log some king of completion message (hopefully "Capture finished"). That's the definitive answer to whether the action was a success or a failure.

2018-01-01 12:18:24.558 -0800 Capture finished (14:19:55)

You then restarted your computer and repaired the archive. Because there was nothing wrong with the archive, the repair completed successfully and reported no issues:

2018-01-02 14:43:12.323 -0800 Repair finished (2:31:33)

You then ran another capture followed by another verify:

2018-01-02 16:37:01.695 -0800 Capture finished (38:32)

2018-01-03 10:17:55.594 -0800 Verify finished (2:33:33)

Again, except for not having permissions to capture any system files and some communication failures, both actions ran successfully and to completion.

At this point I would suggest you re-pre-authorize QRecall to use administrative privileges. Go to QRecall > Preferences&hellips; > Authorization and turn on "Capture and recall items using administrative privileges". Then recapture your hard drive (this will take a bit since there is now a whole bunch of new stuff to capture).

Finally, I don't know what's causing the QRecall app to freeze on you. I don't see any '.hang' reports auto-collected by the system. If it happens again, launch Activity Monitor, select the QRecall app (which should say "not responding"), choose View > Sample Process (Option+Command+S), and send me the resulting report.