QRecallDownloadIdentity KeysForumsSupport
  [Search] Search   [Recent Topics] Recent Topics   [Hottest Topics] Hottest Topics   [Groups] Back to home page 
Messages posted by: James Bucanek
Forum Index » Profile for James Bucanek » Messages posted by James Bucanek
Author Message
Jon,

Got your diagnostic report. And yes, QRecall cannot start its privileged helper. You also have a lot of "not found" errors on your archives, but we'll need to straighten out the helper first.

This problem is always an installation problem. To gain elevated privileges, QRecall must request that macOS install a "privileged helper" on its behalf. Sometimes (and I'm not always sure why) this doesn't happen or it gets messed up. Once messed up, a new helper can't be installed and QRecall can't un-install itself (because un-installing requires the privileged helper). A true catch-22.

The fix is simply to manually uninstall QRecall. This takes a little bit of work, but I'll give you a shortcut.

The steps are in the help under QRecall > Help > Guide > Advanced > Uninstall > Uninstall QRecall (the hard way). There's also a copy on the web.

As a shortcut, start by performing just steps 3 and 4. (I suspect this is the root of the problem.) After restarting, launch QRecall. It should prompt for you admin credentials and reinstall QRecall. If it doesn't prompt you, make sure QRecall > Preferences > Authorization > "Capture and recall using administrative privileges" setting is turned on.

If that doesn't solve the problem, perform the entire un-install procedure (make sure you skip the † "dagger" steps and step 9, as you don't want to start over when you re-install). Restart again and then launch QRecall.

If this still doesn't solve the "can't launch privileged helper" problem, please send another diagnostic report (QRecall > Help > Send Report) and we'll dig deeper.

What's most likely is that the B1 volume got reformatted/resized/whatever and appears to QRecall as two different volumes. You can confirm this by looking at the timeline (or info sidebar) and see that all of the captures of the first B1 occur before the second B1.

If you want to keep all of this history, simply select both B1 icons and choose Archive > Combine Items. This command takes two volumes, or two owners, that are actually the same volume/owner and stitches them together to form a single item with one time lime.

Alternatively, if you're not interested in that much history, find the older B1 (again, use the info sidebar) and delete it (Archive > Delete Item).
Working backwards...

Yes, redundancy increases the size of the archive. If you selected the default redundancy level (1:8) then every eight K of data is accompanied by an additional one K of redundant information, making everything you store 12.5% bigger.

There are also meta data, index files, and other database related information in the archive which makes it bigger than the actual file data. But that typically doesn't account for more than a couple of percentage points.

The "insufficient free space" message is the compact action trying not to waste time and endanger your data by moving it around pointlessly. The compact action won't physically compact the archive unless at least 12% of the archive's database file is empty space. This prevents the compact action from moving hundreds of Gigabytes of data, only to recover 4K. You can adjust this threshold in the advanced settings ("Compact Free Space Ratio Minimum"), or you can run the compact from the QRecall menu: open the archive in the browser, then choose Archive > Compact. When run from the menu, it ignores the free space ratio.

But while you have the archive open, make sure you navigate to the root of the archive to make sure you don't have any other owners or volumes that might contain older versions of data. If you do, consider deleting the older owners/volumes (before compacting).

Finally, if you want the smallest possible archive size for files you're keeping for posterity, consider turning on compression in the archive's settings. If you up the compact compression level, then compact the archive, the entire archive will be compressed. (Note it only does this once, and once compressed it won't uncompress or recompress the data). If you still need even more space back, consider reducing the redundancy level.

I hope that helps!
It's hard to say exactly what was going on, but I wouldn't say this behavior is surprising.

When a volume fills up, it becomes harder and harder for the filesystem to locate free clusters needed while writing, and files become increasingly fragmented. If the volume is really really full, this slowdown can be excruciating; like 1,000 or more times slower than normal. In other words, an operation that would normally have finished in a 1/10 of a second could take a minute or more.

The first compact after a capture performs garbage collection, which results in rewriting several quanta index files. If this was happening on a nearly full drive, I can imagine abysmal performance.

Your clean up of 300 MB was probably enough to save the volume from fragmentation purgatory, and as soon as the compact was stopped it would have deleted its own temporary index files, freeing more space, and probably restoring much of the filesystem's typical performance.
Looking at the code, the layer size should read "Unknown" for a repaired layer. Not sure why you're seeing Zero KB, but I'll dig a little deeper into it. Probably a bug.
Following a repair, QRecall makes the following guarantees:

  • All items (unless you set some of the special repair options) in a layer have been verified to be undamaged and can be recalled.

  • Layers that are not marked as "repaired" are complete and aren't missing any items

  • The inverse implication is that layered marked as "repaired" might be missing items—there's no way for QRecall to tell—but the items that are there are OK.

    Now to cover some of the minutia of the repair process...

    A "Zero KB" just mean that QRecall couldn't (easily) reconstruct the size of that layer. Like the Finder, QRecall would have to scan every folder and item in a layer to fully reconstruct its size, and repair won't do this more than one folder level deep. Maybe that's a bug. Maybe it's just "hey, the layer was damaged; there's stuff about it we don't know anymore, sorry." So those layers aren't empty and you should find items in them if you browse them on their own (move both layer shades so only that layer is visible).

    About the dates and layer order. If layer records are missing, the repair can reconstruct most of the information about that layer from the individual folder and file records. Every folder record has a "layer ID", so by scanning the folder records it's possible to put them back into their original position in the layers. But if the layer records were damaged or missing, some information (like that date that layer was captured/merged), has to be inferred from the dates of the individual folders. It should to be close, but it won't be exact.

    Finally, if you merge a repaired layer with a subsequent layer and that subsequent layer completely captured all the same items, that repaired layer will disappear. If not, then the merged layer will still have remnants of the repaired layer and will also be marked as "repaired".

    Let me know if that helps, or confuses the matter even more.


    Nico Dudek wrote:Hi James, First I hope you are healthy and well!

    Everyone here is well, thank you!

    After copying the archive I was asked to reindex it. Is this normal and worth to do? Or would you recommend a new archive instead of moving the old one?

    That is not normal. Simply copying an archive should not corrupt it. However, if some other action were to modify the archive while it was being copied, then that will mess up the copy. Before copying/moving an archive, I suggest you hold all scheduled actions to make sure nothing starts during the copy.

    You can try a reindex. A reindex simply rebuilds the various index files QRecall needs to access the archive data more efficiently, but it does not in any way try to repair that data. So if a reindex is successful, your archive is intact and undamaged. If the reindex fails then the archive data was corrupted during the copy. If you don't still have the original, it's easiest to just start over. If you do have the original, verify it and then try copying it again (making sure no other actions are running).

    As a rule, you can freely relocate and rename archives as you please. After moving it, open up an action for that archive, click on the archive action button, pick "Choose Archive", and select the relocated archive. That action now refers to the archive at its new location. Save and close that action. If you have other actions, QRecall will offer to relocate those for you.

    Let us know if you have any other questions.
    Prey Lawn wrote:From where can I purchase a key?

    Identity keys are available on the QRecall website. See the "One Key" section to purchase a key.
    Without seeing the errors that you're getting, I'm going to take a wild guess and suggest this is probably a keychain problem.

    QRecall will attempt to mount networked volumes before it starts an action.

    Most networked volume require authentication (typically an account name and password) to connect. To mount them automatically, that authentication information can be stored on your keychain.

    Therefore, the requirements of automatically mounting a volume require that your keychain be open, which implies that your user is logged in and active. If you've logged out or you have security set up so your keychain gets closed after a period of inactivity, QRecall can't access the credentials necessary to mount the volume.

    If that's the problem, the simple solutions are (a) change your security settings so your keychain doesn't get locked automatically or (b) pre-mount the networked volume so QRecall doesn't need to mount it when the action starts.

    Let us know if that helps.
    Norbert Karls wrote:Hi again,

    Hello!

    QRecallHelper will not exceed its allowance of »wired« or »reserved« memory, but it will take on more and more »virtual« memory over time

    Note that QRecall uses a lot of "mapped" files. So most of its virtual memory is really "virtual." In other words, it's just reserving address space; it doesn't mean any actual memory is involved (although some will, eventually) and it is separate from the VM backing store.

    For example, when a process "maps" a 500MB file to memory what happens is the system simply reserves a span of logical memory addresses (say, 0x0006000c6a0000000 through 0x0006000c6bfffffff) and sets up the contents of that file to correspond to those addresses. The "virtual" memory of the process immediately increases by 0.5 GB, but nothing has really happened; no actually memory has been allocated or used at this point. The real measure of a process' impact is how much of the VM store and heap space it has consumed, which you can't tell from a casual look at its virtual memory footprint. That's why the latest macOS and Activity Monitor app has a "memory pressure" metric that takes these kinds of things into account.

    The logs don't mention any reason for the process ending, there's just the same chatter on and on for hours and then it just stops. A caught error leading to termination would be logged and could then be found near the end of the logfile; this is not the case, though.

    That would indicate that the process is crashing for some reason, and it appears to be repeatable. That information would be in a system crash log. If you send a diagnostic report from QRecall (Help > Send Report...) those crash logs will be sent to use and could indicate what's failing.

    I then put my hopes into trying an incremental approach

    The repair (and index) is really an "all or nothing" proposition. It has to finish, or it has to start over.

    Finally I looked around the various options to the commands, hoping to find a way to have it just extract individual files or layers or some way to force reindexing ignoring errors to create an index that would allow me to get something instead of nothing, but without any luck.

    That's literally what the repair command does, so we need to figure out why it's crashing. (Note that a program can ignore errors, it can't ignore a crash.)

    Here's a thing that came to my mind: A repair action will first dig through the entirety of an archive analyzing it before then starting to take action (with the exception of a reindex, but it refuses that here). Couldn't we have some kind of scavenge mode, especially with the --recover option, that will start writing a new archive immediately, adding files and layers at the pace they are discovered in the source archive?

    That's pretty much what it's already doing .. the problem is the crash. However, it did occur to me that some aspects of the repair, like dealing with damaged records, could be made more incremental and I'll add that to the wish list.

    Do stay healthy

    We're in this together!
    Steven,

    I'm afraid you'll have to ignore this one.

    The com.apple.TCC file contains sensitive information that Apple doesn't want malware to have access to. QRecall works around most file restrictions with special privileges, but the com.apple.TCC directory seems to be different. Also note that it's very inconsistent; some users report problems reading this file, while most systems capture it every day just fine.

    The folder contains a database that stores, among other things, the security preferences and allowances defined for various apps. Obviously, macOS is super paranoid about letting a bad actor modify (or even read) this information.

    The dvp database is similarly protected.

    I would suggest adding an exclusion to your archive that skips these items. Should you need to perform a complete restore of your start-up volume, it just means you'll need to re-grant apps access to your private information (contacts, external volumes, etc). Which is probably a good thing to review from time to time anyway.

    That is correct. When you change identity keys, you are a new "owner" so all of your items will be captured for the first time.

    Of course, since the archive you're using has already captured these items, most (if not all) of the data is duplicate. What's taking time is reading each block of every file, looking that up that block in the archive's database, and comparing it to make sure it's a duplicate. That's a lot of work.

    If you're not concerned with the (short) history of changes you already have, then it won't hurt to simply wipe the archive and start over: choose File > New Archive and overwrite your existing archive with an empty one. Then restart the capture.

    If you do want to keep your history, let the capture finish (or stop the one in progress). Open the archive, navigate to the root, choose both owners, and choose Archive > Combine Items.... This will change the owner of all of your previously captured items so they now belong to your new owner (identity). From there on out, it will be as if you had started with your permanent identity key from the beginning.

    I hope that helps.
    That's the "next action" from the scheduler. You could try restarting the scheduler, but it should (eventually) just go away.
    Is it an action that's running? (Or one that has stopped with an error?) The activity window only shows the progress of running actions, the results of failed ones, and the next scheduled action. Deleting an action document, archive, etc. won't affect the display of a process that's already started/finished.

    If you think it's cosmetic issue, you can try restarting the monitor (Quit/SIGTERM the QRecall Monitor process).
    maxbraketorque wrote:For the second question, there are no orphaned actions listed in the Actions window.

    Oh, I think you might be referring to the Status window, not the actions or activity window.

    The Status window shows the state of all known, and previously known, archives. To make QRecall forget about the status of an archive, click on the action (gear) button, or Right/Control-click anywhere in that pane, then choose Forget.
     
    Forum Index » Profile for James Bucanek » Messages posted by James Bucanek
    Go to:   
    Powered by JForum 2.1.8 © JForum Team