For two straight weeks my system partition archive needed repair when verified. In both times disk verification showed no error and the repair was successful, but 6 out of the last 7 daily layers are now "incomplete", while 4 from the previous week were incomplete. (Note that I removed those incomplete layers and compacted the archive after the first repair.)

As can be seen from the screenshot below, the repair log indicates there are 4 "unreachable" folders under my system partition "carlosys", including one under my user profile directory tree. The 4 folder serial numbers are the same for each day. Unfortunately I can't tell which folders they are referring to so I can't investigate further.

I'm still on Yosemite (10.10.5), if that matters.

edit: forgot to say, a diag report was sent.

Yeah, I saw it in the release notes. Thank you!

It seems v2 is almost ready. I have had no bug to report for weeks.

It's another Sunday and I'm happy to report that the issue has indeed been fixed in b14. All merge and verification jobs reserved for Sat. and Sun. morning have finished successfully, and b14 has been running perfectly well here. Thank you.

Johannes wrote:A solution for both issues would be simple filters on file/folder names (exclude/include *.dtBase2, peaks/*, analysis/* export/* ect).

Your wish is mine, too, as I made a similar request just a few days ago.

Any other suggestions?

Before that happens, QR2's command line tool "qrecall" might provide a workaround. The following line should work to exclude all "peaks/*" under the /project_folder tree from QR2 backup:

find /project_folder -type d -name "peaks" -print0 | xargs -0 qrecall captureprefs exclude

Similarly, the following line should get "*.dtBase2" excluded:

find /project_folder -type f -name "*.dtBase2" -print0 | xargs -0 qrecall captureprefs exclude

Put them in a bash script and run them as often as necessary. You can use Hazel, Keyboard Maestro or the like to automate the task. Even the good old crontab can do the job.

Having seen no error/warning messages in the log window, I'd been very happy with b13 until I noticed several yellow status icons in the status window. Several archives have not been verified for almost two weeks. That should not be possible since all my archives are scheduled to be verified at least once a week.

As there's still no warnings or errors in the log window, I dialed the detail level to the max and saw numerous question marks associated with a "Problem communicating with helper, retrying" message. Given your detailed explanation earlier these don't concern me.

After digging into the various records associated with those verification actions scheduled for early Sat. and Sun. morning, however, I found several of them--the last one on Sat. morning, at around 4:40am, and all 3 this morning--did not have a "Verify finished" line. And sure enough, those are exactly the 4 archives listed as "Verified 13 days ago" in the status window.

Note that all weekend verification jobs are preceded by a merge action. They all finished without issue according to the log. Manual verifications on all 4 finished successfully as well.

One side issue: my investigation was prompted by the "red" dot inside QR's menubar icon. As it turned out, the non-verification status for those 4 archives merely earned them a "yellow" dot. The lone "red" dot in the status window is for my music archive that has not been captured for 11 days. It's nevertheless a non-issue as I haven't got the time to rip my CDs for quite a while. (The capture done 11 days ago was for meaningless changes in .DS_Store files.)

Now, I don't want to disable status warnings in the menubar icon; nor do I want to "forget" the archive, but the red dot bugs me, and might numb me someday. So please consider allowing some customization there in the future. Thanks!

James Bucanek wrote:

Bruce Giles wrote:

Feature request: Two-column sorts of the Actions window. I'd really like to see actions sorted first by the Archive column, and then by either the Next or the Schedule column.

I'll add that to the list. Most of the UI enhancements are being addressed in version 2.1, so I'll probably get to it then.

I thought one could get the same result by simply clicking on the Next or Schedule column first before clicking on the Archive column. No? Not that I'm against you doing anything about it. Just saying perhaps Bruce could achieve (most of) what he desires already before v.2.1 arrives.

In the release notes, "scraping" should be spelled as "scrapping", with two "p"s. Mentioned so you can get it right in the help files.

I bow to your superior skills in proofreading, but I have to call you out on this one. It's "scraping" (one 'p'), the noun form of "scrape": "to push or pull a hard or sharp implement across a surface or object so as to remove dirt or other matter..."

I thought as much, but didn't speak because English is not my first language.

The cause for confusion probably lies in the beta release notes, however. In the section titled "Excluding, Ignoring, and Scraping Items", "scrap"--not "scrape"--is used numerous times as a verb.

b13 has been working for 24 hours and there's not a single error message or warning in the log. Lucky 13 indeed. Well done. Thank you!

I've made substantial rearrangement to my action schedule, with all except the ever-changing system folders on file change event-based schedules. It's a great feature for v2. Thank you very much. Two small issues arise out of this change, however.

1. Those event-driven actions take place more often than I expected due to meaningless (non)change to .DS_Store files. There are now numerous layers in those archives that contains nothing but one or two .DS_Store files. I know they take up very little space in the archives, and the layers will evently merge with those with substantive changes. And yet I still hope I can exclude all the .DS_Store files from most archives.

Currently there's no way to specify a .DS_Store file in archive settings UI because you can't select a hidden file. (The trick of using SHFT_CMD_G to go into hidden folders don't work here.) The "qrecall captureprefs" command works, but I wonder if the extended attribute QR uses would stick when OS X makes changes to .DS_Store files (I guess it depends on whether the system writes changes directly into existing files or creates a new one and then removes the old one).

More importantly, new .DS_Store are created all the time, and it's not practical to exclude them one by one. Hence my request: could you please allow more flexibility in this area? I would like to exclude, e.g., all .DS_Store files, all *.bak files, all *.*-journal files, and all "minidump" folders, just to name a few.

2. The current explanation for the option "Ignore new events for ___ (min./hours/days/weeks)" reads:

Ignore New Events

The ignore option will ignore new events that occur after the event that ran the action. To prevent new events from running the action again too quickly, choose the Ignore new events option and set a time period. The next event that occurs after the ignore period will run the action again.

I had taken it to mean--until I experimented with the option--file changes taking place in the "timeout" period would be backed up only when a new file change event after the timeout triggered the capture action. Should no new changes be made to the watched folders for a long time after the timeout, as a result, changes during the timeout would be without a backup for a long time.

In fact, QR behaves as I expect/want it to behave: all changes taking place during the timeout period are backed up as soon as the timeout period expires.

If my understanding is correct, I would suggest changing the option to something like "minimum time between actions" or "minimum time before the next action".

Thanks for listening.

Thanks! Can't wait to test the lucky 13.

B12 went through the grueling early Sat. morning schedule (moved to this morning as usual) for the first time here and survived. It's still working fine, and doesn't give me spinning rainbow wheels. Found some peculiar entries in the Log Window, however.

1. Five consecutive "could not communicate with helper" entries at address 8:46am, right about when I touched my computer for the first time in the morning. There were 6 of them yesterday morning (I forgot to add Tuesday morning to the group of "Sat. morning actions", so I didn't report back yesterday morning), also at about the time I touched my computer for the first time.

Note that the system sleep timer is set to "never", while HDs and the displays are allowed to sleep.

2. There are 2 errors associated with the actions that took place overnight. The first at 4:49:55am, "Helper did not start" after a successful "merge" action. Then "helped did not start" again at 5:34:55am, which stopped a "verify" action. Both errors gave the same description (with different timestamps, of course):

Distributed objects message send timed out (timeout: 462228595.910597 at time: 462228595.922145) 1

A diag report has been sent.

Many thanks for the detailed explanation, as always.

James Bucanek wrote:You might consider changing the schedule of your actions that capture a specific subset of files (like documents, photos, or music) to use the new "Capture Items Changed" event schedule.

This is something I've meant to try since v2 beta testing, but never got around to really set it up. Good. Will change the every-15-min. action (there's only one) right away.

One question: how reliable is OSX/HFS's file change mechanism? Do I need to compliment the action with, e.g., daily capture?

James Bucanek wrote:

1. A capture action is scheduled for my system partition at 3:20am daily. The archive's layer map says it's done this morning, with 793 captured items totaling 273.7 MB. I can't find any entry for it in the QR Log (see attached screenshot), even with maximum detail level.

If you send another diagnostic report I can review the log.

It happened in the early morning on 8/19, Taiwan time, so it's already in the diag. report I gave you that day.

B12 was installed yesterday afternoon, but it didn't work. Not a single action was executed until I rebooted the system this morning.

As I'll be out of town for a few days, I won't be able to tell if B12 can survive the early morning schedule without issue until next Tuesday.

Since the problem doesn't seem to be related to competing use of system resources (my own tentative conclusion while you investigate), I decided to take a closer look at the logs and see if I can spot anything that escaped my attention.

1. A capture action is scheduled for my system partition at 3:20am daily. The archive's layer map says it's done this morning, with 793 captured items totaling 273.7 MB. I can't find any entry for it in the QR Log (see attached screenshot), even with maximum detail level.

There are several QR related entries in the system.log at the same time, however:

Aug 19 03:20:00 carlo com.apple.xpc.launchd[1] (com.qrecall.hoist[505]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): QRHoist

Aug 19 03:20:16 carlo QRecallHelper[506]: Failed to initialize IconCache named: com.apple.iconservices with error: Error Domain=NSCocoaErrorDomain Code=4097 "Couldn?t communicate with a helper application." (connection to service named com.apple.iconservices) UserInfo=0x7ff2b941f840 {NSDebugDescription=connection to service named com.apple.iconservices}

Aug 19 03:20:16 carlo QRecallHelper[506]: Error returned from iconservicesagent: Error Domain=NSCocoaErrorDomain Code=4097 "Couldn?t communicate with a helper application." (connection to service named com.apple.iconservices) UserInfo=0x7ff2be2bf6f0 {NSDebugDescription=connection to service named com.apple.iconservices}

... (the previous one repeated for 12 times)

Aug 19 03:20:16 carlo QRecallHelper[506]: ImageIO: CGImageDestinationFinalize image destination must have at least one image

... (Another batch of the previous two, including the repetition.)

2. The capture actions scheduled for 3:50am was indeed run and concluded successfully, with nothing to capture. Don't know why QR executed it again after reboot this morning (as reported in my previous post).

3. The first in a string of "Unable to communicate with helper" warnings (at 4:20:25 am) in the QR Log has a note that says:

[NOTE: this exception originated in the server.] | connection timeout: did not receive reply

The others says:

cannot connect with helper
port: QRecallHelper.2307054f

There is no QR-related entry in the system.log at this time.

4. QR scheduler managed to get itself up and executed a capture job at 6:34:21am.

Before going to bed last night, I removed Crashplan's plist from /Library/LaunchDaemons, Foxtrot (a fulltext search engin) indexer plist from ~/Library/LaunchAgents, as well as all Login Items. The system was then rebooted into a state with none of my usual background unities (except QRecall) running. I also stopped Spotlight indexing for all drives. In short, the system was in a state as idle as it could be.

All QR actions scheduled for early Sat. morning were again moved to this morning, and a command line window was open, running the "top" command as last night. That's it.

Still got the rainbow wheel again when trying to click QR Monitor's icon in the menubar this morning. The system is otherwise normal, meaning I could launch Firefox and other applications without delay. I launched QR from the Launchpad instead, but again clicking anything on the menu gave me the rainbow wheel. Be patient, and the menu did pop up after a few sec., and I was able to get the Log window open, and send the diag. report. The toplog was also sent, but I believe you'll see a system mostly idle throughout the night. The OSX Activity Monitor again showed QR Scheduler is alive, but it's apparently not working.

The Log again shows QR finished a "Capture" job without issue at 3:35am, but was "Unable to communicate with helper" at 4:20am. (No log entry in-between.)

I rebooted my system again after putting everything back to normal (Crashplan and all login items), and QR has been busy catching up with the unfinished jobs last night as I write. The strange thing is, the first job QR did was a "Capture" job scheduled at 3:50am, not the "Verify" job at 4:20am. Didn't notice this the previous days.

Another thing: QR has just finished all overdue actions, and the last one--a "Verify" job--found the archive corrupt. The archive is for my system partition, and it's captured daily. All recent captures came through without issue, including the one at 3:20 this morning. Yesterday's verification was canceled, according to the Log, and the one done a day earlier was successful. I'll try to repair it later.

That's it for today's morning report.

Adrian Chapman wrote:I'm also seeing a lot of "unable to communicate" messages, but I am not aware of any slowdowns. The messages occur as a string over very short intervals, maybe a second or two at a time.

I have those as well, though the "short intervals" last a little longer here, from less than 10 sec. to about half a min. at a time. But they seem harmless, for normal QR activities resumed afterward.

What I'm reporting here are not the same. QR is no longer working each morning.