Fix deadlock in beginJob when compiled debug

[Copilot]

Debug-compiled EventDisplay deadlocks in beginJob() at cv_.wait(lock). GDB shows only 2 threads with the app thread apparently never reaching its event loop.

Root cause: ABBA lock ordering between m_ (held by beginJob) and gSystemMutex (acquired inside XThreadTimer constructor via R__LOCKGUARD2). The app thread holds gSystemMutex in gSystem->ProcessEvents() while timer callbacks attempt to acquire m_. Debug timing makes this race deterministic. Additionally, bare cv_.wait() without a predicate loses notifications if the signal arrives before the wait begins.

Changes:

• Restructured beginJob() to only hold m_ during cv_.wait(), not during XThreadTimer construction — eliminates lock ordering cycle
• Added appStarted_ and eveSetupDone_ predicate flags for the two synchronization points
• Switched to cv_.wait(lock, predicate) — handles both missed notifications and spurious wakeups
• signalAppStart() and setup_eve() set their respective flags under lock before notifying

// Before: holds m_ across timer creation → lock ordering deadlock
std::unique_lock lock{m_};
appThread_ = std::thread{[this] { run_application(); }};
XThreadTimer sut([this]{ signalAppStart(); });  // acquires gSystemMutex
cv_.wait(lock);

// After: m_ only held for the wait
appThread_ = std::thread{[this] { run_application(); }};
XThreadTimer sut([this]{ signalAppStart(); });
{
    std::unique_lock lock{m_};
    cv_.wait(lock, [this]{ return appStarted_; });
}

[brownd1978]

This PR does not change the behavior: Mu2eEventDisplay complied debug still hangs in beginJob at line 273.

[sophiemiddleton]

thanks Dave, I will try to investigate. I've never seen an issue, but I've also never compiled in debug mode.

The multi-threading aspect of the display does cause some difficulties, and I've tried debugging with AI in recent weeks, with little insight appart from confirmation of what I already knew.

This specific issue may be fixable though, I will take a look

[brownd1978]

Hi Sophie, thanks for taking a look. There are other problems associated with the locks: currently when I run the display from LBL it segvs when I hit 'next event', also traced back to a 'wait' statement. I was also surprised to see 14 threads active at the point of this crash.
Do you recall the history behind these locks? I vaguely remember they addressed a race condition between art and root. I'd like to review the locks from first principles and see if we can understand what their purpose is, what's going wrong and how to permanently fix it.