Watcher: In-Situ Failure Diagnosis

In one paragraph

When a production program crashes, developers usually get a core dump or a log and a long investigation. Shipping the dump elsewhere also leaks whatever was in memory. Watcher performs the diagnosis in situ, inside the failing process itself. It records execution with iReplayer at low overhead; on a failure it replays the last window identically, then combines binary analysis, dynamic slicing, and hardware tracing to walk back from the failure signal to the instruction and the data that caused it. Only the root cause leaves the machine. On 24 real failures in software such as Memcached, SQLite, and OpenJPEG, it found the root causes in a few seconds.

Key ideas

• In-situ diagnosis: the analysis runs inside the failing process, so no core file or memory dump has to be shared. That removes the privacy problem of offline diagnosis.
• A hybrid analysis: identical record-and-replay (iReplayer) plus binary analysis, dynamic slicing, and happens-before analysis, with last-win and value-confirmation checks that use the live process to prune wrong dependences.
• Expensive work only on failure: normal execution pays only the recording cost. Slicing and re-execution start after a failure signal such as SIGSEGV or SIGFPE.
• Software breakpoints and hardware-assisted tracing extend the diagnosis to failures caused by control-flow hijacks, where the crash site itself is corrupted.
• No custom hardware, no custom kernel, no program modification, no recompilation.

Results

• 24 real-world failures diagnosed, including in Memcached, SQLite, and OpenJPEG. Root causes were reported in a few seconds, with enough information to fix the bug without further debugging.
• About 1.76% average overhead on normal, failure-free execution compared with the default Linux libraries, and under 7% for most applications.
• Binary-level analysis on the running program: no change or recompilation of the application, no OS modification, no extra hardware.