GPU Runtime Engineer

About the role

Replaying CPU workloads is solved. Replaying CUDA streams and NCCL collectives deterministically is the frontier, and it is what turns Watcher from diagnosis into automatic recovery for GPU training. You will instrument the CUDA runtime boundary, pin collective algorithm choice and rank arrival order, and build the device-health layer that quarantines a failing GPU before it takes the run down.

What you will do

• Record and enforce CUDA stream ordering, events, and synchronization points
• Capture NCCL algorithm selection and rank arrival order so collectives replay exactly
• Build per-GPU health signals (ECC errors, thermal throttling, Xid events) and the quarantine policy on top of them
• Benchmark overhead and recovery time on multi-node clusters
• Work with hardware partners on device-level failure signatures

What we are looking for

• Shipped software that sits at the CUDA runtime or driver boundary
• Working knowledge of NCCL and multi-GPU communication patterns
• Strong C++ and performance engineering instincts
• Experience operating or debugging multi-node GPU jobs

Nice to have

• Experience with GPU virtualization, interception layers, or profilers
• Familiarity with PyTorch distributed internals