How to hire a GPU cluster operations team

Few engineers have run NVIDIA's Quantum-X800 XDR switches in production yet. Most strong candidates have run NDR fabrics on Quantum-2. We screen for UFM, SHARP and NCCL debugging and treat XDR as a bonus.

Engineers from large RoCE Ethernet fabrics can learn InfiniBand, and on an Ethernet-based cluster they are the first pool to search. Ask early which fabric the cluster runs, because it decides which pool the search starts in.

This is usually the hardest seat. Few people have tuned a parallel file system for checkpoint traffic from thousands of GPUs. The closest pool is HPC centers and national labs running Lustre or GPFS, where the I/O patterns are similar.

If the storage platform is already chosen, search for that file system first. A GPFS engineer can learn WEKA, and the reverse is true, but on a cluster going live in weeks the first hire should already know the platform.

Engineers who run Slurm on a large fleet and can read DCGM output and Xid errors mostly come from GPU clouds, HPC centers and university research computing groups. Platform engineers who have only run Kubernetes can do the job on a Kubernetes cluster, and most have not handled GPU hardware faults at scale.

This seat pulls from data center technicians, smart hands and NOC staff. It works well as contract-to-hire and runs as its own search, with its own screening: cable management in dense racks, optics handling, and following an RMA process.

1. Network: an all-reduce across 512 GPUs runs at half the expected bandwidth. Where do you start?

   Strong: Runs NCCL tests to isolate the slow ranks, checks UFM for link errors and congestion, looks at port counters for a bad cable or optic, and confirms the SHARP and adaptive routing settings.

   Red flag: Only talks about rebooting nodes or about Ethernet tools on an InfiniBand fabric.

2. Storage: a training job stalls every time it writes a checkpoint. What do you look at?

   Strong: Metadata server load, stripe and block settings, whether thousands of ranks write into one directory, client caching, and the network path to storage. Names the tools for their file system, such as mmperfmon on Storage Scale or lfs and lctl on Lustre.

3. Systems: a node keeps logging Xid 79. What does it mean and what do you do?

   Strong: The GPU has fallen off the bus. Drains the node in Slurm, checks DCGM and the kernel log, reseats or opens an RMA, and tracks nodes that keep failing.

   Red flag: Reboots the node and puts it back in the pool.

4. Cluster operations: walk me through replacing a failed optic in a dense rack.

   Strong: Works from a ticket and change window, confirms the port in the fabric manager, uses ESD protection, inspects and cleans the connector with a scope, labels the cable, and checks link state and error counters afterward.

One question we ask for each seat, with what a strong answer includes.

For the network, storage and systems seats you are recruiting against CoreWeave, Lambda, Crusoe, Together AI and NVIDIA, which all post for the same skills, along with the national labs. Candidates in these pools often have more than one conversation going, so a short interview loop and a written offer range help.

Posted base pay on recent U.S. postings for these seats:

• ●CoreWeave Senior Network Engineer, Data Center, five U.S. locations: $139,000 to $242,000, posted March 7, 2026
• ●CoreWeave HPC Network Engineer, Bellevue, WA: $109,000 to $204,000, posting closed June 10, 2026
• ●NVIDIA Senior HPC Storage Engineer, Santa Clara and Austin: $184,000 to $287,500 at level 4 and $224,000 to $356,500 at level 5, posted about March 2026
• ●CoreWeave Data Center Technician, Sandusky, OH: $65,000 to $85,000, posted February 23, 2026

Ranges are as posted, and some of these postings have closed. Lambda's HPC operations postings we read did not show a range. We read these on October 30, 2026.

Open the network and storage searches first, since their candidate pools are the smallest. Systems follows. Cluster operations can run in parallel as contract-to-hire.