Senior Infrastructure Engineer

Staff Infrastructure Engineer, Managed Inference

San Francisco - On-site

Compensation: $350,000–$450,000

My client is hiring a staff infrastructure engineer to own the production serving layer for managed inference. The role is an individual contributor seat, at staff scope.

What you will do

• Design, operate, and scale Kubernetes across AWS, GCP, and GPU cloud providers, supporting global inference demand at 1,000+ tokens per second.
• Own the production serving layer for LLM inference.
• Debug NVIDIA GPU systems end to end, including drivers, CUDA, NCCL, node health, and the network fabric.
• Work with the model team so performance work holds up in production, on real hardware and real traffic.
• Extend the deployment tooling so a model can be shipped the same way onto NVIDIA, Trainium, and TPU hosts, including mixed fleets.
• Build observability for time to first token, inter-token latency, throughput, and availability, measured per model, per chip, and per region.
• Own reliability engineering: alerting, automated failover, self-healing infrastructure, and traffic routing across models, chips, and regions.

What you bring

• Five or more years of infrastructure engineering, currently operating at staff level or above in an individual contributor role.
• Hands-on experience operating production Kubernetes at scale across AWS and GCP.
• Direct, day-to-day experience with inference infrastructure.
• Hands-on experience with NVIDIA GPU systems: drivers, CUDA, NCCL, and node health.
• Hands-on experience with an inference serving framework such as vLLM, SGLang, TensorRT-LLM, or Dynamo.
• Experience with HPC networking, including InfiniBand, RDMA/EFA, or MPI.
• Experience with observability and reliability for latency-sensitive services, including metrics, alerting, and automated failover.
• A track record of building highly available, multi-cloud infrastructure.
• Helpful, not required
• Experience on a production inference platform, for example Fireworks, Baseten, Nebius, or Anyscale.
• Experience deploying more than one accelerator family, such as Trainium, TPU, or a mixed fleet.
• Previous experience at Meta.

Location and pay

This role is on-site in San Francisco. The range is $350,000–$450,000.

How to apply

Apply on this posting, or message Adam Rumpler at Xpertise if you would like to talk first.