Note: The job is a remote job and is open to candidates in USA. Nscale is the GPU cloud engineered for AI, providing high-performance infrastructure for AI-focused companies. The Principal Network Engineer will lead the technical direction and operational strategy for AI interconnect networks, focusing on reliability and scalability while guiding complex network challenges.
Responsibilities
- Owning the technical direction and operational strategy for Nscale’s AI interconnect networks
- Designing, reviewing, and evolving large-scale Infiniband and RoCE fabric architectures to support future growth and workload demands
- Acting as the senior escalation point for the most complex network incidents, guiding deep technical investigations and systemic fixes
- Driving cross-team initiatives to improve fabric reliability, performance predictability, and operational maturity
- Defining standards for hardware configuration, congestion control, routing, firmware lifecycle management, and change safety
- Partnering with SRE, Compute Platform, and Network Architecture teams to influence end-to-end system design
- Mentoring senior and mid-level network engineers, raising the bar for operational rigor and technical excellence
- Driving measurable improvements in uptime, latency consistency, capacity efficiency, and incident reduction
Skills
- 10+ years of experience in network engineering, with deep focus on HPC, AI, or hyperscale data centre networking
- Expert-level operational and architectural experience with Infiniband and/or large-scale RoCE fabrics
- Deep understanding of RDMA internals, congestion management, and fabric-level failure modes
- Strong expertise in modern data centre routing and control planes (BGP, OSPF, ECMP)
- Proven ability to debug and resolve cross-layer issues spanning hardware, firmware, kernel, and application communication libraries
- Demonstrated ability to lead complex technical initiatives across teams without direct authority
- A systems-level mindset, balancing performance, reliability, scalability, and operational cost
- Extensive experience with NVIDIA/Mellanox networking platforms in production AI or HPC environments
- Deep familiarity with distributed training frameworks and GPU communication patterns
- Experience designing network observability systems for high-cardinality, high-throughput environments
- Prior experience influencing platform or infrastructure strategy at scale
Benefits
- Bonus
- Equity
- Commission programs
- Medical
- Dental
- Vision
- Flexible paid time off
- Parental leave
- Retirement plan participation
Company Overview