Network Operations Engineer, AI Networking
Role overview
We are seeking an Infrastructure Operations Engineer to operate and improve the large-scale Ethernet fabrics that support GPU clusters, storage systems, and management infrastructure. This role combines hands-on production operations with automation, observability, and incident response across a global AI network.
The ideal candidate has experience operating high-availability data center, cloud, AI, or HPC networks and can move comfortably from physical-layer troubleshooting to routing and fabric behavior, change execution, and root-cause analysis. You will partner closely with network architecture, systems engineering, GPU engineering, storage engineering, security, deployment, site operations, service providers, colocation partners, and hardware vendors to raise reliability and reduce operational toil.
Responsibilities
- check_circle Own the operational health, availability, and reliability of production AI network infrastructure across 1P and 3P data centers.
- check_circle Monitor, troubleshoot, and resolve network incidents while meeting service-level objectives (SLOs), reducing Mean Time to Detect (MTTD), and minimizing Mean Time to Recovery (MTTR).
- check_circle Operate and maintain large-scale Ethernet fabrics supporting GPU compute, storage, and management networks.
- check_circle Execute production network changes, maintenance windows, and capacity expansions with minimal customer impact.
- check_circle Manage the hardware lifecycle, including switch and optics replacements, RMA coordination, software upgrades, and preventive maintenance.
- check_circle Support new AI cluster deployments, data center expansions, and infrastructure migrations in partnership with deployment and engineering teams.
- check_circle Partner with cloud service providers (CSPs), colocation providers, Smart Hands teams, and hardware vendors to maintain production infrastructure.
- check_circle Perform root-cause analysis (RCA) for production incidents and drive permanent corrective actions that eliminate recurring issues.
- check_circle Build and maintain monitoring, telemetry, dashboards, and alerting to improve network observability and proactive issue detection.
- check_circle Develop and improve operational runbooks, playbooks, troubleshooting documentation, and standard operating procedures.
- check_circle Automate repetitive operational tasks using Python and infrastructure automation frameworks to reduce toil and improve efficiency.
- check_circle Continuously identify opportunities to improve service reliability, scalability, operational maturity, and engineering efficiency.
- check_circle Bachelor’s degree in Computer Science, Network Engineering, or a related discipline, or equivalent practical experience.
- check_circle 5+ years of experience operating large-scale data center, cloud, AI, or HPC network infrastructure.
- check_circle Experience supporting production network environments with high-availability requirements.
- check_circle Hands-on experience with one or more of the following platforms: Cisco NX-OS, Arista EOS, NVIDIA Spectrum / Cumulus Linux, or Juniper JunOS.
- check_circle Strong knowledge of Layer 2 and Layer 3 networking, BGP, OSPF, ECMP, MLAG, LACP, VRFs, and VLANs.
- check_circle Experience troubleshooting physical infrastructure, including fiber optics, transceivers, DAC/AOC cables, and high-speed Ethernet links.
- check_circle Experience performing software upgrades, hardware maintenance, and production change management.
- check_circle Excellent analytical and troubleshooting skills, with the ability to communicate technical risk clearly across teams.
- check_circle Experience operating AI or High-Performance Computing (HPC) network environments.
- check_circle Experience with NVIDIA AI networking technologies and GPU infrastructure.
- check_circle Experience supporting RoCE v2 or RDMA-based Ethernet fabrics, with a strong understanding of Priority Flow Control (PFC), Explicit Congestion Notification (ECN), Data Center Quantized Congestion Notification (DCQCN), Quality of Service (QoS), and lossless Ethernet networking.
- check_circle Experience supporting 100G, 200G, 400G, and 800G Ethernet networks.
- check_circle Experience with GPU platforms including NVIDIA HGX, DGX, GB200, or equivalent AI infrastructure.
- check_circle Experience supporting distributed storage environments such as VAST, DDN, or similar technologies.
- check_circle Experience working with cloud service providers such as AWS, Azure, or Google Cloud, and with third-party colocation providers.
- check_circle Experience with network monitoring and telemetry technologies, including Prometheus, Grafana, gNMI, streaming telemetry, SNMP, or similar tools.
- check_circle Experience developing automation using Python, Git, REST APIs, Terraform, or similar automation frameworks.
- check_circle Participate in a 24x7 on-call rotation supporting mission-critical AI infrastructure.
- check_circle Support time-sensitive production incidents, maintenance windows, capacity expansions, and network changes with a focus on service availability and minimal customer impact.
- check_circle This role requires up to 30% travel to data center locations for new turnups and acceptance activities, as needed.
About the company
OpenAI’s Infrastructure Operations team is responsible for the availability, reliability, and operational excellence of one of the world’s largest AI infrastructure networks. The team owns day-to-day operations of production AI networks across first-party (1P) and third-party (3P) data centers, working with colocation providers, deployment teams, and hardware vendors to deliver highly available GPU infrastructure for AI training and inference workloads.
Tags & Focus Areas
About OpenAI
OpenAI is an AI research and deployment company. Our mission is to ensure that artificial general intelligence benefits all of humanity.
Ready to Join the Team?
Apply once with DevFound — we route your profile to OpenAI and keep you posted on matching AI roles.