Senior Platform Reliability Engineer (Fabric and Interconnect)

Founded in Australia in 2019, our mission is to create the most efficient AI infrastructure by combining cutting-edge technology with a steadfast commitment to sustainability.

At Firmus, we are unique in our approach. We design, build, and operate a new class of digital infrastructure – the AI Factory. Through our model-to-grid technology approach, we have pushed the boundaries of multi-generational liquid cooling systems, energy management, AI software orchestration, and construction. Fo...

Our large-scale GPU cloud platform, Firmus AI Cloud, is purpose-built to deliver energy-efficient AI compute at scale to customers.

Responsible for the reliable operation, automation and continuous improvement of the estate's GPU interconnect and network fabrics (for example NVLink and NVSwitch domains, InfiniBand and Spectrum-X). 
Build and maintain the guarded automation and remediation tooling for fabric faults, contributing to the software-driven remediation of AI clusters, including fault isolation and fabric reconvergence. 
Diagnose and tune performance across the interconnect stack, from application collective communication down to link level, working with technologies including NVLink, InfiniBand, RoCE and congestion control tuning. 
Operate DPU-based host networking across the fleet, including offload path configuration and driver and firmware compatibility. 
Execute firmware upgrade waves, fabric expansions and capacity changes to the supported paths and scaling patterns defined by AI Infrastructure, owning the production window, the staged or canary path, verification against declared success criteria, and rollback execution. 
Provide the deepest technical expertise for fabric and interconnect faults, correlating a collective communication failure to a specific physical link and diagnosing the faults that require internals-level knowledge to root cause, and driving the permanent fix to closure through AI Infrastructure. 
Lead vendor escalations at engineering level, reproducing faults to the vendor's standard and pushing back credibly when a diagnosis does not explain the observed behaviour. 
Lead technical recovery during major fabric incidents, drive the changes that remove repeat causes, share the follow-the-sun on-call roster, and mentor the engineers who carry frontline diagnosis, documenting operational procedures, runbooks and performance results. 

 

Strong skills in high-performance networking and systems engineering, with 8+ years of experience including substantial ownership of production network or interconnect infrastructure in a 24/7 environment. 
Deep operational experience with high-performance GPU interconnect fabrics (for example NVLink and NVSwitch), including domain topology and failure diagnosis. 
Extensive experience with high-performance networking fabrics (for example InfiniBand or RoCE-based Ethernet such as NVIDIA Spectrum-X), including routing internals and congestion control tuning. 
Experience with DPU or SmartNIC-based host networking, including offload paths and driver and firmware coordination. 
Experience planning and executing firmware upgrade waves and capacity expansions on production fabrics with defined rollback. 
Strong skills in infrastructure automation and infrastructure-as-code practices, with change delivered through peer review and progressive rollout. 
Practical experience with scripting or programming for operational automation and tooling, such as Python, Go or Bash. 
Proven ability to act as a senior escalation point in production, including major incident response, on-call participation, vendor escalation at engineering level, and the production of runbooks that others can execute successfully. 
Clear technical judgement and communication skills, with the ability to explain complex fabric failures to engineers and non-specialists. 
Experience operating fabrics for large-scale distributed training or inference workloads. 
Experience with NVIDIA rack-scale or multi-node GPU systems and their interconnect topology. 
Experience in a multi-tenant service provider, cloud or colocation environment. 
Knowledge of data centre and hardware fundamentals, including cabling and optics, firmware management and hardware fault workflows. 
Relevant vendor certification. 

 
 
On-call: The function runs 24/7. First line monitoring and first response sit with the operations centre. This role shares the after-hours escalation roster for its domain with the other senior engineers in the function. 
Reporting to:  Reports to the Head of AI FactoryOS Operations while the function is being established, working under broad direction with a high degree of autonomy and direct access to the decision makers. As the function reaches its planned structure, the role will report to the Infrastructure Operations Manager, with the Head of AI FactoryOS Operations remaining accountable for the function. The scope, level and remit of the role do not change under either arrangement. 
 
Permanent full-time