Key Responsibilities
Design, build, operate, and continuously improve the proprietary Kubernetes-native job scheduling platform for AI workloads.
Develop scheduler architecture, custom resource definitions, Kubernetes controllers, admission webhooks, scheduling plugins, APIs, CLI tools, and automation required to support workload submission, placement, execution, monitoring, and recovery.
Define and implement scheduling policies for training, fine-tuning, inference, benchmarking, batch, data-processing, and agentic workloads.
Build topology-aware placement mechanisms that account for GPU locality, NVLink/NVSwitch domains, node topology, NUMA affinity, NIC placement, RDMA paths, network-fabric topology, storage locality, and fault-domain boundaries.
Build AI-factory resource-aware scheduling mechanisms that account for cluster capacity, node health, fabric condition, storage performance, GPU availability, maintenance windows, software compatibility, capacity reservations, power limits, thermal conditions, and operational constraints.
Implement workload-control capabilities including admission control, queueing, priorities, quotas, fair sharing, reservations, preemption, gang scheduling, co-scheduling, backfilling, workload aging, retry policies, checkpoint-aware scheduling, deferred execution, and failure recovery.
Develop workload demand-shaping and grid-aware controls that allow eligible workloads to be delayed, paced, prioritized, rescheduled, right-sized, or placed differently in response to capacity, health, power, thermal, maintenance, or other AI-factory operating signals.
Integrate the scheduler with Kubernetes control-plane capabilities, GPU device plugins, node-feature discovery, network and storage services, observability systems, and policy engines.
Integrate, where appropriate, with Slurm, Slinky, vCluster, Kueue, KAI, Volcano, YuniKorn, or related systems to deliver a coherent workload-management experience across AI and HPC use cases.
Build a stable developer and user experience for workload submission and management, including APIs, SDKs, CLI workflows, templates, workload definitions, job-status visibility, event streams, scheduling explanations, and self-service troubleshooting capabilities.
Work with AI and inference engineers to encode validated model and workload recipes into scheduler-aware templates, including requirements for model size, precision, distributed parallelism, GPU count, topology, network, storage, runtime, benchmark target, and expected resource profile.
Enable Model-to-Grid benchmarking by exposing scheduler, placement, resource, and workload-lifecycle data for end-to-end analysis across models, runtimes, GPUs, network, storage, power, thermals, and application performance.
Build observability into the scheduling platform, including queue depth, scheduling latency, admission outcomes, placement decisions, resource fragmentation, topology quality, GPU utilization, job lifecycle, failure reasons, preemption events, retry behavior, power and thermal signals, and workload performance.
Develop actionable scheduler explanations and operator views that show why a workload was queued, admitted, placed, deferred, preempted, or rescheduled and what changes could improve execution outcomes.
Establish performance testing, load testing, scale testing, chaos testing, fault recovery testing, and regression testing for scheduler components and Kubernetes platform integrations.
Operate production Kubernetes clusters and associated scheduling services with strong reliability, security, change-management, backup, recovery, and incident-response practices.
Build CI/CD and GitOps workflows for scheduler code, configuration, CRDs, controllers, policy definitions, workload templates, and cluster-service releases.
Implement secure multi-tenancy and resource-governance controls, including namespace and tenant isolation, RBAC, quotas, workload identity, policy enforcement, audit trails, and controlled access to GPU, data, and infrastructure resources.
Partner with platform, infrastructure, networking, storage, and operations teams to ensure scheduler decisions reflect actual infrastructure capabilities, constraints, health, performance, maintenance, and capacity conditions.
Partner with the Model-to-Grid product and program lead to define scheduler feature scope, milestone plans, acceptance criteria, release gates, benchmark targets, documentation, and adoption measures.
Contribute to technical roadmaps, architecture decisions, design reviews, release readiness, incident postmortems, and continuous improvement of Model-to-Grid and AI-factory operations.
Required Skills & Abilities
5+ years of experience in DevOps, site reliability engineering, platform engineering, distributed systems, cloud infrastructure, or comparable software and systems-engineering roles.
Deep practical expertise in Kubernetes architecture and operations, including control planes, scheduling, controllers, operators, CRDs, admission controllers, APIs, networking, storage, security, and multi-tenancy.
Demonstrated experience building, extending, or deeply integrating workload schedulers, resource managers, or orchestration systems, such as Kubernetes Scheduler Framework, Kueue, KAI, Volcano, YuniKorn, Slurm, Slinky, Run:ai, or proprietary implementations.
Strong programming skills in Go, with Python proficiency for automation, integration, tooling, and performance analysis.
Experience building production Kubernetes controllers, operators, scheduling plugins, admission webhooks, REST or gRPC APIs, CLI tools, and event-driven distributed services.
Strong understanding of GPU-accelerated AI/ML workloads, including distributed training, fine-tuning, model serving, inference, benchmarking, and batch processing.
Practical experience with GPU scheduling and resource allocation, including GPU partitioning, MIG where applicable, GPU affinity, multi-GPU workloads, gang scheduling, and topology-aware placement.
Understanding of GPU system topology and high-performance AI networking, including NVLink, NVSwitch, PCIe, NUMA, RDMA, RoCEv2, NIC affinity, collective communication, and network contention.
Familiarity with large-scale NVIDIA GPU architectures, including NVL72 GB300-class systems and future-generation high-density GPU platforms, and their implications for cluster scheduling, workload placement, network topology, operations, and fault handling.
Experience working with storage and data-path considerations for AI workloads, including high-throughput shared file systems, object storage, caching, data locality, checkpointing, and storage-performance constraints.
Understanding of AI-factory or data-center operations, including capacity planning, workload demand forecasting, power-aware scheduling, energy efficiency, thermal constraints, maintenance coordination, system health, resiliency, and infrastructure telemetry.
Experience with observability and performance analysis using tools and concepts such as Prometheus, Grafana, OpenTelemetry, logs, traces, metrics, DCGM, GPU telemetry, network telemetry, workload profiling, and SLOs.
Experience with CI/CD, GitOps, infrastructure as code, policy as code, and safe production rollout practices using tools such as GitHub Actions, GitLab CI, Argo CD, Flux, Terraform, Helm, Kustomize, OPA, or Kyverno.
Familiarity with cloud-native security practices, including container security, image signing, software supply-chain controls, secrets management, RBAC, workload identity, network policies, runtime security, and vulnerability remediation.
Strong troubleshooting skills across Kubernetes, distributed systems, GPU workloads, networking, storage, operating systems, and application-level job execution.
Kubernetes scheduler and controller engineering.
Distributed systems architecture, reliability, and performance engineering.
GPU workload orchestration for training, inference, benchmarking, and AI applications.
Network-topology-aware and AI-factory-resource-aware workload placement.
Deep understanding of the relationship between workload intent, model configuration, runtime behavior, GPU topology, network, storage, and end-to-end performance.
Model-to-Grid engineering: connecting AI demand, scheduler decisions, benchmark data, infrastructure telemetry, and AI-factory operating constraints.
Multi-objective scheduling design across user throughput, latency, time-to-results, utilization, fairness, cost, power, energy, resilience, and grid or operator constraints.
Developer experience for AI workload submission, templates, APIs, observability, and troubleshooting.
Production ownership, incident response, change control, testing discipline, and operational excellence.
Security-conscious multi-tenant platform engineering.
Pragmatic architecture and technical decision-making under evolving hardware, software, and operational constraints.
Cross-functional collaboration with AI, applications, platform, infrastructure, networking, storage, security, operations, and product teams.
Delivery and adoption of a reliable proprietary Kubernetes-native scheduler that supports the AI & Applications team’s Model-to-Grid product roadmap.
Scheduler availability, reliability, recovery performance, and successful completion of scheduled workload operations.
Improvement in end-user workload outcomes, including lower queue time, reduced time-to-start, improved placement quality, increased job completion rate, faster time-to-results, higher training scaling efficiency, and improved inference throughput and latency.
Improvement in AI-factory resource efficiency, including GPU utilization, reduced resource fragmentation, improved multi-node placement, increased cluster throughput, more effective capacity use, and reduced avoidable network or storage contention.
Percentage of eligible workloads scheduled using topology-aware placement policies that account for GPU, NVLink/NVSwitch, network, RDMA, node, storage, and fault-domain characteristics.
Percentage of eligible workloads scheduled using AI-factory-resource-aware policies incorporating relevant capacity, health, maintenance, power, thermal, storage, or fabric signals.
Measurable accuracy and effectiveness of scheduler placement and admission decisions, based on workload performance relative to expected benchmarks, queue-time outcomes, job-success rate, resource utilization, user overrides, and operator intervention rates.
Reduction in manual scheduling, workload-placement, capacity-allocation, and operational-triage effort for users and AI-factory operators.
Quality and adoption of scheduler APIs, CLI tools, workload templates, model recipes, self-service workflows, scheduling explanations, and developer documentation.
Coverage and quality of scheduler observability, including end-to-end visibility of workload demand, queue state, scheduling decisions, placement rationale, resource use, failure modes, topology signals, and operational constraints.
Successful integration of scheduler telemetry and policy outcomes into Model-to-Grid benchmarking, capacity planning, performance optimization, and operator-facing insights.
Demonstrated support for grid and operator objectives, including improved workload-demand visibility, more predictable capacity use, controlled workload behavior, support for maintenance coordination, and reduced exposure to avoidable power, thermal, capacity, or infrastructure hotspots.
Release quality for scheduler and platform components, measured through automated-test coverage, performance-regression prevention, scale-test results, fault-recovery validation, security posture, deployment reliability, rollback readiness, and incident trends.
Effective collaboration with the Model-to-Grid product and program lead, Platform, Infrastructure, Security, and operations teams, demonstrated by predictable delivery against agreed feature milestones, early dependency identification, and timely resolution of critical blockers
