Position OverviewYou own the IO layer that trains the models - and the signals we need to predict storage faults before a checkpoint stalls a $50M training run.
Bitdeer is building an AI-operated GPU cloud. Storage is where AI workloads either fly or fall over: a slow parallel read can starve a 1,000-GPU job; a stalled checkpoint can waste a full training epoch. In this role you deploy and operate the high-performance storage layer for AI training and inference across NeoCloud's US DCs, and you feed the AIOps substrate with the signals it needs to catch storage regressions before they page a customer.
What you'll own- Deploy and operate parallel/distributed storage systems: WEKA, VAST Data, Ceph, DDN/Lustre. Design storage architectures optimized for AI workload patterns - checkpoint I/O bursts, sequential dataset reads, KV cache for inference.
- Implement multi-tenant storage isolation with per-tenant QoS, quotas, and access controls; configure and optimize GPU Direct Storage for direct GPU-to-storage data paths.
- Deploy and manage storage networking (NFS over RDMA, NVMe-oF, high-speed storage fabrics) and Nvidia CMX for cluster-wide storage orchestration.
- Diagnose and tune storage performance: IOPS, throughput, latency profiling with fio, IOR, mdtest; own the runbook for common failure modes.
- Plan storage capacity aligned with GPU cluster growth and customer workload projections; manage firmware, data migration, and DR procedures.
Feed the AIOps substrate- Instrument storage telemetry - IO tail latency, checkpoint durations, NVMe SMART, filesystem health, RDMA counters - into the metrics/logs/traces store the platform team runs.
- Partner with the platform team to define the storage-fault predictor: which signals, which labels (from your incidents), which false-positive tolerances.
- Convert every novel incident into an automation: SOPs become runbook-as-code, runbook-as-code becomes an agent-executable remediation.
What success looks like in year 1- Observability and a baseline predictor for the top 3 storage-fault classes on our fabric.
- Storage-incident MTTR measurably lower than at hire.
- The Nvidia GB200-class clusters we build out ship on your storage design.
Job Requirement:- 5+ years in enterprise or HPC storage operations, with at least 2 years supporting AI/ML workloads
- Hands-on deployment and operations experience with at least two of: WEKA, VAST Data, Ceph, DDN/Lustre
- Strong understanding of AI training I/O patterns: checkpoint frequency, dataset loading, shuffle buffers
- Experience with high-performance storage networking (NFS over RDMA, NVMe-oF)
- Knowledge of GPU Direct Storage and RDMA-based data transfer
- Proficiency in storage performance benchmarking and tuning (fio, IOR, mdtest)
- Experience implementing multi-tenant storage with isolation and QoS
- Strong Linux systems knowledge (kernel tuning, filesystem internals, block device management)
- Instinct for turning ops toil into ML signal - you've either shipped an anomaly detector for storage/IO telemetry or you can articulate the labels and features you'd need to.
- Runbook-as-code mindset - every SOP you write should be executable by a machine within a quarter.
