We look for data engineers who are excited to tackle unsolved problems. Physical observations arrive continuously, in many formats, at a scale that dwarfs what is used to train today's LLMs. Your mission is to build the data platform underneath it all - the storage, compute, and loading systems that make every dataset cheap to ingest, fast to query, and immediately available to training.
Responsibilities- Design and operate petabyte-scale storage: lakehouse architecture, file formats, and data layout optimized for both batch and real-time queries
- Own the shared compute and orchestration platform (e.g. Spark, Ray, workflow scheduling) that ingestion and research pipelines run on
- Optimize data strategy end to end from storage to loading, owning high-throughput data loading into training up to the tensor boundary
- Build systems for cataloging, deduplication, lineage, search, and reproducibility at every stage of the data lifecycle
- Implement the platform-level quality and monitoring tooling that data and research teams build their checks on
- Scale infrastructure to improve engineering velocity and ensure reliability, with monitoring and alerting to match
- Work across the full data lifecycle when the mission needs it - including building and operating ingestion pipelines for critical data sources directly
What we're looking forWe value a relentless approach to problem-solving, rapid execution, and the ability to quickly learn in unfamiliar domains.
- Demonstrated experience building large-scale data pipelines and distributed compute systems (e.g. Spark, Ray, Beam)
- Knowledge of state-of-the-art methods and tools for data ingestion, storage, and loading - including file formats and storage systems (e.g. Parquet, Zarr, Delta Lake) and how they impact performance and scalability
- Deep familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines
- Understanding of how data loading throughput affects large-scale training, and experience optimizing it
- Owns deliverables end-to-end, from collecting and translating requirements to autonomously driving execution
