Deep experience in ML dataset infrastructure, including storage formats and dataset versioning.
Strong Python skills with familiarity in PyTorch and distributed systems.
Proven track record in technical leadership for platform or infrastructure projects.
Experience managing and mentoring engineering teams while balancing technical and strategic responsibilities.
Excellent communication skills for collaboration across diverse teams.
Responsibilities
Lead and develop the ML Dataset Platform team, setting technical direction and roadmap.
Define APIs, tools, and storage formats for efficient ML dataset management.
Build scalable tooling for dataset versioning and incremental updates.
Design APIs for training-time data transformations, balancing reproducibility and flexibility.
Manage common dataloading tools, ensuring observability and performance optimization.
Streamline multi-cloud data access for predictable cost and latency.
Ensure automatic data lineage tracking from raw data to trained models.
Collaborate with ML Training Platform and development teams to enhance shared-dataset tooling.
Benefits
Competitive compensation and equity awards.
Comprehensive health and wellness benefits including medical, dental, and vision coverage.
Unlimited vacation policy.
Flexible working hours and support for remote work.
Daily drinks, snacks, and catered meals when in the office.
Regular team-building activities and social events, both in-person and virtual.
Full Job Description
As TLM of the ML Datasets Platform, you will play a critical role defining the APIs, tooling, and storage formats that make datasets a shared, first-class asset at Waabi: versioned, efficiently updatable, fully traceable, and fast to load across GPU workloads. The technical challenge is to find the right abstractions so that every ML team can integrate with the platform for its model development use case without being forced into a one-size-fits-all mold, and without building its own data infrastructure on the side. You will do this as a hands-on technical leader who also grows and manages the team, sets its roadmap, and partners closely with the ML Training Platform team and the ML development teams who are the platform's customers.
You will...
Lead and grow the ML Dataset Platform team as both its technical lead and people manager: set the technical direction, define the roadmap, hire, and mentor engineers.
Define the APIs, tools and storage formats for Waabi's ML datasets, balancing maximal reuse of data and tooling against the specific needs of each training workload.
Build the tooling to manage datasets at scale, including versioning and efficient incremental updates without full rebuilds.
Design the APIs that let teams express training-time transformations along the continuum between fully reproducible ahead-of-time ETL and dynamic transformation within data loaders.
Own the common dataloading tools used across workloads, including observability into bottlenecks and the throughput of training jobs.
Streamline data access for multi-cloud training so that datasets are available as needed with predictable cost and latency.
Make data lineage a first-class, fully automatic property of every dataset, from raw data through to the model trained on them.
Partner with the ML Training Platform team on the boundary between data and training infrastructure, and with the ML development teams on transitioning shared-dataset tooling into the new platform.
Drive adoption: work directly with model teams to migrate their training data onto the platform and feed what you learn back into your team's roadmap.
Qualifications:
Deep ML dataset experience: You have designed and built large-scale data infrastructure for ML training (storage formats, dataset versioning, ETL, dataloading) and understand the performance characteristics of petabyte-scale sensor or multimodal data.
Strong software engineering and implementation: You have very strong Python skills and are comfortable in PyTorch training loops; you have strong software-engineering fundamentals and experience with distributed systems, cloud storage, and data processing frameworks (e.g., Spark, Ray, Beam).
Proven technical leadership: You have led a platform or infrastructure effort used by multiple teams, defined its APIs, and driven adoption across an organization.
People leadership: You have managed or tech-led a team of engineers, including hiring and mentoring, and can balance hands-on technical contribution with roadmap ownership and stakeholder management.
Team player: You communicate clearly with researchers and engineers, build consensus across teams with different needs, and deliver in a fast-paced environment.
Bonus:
Experience with ML data infrastructure in autonomous driving or robotics (multi-sensor logs, LiDAR, camera, HD maps).
Experience with columnar and tensor storage formats (e.g., Parquet, Lance, WebDataset, Zarr) and dataset versioning tools.
Experience with multi-cloud or hybrid training environments and the data-movement challenges that come with them.
Experience with data lineage, metadata catalogs, or feature-store systems.
The US yearly salary range for this role is: $200,000 - $280,000 USD in addition to competitive perks & benefits. Waabi US Inc.'s yearly salary ranges are determined based on several factors in accordance with the Company's compensation practices. Note: The Company provides additional compensation for employees in this role, including equity incentive awards and an annual performance bonus.
Perks/Benefits:
Competitive compensation and equity awards.
Health and Wellness benefits encompassing Medical, Dental and Vision coverage (for full-time employees only).
Unlimited Vacation.
Flexible hours and Work from Home support.
Daily drinks, snacks and catered meals (when in office).
Regularly scheduled team building activities and social events both on-site, off-site & virtually.
As we grow, this list continues to evolve!
About Waabi
Waabi is an artificial intelligence company that develops self-driving car technology. The company's focus is on developing a system that can safely and efficiently navigate complex urban environments. Waabi's technology is based on deep learning algorithms that allow the system to learn from experience and improve over time. The company was founded in 2021 by a team of experts in the field of autonomous vehicles and is headquartered in San Francisco, California.
