5-7 years of experience in distributed systems design
Proficiency in Python and associated libraries like Pandas and NumPy
Familiarity with container orchestration using Kubernetes
Understanding of ETL system design and implementation
Knowledge of networking principles, including InfiniBand and RDMA
Experience with large-scale data processing workflows
Background in applied machine learning techniques
Responsibilities
Design multi-stage data pipelines for processing petabyte-scale datasets
Manage distributed training across 1000+ GPU clusters
Optimize performance of dataloaders for high-throughput image processing
Implement fault tolerance mechanisms for large-scale AI pretraining
Collaborate with researchers on enhancing reinforcement learning infrastructure
Develop custom distributed data stores and orchestration systems
Build systems that optimize the use of raw cluster resources for research outcomes
Benefits
Flexible work environment with opportunities for remote work
Access to cutting-edge technology and tools
Opportunities for professional growth and development
Collaborative culture emphasizing innovation and experimentation
Engagement with high-impact research projects in AI and machine learning
Full Job Description
Supercomputing / AI Infra at Krea
We build and operate the infrastructure for Krea's research and inference. Distributed training, 1000+ K8s GPU clusters, petabyte scale data pipelines, etc. We build a lot of this from scratch - custom distributed datastores, job orchestration systems, and streaming pipelines that replace tools like Kafka and Ray for modern AI workloads at scale.
Example projects:
Distributed data systems
Design multi-stage pipelines that turn petabytes of raw data into clean, annotated datasets
Run classification models on billions of images
Deploy and combine LLMs to caption massive multimedia data
GPU infrastructure
Manage distributed training and inference on 1000+ GPU Kubernetes clusters
Solve orchestration and scaling for large-scale GPU job processing
Scale workloads and research between clusters in multiple datacenters
Distributed training
Profile and optimize dataloaders streaming thousands of images per second
Profile and debug InfiniBand networking on huge training runs
Build fault tolerance systems for large-scale pretraining
Collaborate with researchers on evolving RL infrastructure
Applied ML pipelines
Find clean scenes in millions of videos using distributed shot-boundary detection
Customize and train models to filter billions of images for questions like "is this a screenshot?"
Build the systems that bridge raw cluster capacity and research output
Who we're looking for:
Systems people. If you've read a blog post about InfiniBand debugging or building a custom distributed database and thought "I want to do that" - this is that team.
You'll spend your time working heavily with Python, Kubernetes, Torch, and data tools like DuckDB, Arrow, etc. It's OK if you don't have K8s or ML experience - the main thing we hire for is an intuition for distributed systems, and a great mental model of how systems interact and function under different conditions.
