Designation : L4 (Account Manager)
Location : US (Remote)
Experience : 9 - 15 years
Job Role : Principal Data Engineer, AI Platforms
Responsibilities :
Detailed Skill Specifications
1. Data Engineering Lead
- Data Pipelines for AI Workloads: Deep expertise in building low-latency ingestion engines, unstructured data processing (audio, text, image, video ETL), and distributed processing using Apache Spark or Ray.
- Vector & Feature Store Engineering: Hands-on experience operationalizing vector databases for retrieval-augmented generation (RAG) and low-latency feature stores for real-time model inference.
- Continuous Integration & MLOps Integration: Ability to orchestrate end-to-end retraining triggers based on data drift, upstream schema changes, and model degradation signals.
- Technical Execution: Proven track record in code quality, distributed systems debugging, load testing, and database tuning.
2. Data Engineering Strategist
- AI Readiness & Roadmapping: Ability to audit existing data assets, identify technical debt, and design multi-phase roadmaps that support generative and predictive AI initiatives.
- Financial & Resource Modeling (Data FinOps): Experience forecasting cloud compute/storage costs associated with large-scale model training, vector indexing, and pipeline scaling.
- Data Product Thinking: Treating data as an internal product by defining data contracts, domain-driven ownership (Data Mesh), and semantic layers for cross-functional consumption.
- Regulatory & Compliance Architecture: Formulating strict data retention, anonymization, and provenance standards to ensure AI training data remains compliant with emerging international AI regulations.
Required skills :
Principal Data Engineer (AI Platforms)
- Must-Have Core Skills:
- Advanced Programming & Querying: Fluency in Python, Scala, or Java, alongside advanced SQL (window functions, query plan optimization, CTEs).
- Distributed Data Processing: Real-world experience with Apache Spark, Ray, Apache Flink, and distributed computing patterns.
- Storage & Lakehouse Architecture: Hands-on design with Apache Iceberg, Delta Lake, Snowflake, or Databricks.
- Orchestration & Workflow Management: Apache Airflow, Prefect, Dagster, or dbt for transformation pipelines.
- AI Product Lifecycle & MLOps:
- Vector & Feature Stores: Implementing and querying vector databases (Milvus, Pinecone, Qdrant) and feature stores (Feast, Hopsworks) for real-time inference and RAG pipelines.
- Unstructured Data Pipelines: Ingesting and preprocessing multi-modal data (text, documents, audio, images) for embedding generation and model training.
- Data Observability & Drift Monitoring: Automated data quality testing (Great Expectations, Soda) and alerting on data/concept drift.
- Cloud, Systems & DevOps:
- Containerization & orchestration (Docker, Kubernetes/EKS/GKE).
- Cloud platform data services (AWS, GCP, or Azure) with Infrastructure as Code (Terraform).
- CI/CD automation for data and model pipelines.
Principal Data Strategist (AI & Governance)
- Must-Have Core Skills:
- Enterprise Data Roadmapping: Designing 3-to-5-year data strategies that directly align with commercial AI product roadmaps.
- Modern Data Stack Evaluation: Structuring "build vs. buy" decision frameworks, vendor RFP evaluations, and total cost of ownership (TCO) analyses.
- Data Product Architecture: Applying Data Mesh principles, establishing data contracts, and defining semantic layers across domain teams.
- AI Readiness & Financial Modeling:
- Data FinOps: Forecasting and optimizing compute, vector index storage, and inference API costs across the AI lifecycle.
- Training Data Maturity & Lineage: Assessing data quality, provenance, and readiness for proprietary model fine-tuning and retrieval systems.
- Governance, Risk & Compliance:
- Responsible AI & Regulatory Compliance: Designing policy controls aligned with the EU AI Act, GDPR, HIPAA, and IP licensing for training datasets.
- Data Access & Security Governance: Defining RBAC/ABAC models, PII masking, data lineage tracking (OpenLineage), and audit readiness.
- Executive Leadership:
- Translating deep technical data constraints into business value for C-suite stakeholders (CDO, CTO, VP of Product)
