Description & RequirementsThe Senior Data Engineer is responsible for designing, building, and maintaining scalable data pipelines, data platforms, and architecture that support analytics, business intelligence, machine learning, AI, and agentic solutions. This role works closely with data scientists, analysts, AI/ML engineers, application teams, and business stakeholders to ensure data solutions are reliable, secure, scalable, and aligned with business needs and objectives.
Key Responsibilities
- Analyze business and functional requirements; design, develop, and optimize data pipelines, workflows, and data services
- Collaborate with data scientists, analysts, AI/ML engineers, product teams, and business stakeholders to gather requirements and deliver data solutions
- Design and implement scalable ELT/ETL processes for large-scale structured, semi-structured, and unstructured data sets
- Build and maintain data products that support analytics, reporting, machine learning, generative AI, retrieval-augmented generation, and intelligent agent workflows
- Develop, operationalize, and optimize data pipelines that support AI and agentic systems, including data ingestion, transformation, feature preparation, metadata enrichment, and knowledge retrieval
- Support AI/ML and GenAI initiatives by preparing high-quality, governed, and discoverable data for model training, evaluation, inference, and monitoring
- Partner with AI teams to integrate enterprise data with vector databases, embeddings, knowledge graphs, semantic layers, APIs, and agent orchestration frameworks where appropriate
- Ensure data quality, observability, lineage, integrity, and reliability across various data sources and downstream consumers
- Monitor, troubleshoot, and optimize data pipeline performance, cost, scalability, and reliability
- Perform code reviews and ensure solutions align with predefined architectural standards, engineering guidelines, security requirements, best practices, and quality standards
- Optimize database, data warehouse, and lakehouse systems for performance, scalability, cost efficiency, and AI-readiness
- Implement data security, privacy, governance, access control, and compliance measures across data and AI-enabled workflows
- Understand and comply with the established software development life cycle methodology
- Proactively identify opportunities for automation, process improvement, and platform modernization
- Establish and enhance technical guidelines and best practices for the data engineering and integration development teams
- Utilize subject matter expertise in enterprise applications and data solutions to evaluate complex, sensitive business problems and architect technical solutions
- Mentor junior data engineers and provide technical guidance on data engineering, AI-enabling data patterns, scalable architecture, and engineering best practices
- Stay current with emerging data, cloud, AI, GenAI, agentic AI, and data platform technologies
Qualifications
- Bachelor's or Master's degree in Computer Science, Engineering, Data Engineering, Information Systems, or a related field
- 8+ years of experience leading design and development in BI, data engineering, or enterprise data environments scaling to hundreds of users and multiple terabytes of content
- Proficiency in SQL and experience with relational databases
- Knowledge of data warehousing and lakehouse solutions such as Snowflake, Redshift, BigQuery, Databricks, or similar platforms
- Experience with big data technologies such as Hadoop, Spark, or distributed data processing frameworks
- Familiarity with cloud platforms such as AWS, Azure, or Google Cloud
- Strong programming skills in languages such as Python, Java, JavaScript, or Scala
- Strong understanding of RDBMS concepts, data modeling techniques including 3NF and dimensional modeling, database programming, and performance tuning
- Experience designing reliable, scalable, and maintainable data pipelines for analytics, machine learning, or AI use cases
- Familiarity with AI/ML data lifecycle concepts, including feature engineering, model-ready data preparation, data quality, evaluation data sets, and production data monitoring
- Understanding of generative AI concepts such as embeddings, vector search, retrieval-augmented generation, prompt workflows, and enterprise knowledge retrieval
- Familiarity with agentic AI patterns, including tool use, orchestration, workflow automation, memory, context management, and integration with enterprise systems
- Experience applying data governance, security, privacy, and compliance controls to data products and AI-enabled systems
- Excellent problem-solving, analytical, communication, and collaboration skills
- Experience working in Agile methodology
Preferred Qualifications
- Experience with real-time data processing and streaming technologies such as Kafka, Flink, Spark Streaming, or Kinesis
- Experience with vector databases or search platforms such as Pinecone, Weaviate, OpenSearch, Elasticsearch, pgvector, Snowflake Cortex Search, or similar technologies
- Experience supporting retrieval-augmented generation, enterprise search, semantic data layers, knowledge graphs, or AI-powered data products
- Familiarity with AI agent frameworks or orchestration tools such as LangChain, LangGraph, LlamaIndex, Semantic Kernel, AutoGen, CrewAI, or similar technologies
- Experience integrating data platforms with APIs, microservices, workflow orchestration tools, or internal developer platforms
- Experience with MLOps, LLMOps, model monitoring, prompt/version management, or AI evaluation frameworks
- Experience with data observability, lineage, cataloging, and governance tools
- Experience in the gaming industry, digital entertainment, customer experience, or large-scale consumer data environments
Post To
External careers site, Internal careers site
LinkedInID
1449
