5-7 years of experience in data engineering or related fields
Expert-level proficiency in Java for ETL framework development
Strong scripting skills in Groovy for ETL configuration
Proficient in Git and version control practices
Deep expertise in Apache Spark for large-scale data processing
Hands-on experience with AWS services, particularly S3 and EMR
Experience with Hadoop ecosystem and data security compliance
Responsibilities
Design and develop ETL workflows using Java for data ingestion
Build and maintain reusable framework components for ETL pipelines
Configure and optimize data ingestion processes with Sqoop and Spark
Monitor and troubleshoot production pipelines for data quality
Implement data security controls for PII data
Collaborate with upstream and downstream teams to ensure data quality
Participate in on-call rotation for critical data feeds
Benefits
Opportunity to work with cutting-edge big data technologies
Collaborative environment with cross-functional teams
Focus on continuous improvement and innovation
Onshore presence in a vibrant San Francisco office
Participation in on-call rotation for critical data feeds
Full Job Description
Job Description
Senior Data Engineer, Data Platform
Location : San Francisco, CA
About the Role
We are seeking a hands-on with senior Big Data Engineer to design, build, and operate mission critical ETL pipelines ingesting petabyte scale database into our Hadoop Ecosystem. You will partner with engineering, analytics and business intelligence teams to improve data delivery speed, pipeline reliability and processing efficiency using Java, Spark, and AWS. This role requires deep technical expertise in distributed systems, production operation mindset and ability to troubleshoot complex data issues on a scale.
Key Responsibilities
Design and develop ETL workflows using Java to ingest data from databases, APIs , and file feeds into Hadoop/Hive
Build and maintain reusable framework components(Actions, UDFs, Serds) for the ETL pipelines
Configure and optimize data ingestion using Sqoop, Spark, and Oozie with incremental load, partitioning and CDC strategies
Monitor and troubleshoot production pipelines ,resolve data quality issues, investigate failures , and optimize performance bottleneck.
Implement data security controls - Encryption/decryption of PII data, AWS Secrets Manager integration, and compliance requirements.
Collaborate with upstream/downstream teams - sources system owners , data analyst and BI teams to ensure data availability and quality.
Participate in on-call rotation for critical production data feeds and SLA compliance.
Must-Have Skills
Programming and Development
Java - Expert -level proficiency in building production grade ETL framework, customers actions, and reusable libraries. Must-Have have experience with Maven multi-model projects, dependency management with shaded JARs, and handling Hadoop class path conflicts.
Groovy - Strong scripting skills for configuration DSLs, row processing, and partition logic in declarative ETL framework.
Git and version controls - Proficient with branching strategies, pull request workflows, code reviews and Jira -integrated commit discipline.
Scala - Functional programming experience for spark development with ability to write concise idiomatic Scala for data processing at scale.
SQL - Ability to write complex SQL queries across multiple databases (MySQL, PG, Oracle, SQL server etc.)
Data Engineering
ETL Pipeline Development - Proven track record building production ETL framework with reusable components. Deep understanding of ETL concepts, load patterns, CDC processing, and data lineage.
Apache Spark - Deep expertise with spark on production scale datasets. Must-Have have hands on experience optimizing spark jobs, tuning memory/shuffle and working with DataFrame/RDDs in Scala or Java.
Kafka/ Golden Gate streaming - Real/Near real-time data processing with Kafka, Golden gate or spark streaming.
Airflow - Experience migrating from Oozie to modern workflow orchestration
Databricks - Exposure to unified analytics platforms, Delta Lake and notebook driven development.
Big Data and Distributed systems
Hadoop Ecosystem - Strong production scale experience with HDFS operations, Hive query optimization and managing portion tables at scale. Understanding of Data locality, replication factors and cluster resource management.
Hive/HQL - Experience in complex analytics queries on multi - million row tables.
Oozie - Proficient in designing coordinator/workflow orchestration with dataset dependency, SLA monitoring and handling late - data scenarios.
Sqoop - Hands-on on experience Configuring bulk imports with parallel rappers, split by strategies, incremental modes and handling large scale database exports
Cloud and Infrastructure
AWS - Production experience with S3 data lakes (lifecycle policies, versioning, encryption), EMR cluster operations, IAM role-based access, and Secrets Manager for credentials management. Understanding of AWS networking, VPC and security groups.
Data Security and compliance - Hands-on on experience implementing PII encryption/ decryption, SafeNet, Data masking, and working on regulated environment.
Good-to-Have Skills
Banking, lending or other regulated financial services experience
Data modeling
Observability and monitoring frameworks for data pipelines
Databricks Certified Data Engineer Professional
Presto/Trino
NoSQL databases
Soft Skills
Strong problem solving and communication skills with ability to explain data engineering concepts to non-technical stakeholders.
Experience working on client facing roles.
Location and Work Arrangements
San Francisco Bay Area
This position requires onshore presence in our San Francisco office.
Responsibilities
Manage data engineering projects. Provide strategic guidance. Oversee data engineering team. Ensure continuous improvement of data processes.
Qualifications
Graduate in Computer Science, Data Science, or related field. 3-4 years of experience in data engineering or a related field.
About EXL Service
EXL Service is a leading operations management and analytics company that helps businesses enhance growth and profitability. The company provides services in areas such as finance and accounting, customer service, and healthcare. EXL Service was founded in 1999 and is headquartered in New York, New York.