Technical Lead - Databricks Data Engineering (Onshore)
Experience 10-15 years
Primary Responsibilities Lead the technical delivery of the AWS Databricks data modernization program across data ingestion, Bronze, Silver, Gold, and Semantic Layer workstreams. Review and assess the existing data platform, including CDC, AWS S3 Landing Zone, Bronze and Silver layers, and Databricks architecture. Provide technical leadership for data engineering activities and ensure alignment with enterprise architecture and data modeling standards. Design and review scalable data pipelines, transformation frameworks, and data processing patterns using Databricks and Spark.Validate existing ingestion, transformation, deduplication, standardization, and reconciliation processes, and recommend improvements where required. Collaborate closely with Enterprise Architects, Lead Data Modelers, Business Analysts, and client technical teams to translate business requirements into scalable technical solutions.Define engineering standards, coding guidelines, performance optimization strategies, and reusable development frameworks.Lead technical design reviews, code reviews, and solution walkthroughs with the offshore development team.Support enterprise data model implementation by ensuring data engineering solutions align with conceptual, logical, and physical data models.Oversee Gold layer implementation, data lineage, metadata integration, and semantic layer enablement.
Ensure data quality, security, scalability, monitoring, and operational readiness of the data platform.Troubleshoot complex technical issues and provide guidance on performance tuning, optimization, and best practices.Mentor and provide technical direction to offshore Data Engineers throughout the project lifecycle.
Mandatory Skills
AWS Databricks, Apache Spark (PySpark/Scala), Delta Lake, SQL, Python, AWS (S3, IAM, Lambda, Glue), Data Lakehouse Architecture, ETL/ELT Design, Data Pipeline Development, Performance Tuning, Data Engineering Best Practices, CI/CD, Git, Code Review, Solution Design.
Preferred Skills
Unity Catalog, Delta Live Tables (DLT), Databricks Workflows, Structured Streaming, Change Data Capture (CDC), Terraform, Infrastructure as Code (IaC), Informatica, SSIS Modernization, Azure DevOps/Jira, Power BI integration.
Domain Expertise
