Designation : Consulant.
Location : San jose, California, United States.
Experience : 5 to 10 Years
Job Role : We're looking for a Senior Data Engineer with deep expertise in building and scaling modern data platforms. You'll design, build, and maintain robust data pipelines and infrastructure that power analytics and business-critical applications, with a strong emphasis on the Databricks and AWS ecosystem.
Required Qualifications - 7+ years of experience in Data Engineering roles
- Extensive hands-on experience with Databricks (Delta Lake, Unity Catalog, cluster management, notebooks, job orchestration)
- Strong experience with AWS cloud services (S3, IAM, Glue, EMR, Lambda, Redshift, CloudWatch)
- Proven expertise in Apache Airflow for workflow orchestration and scheduling
- Strong programming skills in Python and SQL
- Solid understanding of Apache Spark (PySpark, Spark SQL, performance tuning)
- Experience with data modeling, warehousing concepts, and Lakehouse architecture
- Familiarity with version control (Git) and CI/CD practices
- Strong understanding of data governance, quality, and lineage principles
- Excellent problem-solving and communication skills
Responsibilities : - Design, develop, and maintain scalable ETL/ELT pipelines using Databricks (PySpark/Spark SQL)
- Build, orchestrate, and monitor workflows using Apache Airflow (DAG design, scheduling, dependency management)
- Architect and manage data infrastructure on AWS (S3, Glue, Lambda, EMR, Redshift, IAM, etc.)
- Optimize data pipelines for performance, reliability, and cost-efficiency
- Implement data quality checks, testing frameworks, and observability/monitoring for pipelines
- Collaborate with Data Scientists, Analysts, and Product teams to understand data requirements
- Design and maintain data models (Lakehouse/Medallion architecture - Bronze/Silver/Gold layers)
- Implement CI/CD pipelines for data engineering workflows
- Ensure data security, access controls, and compliance across the data platform
- Document architecture, pipelines, and processes for team knowledge sharing
- Mentor junior data engineers and contribute to engineering best practices
Required skills : Databricks, AWS, Python, SQL, Pyspark, Genie, GenAI, Analytics, Cybersecurity Knowledge
