Role description
JAVA + APACHE SPARK DEVELOPER
Snowflake / Databricks | Enterprise Data Engineering
Experience
5-9 years
Location
New York
Work model
Hybrid: 3 days in office per week
Employment
Full-time / Project deployment
Role Overview
We are seeking a hands-on Java and Apache Spark Developer to build, enhance and support scalable data-processing solutions on modern enterprise data platforms. The role requires strong Java engineering, distributed data-processing expertise and practical experience with either Snowflake or Databricks. The engineer will work across design, development, testing, optimisation, deployment and production support in an Agile environment.
Key Responsibilities
• Design, develop, test and maintain high-volume batch and near-real-time data-processing applications using Java, Apache Spark and Spark SQL.
• Build resilient, reusable and maintainable data pipelines for ingestion, transformation, enrichment and distribution across enterprise systems.
• Develop Java-based services, utilities and integration components that support data workflows and downstream consumers.
• Implement data solutions on Snowflake and/or Databricks, including transformations, curated datasets and consumption-ready data structures.
• Optimise Spark jobs through appropriate partitioning, caching, broadcast strategies, join selection, shuffle reduction, resource tuning and data-skew handling.
• Write complex SQL for transformation, validation, reconciliation and performance-sensitive workloads.
• Partner with architects, business analysts, product owners and engineering teams to translate requirements into practical technical designs.
• Implement data-quality controls, logging, monitoring, ing, lineage and operational support features.
• Create unit, integration, regression and performance tests; support defect resolution, release validation and production deployment.
• Contribute to code reviews, CI/CD pipelines, technical documentation, runbooks and engineering standards.
• Troubleshoot production issues across application code, Spark execution, SQL workloads, data pipelines and platform integrations.
Mandatory Skills and Experience
• 5-7 years of relevant experience in data engineering, backend development or large-scale data-processing applications.
• Strong hands-on programming experience in Java, including object-oriented design, collections, exception handling, multithreading and performance-conscious coding.
• Strong practical experience with Apache Spark using Java, Spark SQL and DataFrame/Dataset APIs.
• Hands-on experience designing and tuning distributed data-processing jobs for scalability, reliability and efficient resource utilisation.
• Hands-on experience with at least one modern data platform: Snowflake or Databricks.
• Strong SQL skills, including complex joins, CTEs, window functions, aggregations, query optimisation and data validation.
• Experience developing ETL/ELT pipelines and working with structured and semi-structured data.
• Experience with Git-based development, code reviews, build tools and CI/CD practices.
• Good understanding of software development lifecycle, Agile delivery, testing and production-support practices.
• Strong analytical, debugging, communication and stakeholder-collaboration skills.
Platform-Specific Expectations
Development
SQL transformations, views, stages, streams/tasks or equivalent platform objects
Notebooks/jobs, Spark workloads, Delta Lake tables and platform workflows
Performance
Warehouse/query optimisation, clustering/partition awareness and workload efficiency
Spark job tuning, partitioning, caching, shuffle optimisation and cluster efficiency
Operations
Loading, monitoring, validation, access controls and production support
Job orchestration, monitoring, data-quality checks, access controls and production support
Preferred Skills
• Experience with Spring Boot, REST APIs or Java-based microservices.
• Exposure to Kafka or another enterprise messaging/streaming platform.
• Experience with cloud services on Microsoft Azure, AWS or Google Cloud Platform.
• Knowledge of Python or PySpark for data engineering and automation.
• Experience with orchestration tools such as Apache Airflow, Azure Data Factory or equivalent.
• Familiarity with data modelling, data governance, lineage, security and regulated-industry controls.
• Exposure to financial services, risk, regulatory or capital-markets data is advantageous.
Technology Landscape
Programming
Java; optional Python
Data processing
Apache Spark, Spark SQL, DataFrame/Dataset APIs
Data platforms
Snowflake and/or Databricks; Delta Lake where applicable
Integration
REST APIs, Kafka or equivalent messaging tools
Engineering toolchain
Git, Maven/Gradle, Jenkins/Azure DevOps or equivalent CI/CD tools
Cloud
Microsoft Azure, AWS or Google Cloud Platform
Ways of working
Agile/Scrum, code reviews, automated testing and production support
Education and Work Arrangement
• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering or a related discipline.
• Must be willing to work from the Mumbai East - Goregaon office at least 3 days per week.
• Must be comfortable collaborating with distributed teams and supporting business-critical applications as required.
