Job Summary
The ML Engineer will design, develop, and maintain scalable data pipelines and machine learning workflows across Azure and Google Cloud Platform (GCP). The role emphasizes Databricks, Python, PySpark, feature engineering, and model behavior, with a focus on building reliable data solutions that support machine learning applications. The engineer will collaborate with cross-functional teams to modernize data assets, implement automated testing, optimize data workflows, and maintain secure, efficient, and scalable data engineering practices throughout the software development lifecycle.
Key Responsibilities
• Design, build, and maintain scalable, efficient, and reliable data pipelines for data ingestion, transformation, and integration across diverse sources and destinations using Databricks, Kafka, and related technologies.
• Develop and maintain data engineering solutions across Azure and GCP environments.
• Modernize and extend data assets across SQL-based, NoSQL-based, cloud-based, and real-time streaming platforms.
• Develop and manage feature engineering pipelines for machine learning workflows using Vertex AI, BigQuery ML, and custom Python libraries.
• Apply advanced feature engineering techniques and support the evaluation of model behavior.
• Implement automated unit, integration, and performance testing frameworks to ensure data quality, reliability, and compliance with organizational standards.
• Optimize data workflows for performance, cost efficiency, and scalability across large datasets and complex environments.
• Develop and maintain CI/CD pipelines and processes to support reliable software and data solution delivery.
• Use version control systems such as Git to manage code and collaborate with development teams.
• Create and review architectural diagrams, interface specifications, and technical design documentation.
• Take ownership of systems, processes, and technology components while driving features through all phases of the software development lifecycle.
• Collaborate with cross-functional teams to deliver internal and external-facing applications and process improvements.
Required Qualifications
• Bachelor's degree in Computer Science, Management Information Systems, Mathematics, Business Analytics, or another STEM field.
• 4+ years of professional data development experience.
• 4+ years of experience with SQL and NoSQL technologies.
• 3+ years of experience building and maintaining data pipelines and workflows.
• 2+ years of experience developing with Python.
• Experience with feature engineering for machine learning pipelines.
• Experience with CI/CD pipelines and processes.
• Experience implementing automated unit, integration, and performance testing.
• Experience with version control software such as Git.
• Strong understanding of ETL processes and data warehousing concepts.
• Understanding of Agile principles and Scrum methodologies.
Preferred Qualifications
• Experience with Databricks concepts and development.
• Experience with PySpark and Apache Spark development.
• Knowledge of structured streaming technologies, including Spark Structured Streaming, Kafka, Event Hubs, or similar platforms.
• Experience with GCP services or equivalent Databricks capabilities, including BigQuery, Vertex AI Platform, Cloud Storage, AutoMLOps, and Dataflow.
• Experience with GitHub SaaS and GitHub Actions.
• Experience with service-oriented architecture.
