Data Platform Engineer (DevOps)We're looking for a
Data Platform Engineer (DevOps) to join our TecsysIQ Data & AI team. This is a hybrid role that builds and operates the cloud data platform powering our analytics and AI products on Databricks and AWS.
You will own the infrastructure-as-code and delivery pipelines that provision and deploy the platform, and the Databricks jobs and streaming pipelines that move and shape data through it. You'll partner closely with Data Platform Developers, application dev teams, and Product Owners - standing up new customer environments, evolving the CI/CD and release process, and creating and maintaining the data pipelines that keep the platform running reliably at scale. If you are equally comfortable writing Terraform and debugging a Spark Structured Streaming job, this role is for you.
ResponsibilitiesPlatform & DevOps- Provision and evolve AWS and Databricks accounts and workspaces using Terraform across multiple regions and environments, managing remote state and per-environment configuration.
- Own the Helm charts and Kubernetes Jobs that onboard new customer environments and deploy the application layer onto provisioned workspaces.
- Build and maintain Docker images and the Java/Maven build pipeline, publishing artifacts to internal artifact and container registries.
- Integrate build, test, and deployment into CI/CD pipelines, and improve release management and versioning across interdependent repositories.
- Manage secrets and credentials safely through cloud secret management and Kubernetes - never in source control.
Data Platform & Pipelines- Design, build, and maintain Databricks jobs and pipelines - including data curation, transformation, and initial bulk-load workflows.
- Develop and operate real-time streaming and change data capture (CDC) pipelines across the ingestion and transformation layers using Spark Structured Streaming.
- Build and evolve end-to-end data movement across the Bronze, Silver, and Gold data layers, extending pipelines to handle new sources and schema changes as the platform grows.
- Develop and maintain reusable pipeline components and frameworks so new data workflows can be onboarded and shipped quickly.
- Tune clusters and SQL warehouses for cost and performance, and manage catalog, schema, and access governance across environments.
- Collaborate with Data Platform Developers, Product Owners, and business stakeholders in an Agile environment to deliver high-quality data products.
RequirementsQualifications- Hands-on experience with Databricks and Apache Spark, including building batch or streaming data pipelines (mandatory).
- Strong Terraform skills and comfort managing cloud infrastructure on AWS.
- Working knowledge of Kubernetes and Helm, and containerization with Docker.
- Proficiency in Python (PySpark) and/or Java, plus strong SQL for data validation and troubleshooting.
- Experience with Git-based workflows and CI/CD pipelines in a multi-repository codebase.
- Understanding of data quality, ETL/ELT, and data observability concepts.
- Experience working in Agile teams alongside Data Platform Developers and Product Owners.
- Strong analytical, problem-solving, and communication skills.
We understand that experience comes in many forms and that careers are not always linear. If you don't meet every requirement in this posting, we still encourage you to apply.
