5-7 years of experience in software development, preferably in data engineering.
Strong proficiency in Python, with a focus on writing clean and maintainable code.
Familiarity with Apache Spark/PySpark or strong SQL skills with a willingness to learn Spark.
Basic understanding of Git workflows, including branching and merging.
Experience in debugging and troubleshooting code effectively.
Responsibilities
Write and maintain PySpark transforms for data models in the silver and gold layers.
Utilize Lakeflow Declarative Pipelines for change-data-capture merges.
Query and validate data in Unity Catalog across multiple environments using SQL.
Debug failed pipeline runs by analyzing logs and fixing data quality issues.
Maintain reference tables and utility scripts for platform operations.
Develop and update unit/integration tests for transformation logic.
Participate in code reviews and keep documentation current.
Benefits
Opportunity for both permanent and contract positions.
Collaborative work environment alongside a senior engineer.
Focus on professional growth into a data engineering specialist role.
Exposure to cutting-edge technologies like Databricks and Apache Spark.
Engagement with financial services data, enhancing industry knowledge.
Full Job Description
About the Role
We are hiring for both permanent and contract positions for this role.
We're looking for a developer to help build and maintain data pipelines on Databricks for a financial services/wealth management data platform. The platform ingests data from multiple source systems, transforms it through a medallion architecture (bronze - silver - gold) using PySpark, and publishes curated data to downstream consumers via file exports and Kafka. You'll work alongside a senior engineer, writing transform code, fixing pipeline issues, and helping keep the environment healthy across dev/qa/uat.
This is a good fit for someone with solid Python fundamentals and some Spark/SQL exposure who wants to grow into a data engineering specialist. You won't be expected to know Databricks internals on day one - you will be expected to learn fast, ask good questions, and write clean code.
What You'll Work On
Writing and maintaining PySpark transforms (hand-written, not a generic framework) for silver and gold layer tables - things like customer, account, and transaction data models.
Working with Lakeflow Declarative Pipelines (Databricks' current pipeline framework - successor to DLT) and Auto CDC / SCD Type 2 patterns for change-data-capture merges.
Querying and validating data in Unity Catalog across environments (dev/qa/uat) using SQL warehouses.
Debugging failed pipeline runs: reading pipeline event logs, tracing bad records, fixing schema drift or data quality issues.
Maintaining reference/lookup tables and small utility scripts (PowerShell/Python) used to operate the platform.
Writing and updating unit/integration tests for transform logic.
Participating in code review, using Git feature branches and merge requests (GitLab).
Keeping documentation current when you change how something works.
Comfortable reading other people's code and stack traces, and debugging methodically (not guess-and-check).
Nice-to-Have Skills
Direct experience with Databricks (notebooks, jobs, clusters, or SQL warehouses).
Familiarity with Delta Lake, medallion architecture (bronze/silver/gold), or CDC/SCD concepts.
Exposure to Azure (this environment runs on Azure Databricks with Azure AD service principal auth).
Experience with Kafka or other streaming/event systems.
Experience with CI/CD pipelines (this project uses GitLab CI).
Prior work in financial services or a regulated data environment.
What Success Looks Like in the First 90 Days
Can independently pick up a small transform bug or enhancement ticket, make the change, test it, and open a merge request.
Comfortable running existing operational scripts to check pipeline status, query tables, and diagnose failures without hand-holding.
Starting to take ownership of the Databricks environments and help write new transforms
$110,000 - $160,000 a year
About Intelliware
Intelliware Development is a software development company that provides custom software solutions to businesses across a range of industries. The company specializes in agile development methodologies, and works closely with clients to understand their needs and develop software that meets those needs. Intelliware has expertise in a range of technologies, including web and mobile development, cloud computing, and data analytics. The company has a strong focus on innovation, and works to stay at the forefront of emerging technologies and trends. Intelliware has been recognized for its excellence in software development, and has won numerous awards for its work.
