Job Summary
AmeriLife is seeking a Data & AI Engineer to help build, modernize, and scale our enterprise data and artificial intelligence capabilities.
Reporting to the Director of Data Engineering within the Enterprise Data & AI Platforms organization, this individual will play a hands-on engineering role in developing modern data pipelines, integrating enterprise data sources, and driving intelligent automation across the organization.
The role will focus on advancing AmeriLife's Databricks Lakehouse platform, supporting the transition from legacy data environments to scalable, cloud-native architectures, and enabling enterprise analytics, reporting, and automation.
This is a hands-on engineering position requiring exceptional proficiency in Python, PySpark, and SQL, a strong foundation in modern data engineering, and a hyperautomation mindset focused on eliminating manual processes, building reusable engineering frameworks, and improving operational efficiency.
The ideal candidate is a technically curious, execution-oriented engineer who enjoys writing code, solving complex problems, and applying modern technologies to build scalable, automated enterprise data solutions.
Job Description
Key Responsibilities
Enterprise Data Engineering & Databricks
- Design, develop, test, and maintain scalable data ingestion, transformation, and integration pipelines using Databricks, PySpark, Python, and SQL.
- Build and enhance enterprise data pipelines following modern Lakehouse and Medallion Architecture (Bronze, Silver, and Gold) principles.
- Develop reusable, modular, and metadata-driven ETL/ELT frameworks to support enterprise data processing.
- Ingest and integrate structured and semi-structured data from databases, APIs, flat files, cloud storage, and enterprise applications.
- Work with Delta Lake, Unity Catalog, Databricks Workflows, and related platform capabilities.
- Support migration and modernization of legacy SQL Server, SSIS, and traditional data warehouse workloads onto Databricks.
- Optimize data pipelines and transformations for performance, reliability, scalability, and cost efficiency.
- Apply advanced Python, PySpark, and SQL techniques to solve complex data processing and engineering challenges.
- Collaborate with senior engineers and architects to implement enterprise data engineering standards and best practices.
Hyperautomation & Intelligent Engineering
- Apply a hyperautomation-first mindset to identify opportunities for eliminating repetitive manual processes across data engineering and platform operations.
- Develop Python-based automation frameworks, utilities, and reusable engineering components.
- Design and implement metadata-driven and configuration-driven solutions that reduce manual coding and operational dependencies.
- Leverage scripting, APIs, orchestration, and intelligent automation techniques to streamline engineering workflows.
- Explore practical applications of AI-assisted development and intelligent automation to accelerate engineering productivity.
- Continuously identify opportunities to simplify processes, reduce technical debt, and improve engineering efficiency through automation.
Data Integration & Transformation
- Develop efficient data transformation logic using SQL, Python, and PySpark.
- Implement joins, aggregations, business rules, enrichment logic, and incremental processing patterns.
- Work with relational databases, APIs, JSON, Parquet, Delta, and other common enterprise data formats.
- Support dimensional modeling, historical data tracking, and efficient data processing patterns.
- Collaborate with business analysts, data architects, and technical teams to translate requirements into engineering solutions.
- Build scalable and reusable integration components following enterprise engineering standards.
Data Quality, Governance & Reliability
- Implement automated data quality checks covering completeness, accuracy, consistency, uniqueness, and referential integrity.
- Develop reconciliation and validation routines across source systems, transformation layers, and downstream datasets.
- Support enterprise metadata management, data lineage, and cataloging initiatives.
- Apply data governance standards using Unity Catalog and other enterprise governance capabilities.
- Build monitoring, logging, and alerting capabilities for data pipelines and scheduled workloads.
- Troubleshoot pipeline failures, data discrepancies, performance issues, and integration errors.
- Participate in root-cause analysis and implementation of permanent corrective solutions.
- Follow established security, access control, data privacy, and auditability requirements.
Engineering Automation & DevOps
- Develop and maintain engineering code using Git-based version control and collaborative development practices.
- Support automated deployment pipelines using CI/CD frameworks and DevOps tooling.
- Implement unit tests, integration tests, and automated validation for data engineering workloads.
- Create reusable Python utilities, notebooks, SQL scripts, and engineering components.
- Support configuration-driven and metadata-driven pipeline development.
- Contribute to infrastructure automation and environment management practices.
- Document technical designs, data mappings, transformations, deployment procedures, and operational runbooks.
- Continuously identify opportunities to improve engineering productivity through automation and reusable frameworks.
Required Qualifications
- Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical discipline, or equivalent practical experience.
- 2+ years of hands-on experience in data engineering, software engineering, data integration, or related technical roles.
- Very strong Python programming skills, including scripting, modular development, data processing, and automation.
- Very strong SQL proficiency, including complex queries, joins, aggregations, window functions, transformations, and query performance optimization.
- Strong hands-on PySpark experience, including distributed data processing, DataFrame operations, transformations, and performance optimization.
- Experience developing ETL/ELT pipelines and working with relational databases or modern data platforms.
- Working knowledge of Databricks, Apache Spark, Delta Lake, or comparable cloud-based data engineering technologies.
- Understanding of modern data architecture concepts, including data lakes, data warehouses, lakehouses, and data pipelines.
- Experience working with structured data, JSON, CSV, Parquet, APIs, and other common integration formats.
- Familiarity with cloud platforms such as Microsoft Azure or AWS.
- Basic understanding of data modeling, data quality, and data validation practices.
- Experience using Git or comparable source control tools.
- Strong automation and hyperautomation mindset, with demonstrated ability to simplify workflows, eliminate manual tasks, and build reusable engineering solutions.
- Strong analytical, troubleshooting, and problem-solving skills.
- Ability to work independently on assigned engineering tasks while collaborating effectively within a broader technical team.
- Strong communication skills and willingness to learn new technologies.
Preferred Qualifications
- Databricks certification is a plus, particularly Databricks Certified Data Engineer Associate or Professional.
- Hands-on experience with Azure Databricks, Delta Lake, Unity Catalog, and Databricks Workflows.
- Experience with Azure Data Factory, Azure Data Lake Storage, Azure DevOps, or comparable cloud engineering services.
- Experience migrating legacy SQL Server, SSIS, stored procedures, or traditional ETL workloads to modern cloud data platforms.
- Experience building reusable automation frameworks, metadata-driven pipelines, or configuration-driven engineering solutions.
- Familiarity with AI-assisted development tools and practical applications of intelligent automation.
