Job Summary:
We are seeking a highly skilled AI Engineer to design, develop, and support advanced AI-enabled data solutions on the Databricks platform. The role requires strong Databricks engineering capabilities, advanced SQL expertise, and hands-on experience with Python and PySpark to build, monitor, optimize, and troubleshoot enterprise data pipelines. The ideal candidate will focus on declarative pipeline development, data quality, production support, and AI integrations, including AI parsing and agent-based capabilities within Databricks. This position emphasizes engineering and analytics rather than dashboard development, with responsibilities involving reliable data flows, pipeline monitoring, performance optimization, and production incident resolution. The position is based in Houston, Texas, and requires onsite work.
Key Responsibilities:
• Design, develop, and productionize AI and machine learning solutions using the Databricks platform and its native AI capabilities.
• Build and maintain robust Databricks data engineering workflows using declarative pipeline techniques, including Lakeflow and Delta Live Tables.
• Develop scalable data pipelines supporting AI, analytics, and downstream reporting use cases.
• Build new pipelines, monitor their execution, ensure data flows reliably, and troubleshoot failures to maintain production stability.
• Perform advanced SQL analysis across complex datasets, including query development, debugging, reconciliation, and root-cause analysis.
• Investigate poorly performing SQL queries and improve execution patterns, logic, joins, and filters.
• Optimize SQL workloads for performance, reliability, and maintainability within Databricks.
• Work with structured and semi-structured data to support enterprise data engineering and analytics requirements.
• Analyze data quality issues and trace defects across source data, pipelines, and SQL logic.
• Monitor pipeline performance and data availability, identify potential failures, and implement corrective actions.
• Support production incidents and troubleshoot issues across AI solutions, Databricks pipelines, and SQL workloads.
• Collaborate with data engineers, analysts, business stakeholders, and subject matter experts to translate requirements into technical solutions.
• Support AI parsing, agent-based capabilities, and Databricks-native AI functionality, including Genie.
• Apply modern software engineering practices, including modular design, version control, testing, code reviews, and deployment discipline.
• Support analytics and reporting teams by ensuring reliable data availability for Databricks dashboards and Power BI integrations.
• Participate in client-facing discussions and communicate technical findings, pipeline issues, and resolutions effectively.
Required Qualifications:
• Bachelor's degree in Computer Science, Data Science, Engineering, or a related technical field.
• Extensive hands-on experience in an AI Engineering, Data/AI Engineering, or closely related technical role using Databricks in enterprise environments.
• Deep expertise in writing, analyzing, optimizing, and debugging complex SQL queries, including CTEs, window functions, and joins.
• Strong hands-on experience with Python and PySpark.
• Experience building declarative data pipelines using Delta Live Tables or Lakeflow.
• Strong knowledge of Delta Lake and enterprise data engineering practices.
• Experience working with structured and semi-structured data.
• Experience with data quality management, pipeline monitoring, and troubleshooting.
• Ability to build and maintain scalable data pipelines and ensure reliable data flows.
• Experience supporting production incidents, performing root-cause analysis, and resolving issues across data pipelines and SQL workloads.
• Experience with version control, testing, code reviews, and deployment processes.
• Strong analytical, problem-solving, and communication skills.
• Ability to collaborate with engineering teams, business stakeholders, and client-facing teams.
Preferred Qualifications:
• Experience with Databricks Mosaic AI capabilities, MLflow, model serving, Vector Search, and Generative AI integrations.
• Knowledge of Databricks Genie and AI agent or parsing capabilities.
• Knowledge of Unity Catalog, Databricks SQL Warehouses, and governance patterns within the Databricks Data Intelligence Platform.
• Experience with medallion architecture, enterprise lakehouse design patterns, and tuning Spark workloads for cost and performance.
• Familiarity with cloud platforms such as Azure, AWS, or GCP and associated enterprise data services.
• Experience supporting Power BI workspace configurations, data connections, and dashboard integrations, particularly where reporting workloads are transitioning to Databricks.
