6-10 years of experience in data engineering and AI extraction solutions.
Strong hands-on experience with Databricks for AI-based information extraction.
Expertise in document parsing, OCR, and entity extraction.
Proficient in using Large Language Models for document classification and summarization.
Solid knowledge of Python for data processing and API integration.
Experience with Apache Spark and Delta Lake for data management and processing.
Ability to lead technical design and implementation efforts.
Responsibilities
Design and implement AI extraction solutions on Databricks for various document types.
Develop workflows to ingest, preprocess, and validate document content.
Utilize Large Language Models to extract and structure information from enterprise documents.
Create effective prompts and structured output schemas for data extraction.
Integrate Databricks solutions with document repositories and enterprise applications.
Troubleshoot and resolve pipeline issues using monitoring and validation techniques.
Provide technical leadership and guidance to engineering teams.
Benefits
Opportunity to work with cutting-edge AI technologies.
Collaborative work environment with a focus on innovation.
Professional development and technical leadership opportunities.
Access to advanced tools and platforms for data engineering.
Flexible work arrangements and potential for remote work.
Full Job Description
Role description
Role Senior Data Engineer AI Extraction Lead Databricks
Experience 6 to 10 Years
Location Atlanta GA Preferred
Technical Skills Primary
Databricks AI Extraction Strong handson experience designing and implementing AIbased information extraction solutions on Databricks for unstructured and semistructured content including PDFs scanned documents emails images forms tables and free text
Document AI and Information Extraction Handson expertise in document parsing OCR document classification field extraction table extraction entity extraction and conversion of document content into structured data
LLMBased Extraction Experience using Large Language Models and Generative AI capabilities to extract classify summarize and structure information from enterprise documents and other content sources
Prompt Engineering and Structured Outputs Ability to design effective prompts define structured output schemas generate reliable JSON responses and apply validation rules to improve extraction accuracy and consistency
Databricks Lakehouse Platform Strong working knowledge of Databricks notebooks WorkflowsJobs Apache Spark PySpark Delta Lake SQL and Unity Catalog
Extraction Pipeline Development Experience building Databricks workflows to ingest documents preprocess content execute AI extraction validate output and persist structured results in Delta Lake or other target systems
Python Development Strong working knowledge of Python including data processing reusable functions JSON handling API integration exception handling logging validation and development of preprocessing and postprocessing components
Apache Spark and PySpark Handson experience using Apache Spark and PySpark for distributed data processing data transformation enrichment validation and largescale extraction workloads
Delta Lake Experience using Delta Lake for reliable storage of extracted data including ACID transactions schema management data versioning incremental processing and performance optimization
Unity Catalog Experience using Unity Catalog for data governance access control lineage metadata management auditability and secure access to source documents and extracted data
API and System Integration Experience integrating Databricks extraction solutions with document repositories enterprise applications APIs operational systems and downstream dataconsumption platforms
Troubleshooting and Observability Ability to troubleshoot extraction failures and Databricks pipeline issues using appropriate monitoring logging validation errorhandling and retry mechanisms
Technical Leadership Ability to lead solution design and implementation establish engineering standards conduct code reviews document implemented solutions and provide technical guidance to engineering teams
Technical Skills Secondary Nice to Have NOT MANDATORY
Optical Character Recognition Platforms Exposure to OCR and documentintelligence services such as Azure AI Document Intelligence AWS Textract Google Document AI Tesseract or equivalent technologies
Databricks GenAI Capabilities Experience with Databricks Model Serving Mosaic AI Vector Search Foundation Model APIs MLflow or related Databricks AI capabilities
RetrievalAugmented Generation Familiarity with embeddings vector databases semantic search chunking strategies and RAGbased documentprocessing solutions
CICD and DevOps Experience with Gitbased development automated testing deployment pipelines environment management Databricks Asset Bundles and CICD practices
Agile Delivery Experience working within Agile or Scrum delivery models and coordinating work across onsite and offshore teams
Soft Skills
Excellent written verbal and stakeholder communication skills
Strong leadership and mentoring capabilities
Strong problemsolving and analytical skills
Ability to translate business requirements into scalable technical solutions