Job SummaryWe are seeking a highly skilled Senior Data Engineer with a strong background in data engineering and AI/ML technologies.
The ideal candidate will have hands-on experience with Databricks on AWS and a deep understanding of modern data architectures, vector databases, and agentic AI frameworks.
Key Responsibilities- Manage and develop Databricks workspaces, notebooks, MLflow pipelines, and Unity Catalog configurations.
- Design and implement data transformation logic and vector embedding workflows using Python and SQL.
- Work with Lakehouse AI architectures and integrate advanced search capabilities using vector databases.
- Optimize and deploy vector databases such as FAISS, Chroma, Weaviate, or native Databricks vector search.
- Utilize agentic AI frameworks including LangGraph, CrewAI, OpenAgents, or similar technologies.
- Apply LLMOps practices and tools such as LangSmith, PromptLayer, and Weights & Biases.
- Integrate LLM APIs (e.g., OpenAI, HuggingFace) and manage prompt engineering and memory in agentic systems.
Required Qualifications- Minimum 5 years of experience in data engineering or AI/ML engineering.
- Proficiency in Python and SQL.
- Strong understanding of file formats including Apache Iceberg, Delta Lake, Parquet, and Avro.
- Experience with Databricks on AWS.
- Familiarity with vector search and embedding workflows.
Preferred Qualifications- Experience with Lakehouse AI and agentic AI frameworks.
- Knowledge of LLMOps tools and practices.
- Understanding of LLM APIs and prompt engineering techniques.
Education: Bachelors Degree
