Qualifications

• Experience in developing scalable vectorization pipelines and embedding models.
• Proficiency in managing and optimizing vector databases such as Pinecone or Weaviate.
• Strong understanding of advanced chunking and parsing techniques for data retrieval.
• Familiarity with hybrid search workflows that combine dense vector and sparse keyword searches.
• Knowledge of evaluation frameworks and performance metrics in retrieval systems.

Responsibilities

• Lead the design and development of pipelines for converting diverse data formats into embeddings.
• Manage and optimize vector databases, creating efficient indexing strategies.
• Develop chunking and parsing techniques to enhance context retention and retrieval accuracy.
• Build and refine hybrid search workflows integrating different search methodologies.
• Create frameworks to evaluate retrieval performance using various metrics.
• Implement re-ranking processes to improve the accuracy of LLM outputs.
• Collaborate with engineering and product teams to develop high-performance retrieval systems.

Benefits

• Flexible working hours and the possibility of remote work.
• Access to professional development and training opportunities.
• Supportive team culture that encourages innovation and collaboration.
• Health and wellness programs available to employees.