Qualifications

• BS/MS/PhD in Computer Science or a related field.
• 4+ years of industry experience in software development, focusing on data systems and APIs.
• Familiarity with large web crawlers is a plus.
• Strong expertise in large stateful distributed systems and data processing.
• Proficiency in Kubernetes and Infrastructure-as-Code principles.

Responsibilities

• Lead and manage engineering projects focused on data acquisition, including web crawling, data ingestion, and API development.
• Collaborate with cross-functional teams (AI/ML, Infrastructure).
• Design and implement highly scalable distributed systems to efficiently handle unstructured data.
• Develop algorithms for data indexing and advanced search functionalities.
• Build and maintain backend services for data storage, utilizing key-value databases and synchronization methods.
• Deploy solutions within a Kubernetes Infrastructure-as-Code environment, ensuring system reliability through routine checks.
• Conduct experiments on data to analyze and enhance system performance.

Benefits

• Opportunity to work on cutting-edge data acquisition technologies.
• Collaborative environment with AI/ML and Infrastructure teams for holistic project execution.
• Focus on scalability and efficiency in systems design.
• Hands-on experience with advanced data processing algorithms.
• Professional development through challenging project leadership roles.