Job Summary
We are seeking an experienced Data Engineer to design, develop, and optimize cloud-based data solutions and ingestion pipelines for structured and unstructured data. The ideal candidate will have strong expertise in AWS, modern data platforms, database technologies, and big data processing frameworks. This role requires a highly analytical professional who can build scalable data architectures, support AI-driven initiatives, and collaborate effectively with both technical and business stakeholders.
Key Responsibilities
• Design, develop, and maintain scalable data ingestion pipelines for structured and unstructured data.
• Build solutions to support efficient data storage, processing, and retrieval across cloud-based platforms.
• Work with AWS cloud services, Snowflake, Databricks, and various database technologies.
• Design and optimize data models utilizing relational, NoSQL, graph, and vector databases.
• Develop and maintain data processing solutions using Python, SQL, and PySpark.
• Support AI and machine learning initiatives through data preparation and architecture design.
• Implement and maintain CI/CD processes for data engineering and platform deployments.
• Utilize tools such as GitHub, JIRA, and Confluence for development, collaboration, and project management.
• Monitor and optimize platform performance using enterprise monitoring and observability tools.
• Collaborate with business and technical teams to gather requirements and deliver scalable data solutions.
• Apply systems thinking and design thinking principles to simplify processes and improve efficiency.
• Create and maintain technical documentation and knowledge-sharing materials.
Required Qualifications
• Bachelor's degree in engineering, Computer Science, or a related field.
• 8+ years of industry experience in database technologies and cloud-based data platforms.
• Experience building data ingestion pipelines for both structured and unstructured data.
• Hands-on experience with AWS cloud services and cloud-based data stores.
• Experience with NoSQL, Graph, and Vector databases.
• Strong proficiency in Python, SQL, and PySpark.
• Experience with Databricks and Snowflake.
• Experience with source code management and collaboration tools including GitHub, JIRA, and Confluence.
• Hands-on experience with CI/CD processes and tools such as Jenkins and Terraform.
• Experience with monitoring and observability tools such as Splunk and Dynatrace.
• Strong analytical, problem-solving, and organizational skills.
• Excellent communication skills with the ability to work effectively with both technical and business teams.
Preferred Qualifications
• Master's degree in engineering, Computer Science, or a related field.
• Exposure to Machine Learning and Artificial Intelligence concepts.
• Knowledge of Retrieval-Augmented Generation (RAG) architectures.
• Experience working with Large Language Models (LLMs).
• Experience with Vector Datastores and AI-enabled data platforms.
• Experience supporting AI/ML data engineering initiatives.