Role descriptionJob DescriptionLead Data EngineerWhat is in it for you?
Data Engineer is responsible for building scalable and high-performance data pipelines, ETL workflows, and data warehouse solutions using Databricks, PySpark, Python, SQL, and cloud technologies.
Responsibilities: -- Design, develop, and maintain scalable and high-performance ETL pipelines and data processing workflows.
- Build and optimize data ingestion frameworks for processing large volumes of structured and unstructured data.
- Perform data preprocessing, transformation, and analytics using Python and PySpark.
- Develop and manage ETL processing pipelines using AWS/Azure Databricks.
- Build complex SQL queries and perform data analysis to support business and reporting requirements.
- Design and implement data warehouse solutions to support enterprise analytics and reporting.
- Work closely with cross-functional teams to understand data requirements and develop scalable solutions.
- Develop reliable, efficient, and reusable data processing frameworks that support evolving business needs.
- Support data validation, quality checks, performance tuning, and optimization of ETL processes.
- Work in a fast-paced environment to deliver innovative data engineering solutions aligned with business objectives.
- Create reports and visualizations using Tableau to support data-driven decision-making.
Experience: -Location: -Educational Qualifications: -- Engineering Degree - BE/ME/BTech/MTech/BSc/MSc.
- Technical certification in multiple technologies is desirable.
Skills: -Mandatory skills- Strong experience with AWS/Azure Databricks for developing and managing large-scale data processing and ETL pipelines.
- Hands-on expertise in Spark (PySpark), Spark SQL, Python, SQL, ETL development, and Data Warehouse concepts.
- Proven experience building scalable, high-performance data processing frameworks and ETL workflows.
- Strong ability to develop complex SQL queries and perform detailed data analysis.
- Experience in data ingestion, data transformation, and data pipeline development.
- Hands-on experience working with cloud platforms such as AWS or Azure.
- Strong understanding of data engineering best practices, performance optimization, and scalable architecture design.
- Excellent visual, verbal, and stakeholder communication skills.
Good to Have Skills- Knowledge of ETL concepts, Data Ingestion, and Shell Scripting (DOS/Bash).
- Experience building ETL processing pipelines using AWS Databricks or Azure Databricks.
- Experience with Tableau for reporting and data visualization.
- Exposure to AWS cloud services and data engineering ecosystems.
- Experience working in Agile and fast-paced delivery environments.
