Python Big Data Engineer

Compunnel

• $120K — $150K *
Reston, VA 20191In-Person
Information Technology
8 - 10 years of experience
Today
Be an Early Applicant
Job Overview by Ladders

Qualifications

  • 10+ years of hands-on Python development experience for big data applications.
  • Extensive experience with Hadoop, Amazon EMR, Spark, and Hive for data processing.
  • 5+ years of AWS service experience (e.g., EMR, Lambda, Redshift).
  • 5+ years in API integration, SQL query development, and data analysis techniques.
  • Strong skills in data manipulation and visualization from large datasets.
  • Experience in developing CI/CD pipelines using GitLab or Terraform.
  • Effective collaboration skills within cross-functional teams.

Responsibilities

  • Design and maintain scalable big data applications in Python.
  • Optimize data processing pipelines with Hadoop, EMR, and Spark.
  • Develop cloud-native solutions leveraging AWS services.
  • Create data ingestion and transformation workflows.
  • Maintain integrations using APIs and SQL queries.
  • Manipulate and visualize large datasets for analysis.
  • Implement CI/CD pipelines for automated deployment.

Benefits

  • Health, dental, and vision insurance
  • Flexible working hours and remote work options
  • Professional development opportunities
  • Access to cutting-edge technology and tools
  • Collaborative work environment with cross-functional teams
Full Job Description
Job Summary

We are seeking an experienced Python Big Data Engineer with expertise in designing, developing, and optimizing large-scale data processing solutions. The ideal candidate will have extensive experience with Python, AWS cloud services, big data technologies, and data engineering best practices to build scalable, high-performance data pipelines and analytics solutions.

Key Responsibilities
• Design, develop, and maintain scalable big data applications using Python.
• Build and optimize data processing pipelines using Hadoop, EMR, Spark, and Hive.
• Develop and support cloud-native data solutions using AWS services.
• Design and implement data ingestion, transformation, and processing workflows.
• Develop and maintain integrations using APIs and SQL queries.
• Perform large-scale data manipulation, transformation, analysis, and visualization.
• Analyze large datasets to extract meaningful business insights.
• Design, implement, and support CI/CD pipelines for automated deployment.
• Collaborate with cross-functional teams to deliver scalable and reliable data solutions.
• Apply analytical and problem-solving skills to identify and resolve complex technical challenges.

Required Qualifications
• 10+ years of hands-on Python development experience for big data applications.
• Extensive experience implementing scalable data processing pipelines using Hadoop, Amazon EMR, Spark, and Hive.
• 5+ years of development experience with AWS services including EMR, Lambda, EventBridge, Step Functions, Redshift, Amazon S3, and AWS Glue.
• 5+ years of experience developing solutions involving API integrations, SQL queries, Spark, data manipulation, data transformation, analysis, and visualization.
• Strong experience analyzing large datasets and extracting meaningful insights.
• Hands-on experience implementing CI/CD pipelines using GitLab, Terraform, or similar tools.
• Strong analytical, critical thinking, and problem-solving skills.
• Ability to collaborate effectively across cross-functional and matrixed teams.

Preferred Qualifications
• Experience with infrastructure as code (IaC).
• Experience optimizing large-scale cloud-based data engineering solutions.
• Experience working in Agile development environments.
* Ladders Estimates

Similar Jobs

Ad banner background

Get Ready For Your
Next Interview

More Jobs at Compunnel

More Information Technology Jobs

Find similar Python Big Data Engineer jobs:

NationwideReston, VA