Cognizant

AWS Pyspark Tech Lead

Cognizant$120K — $135K *
Information Technology
8 - 10 years of experience
Job Overview by Ladders

Qualifications

  • 8-12 years of experience in data engineering and technical leadership
  • Strong expertise in AWS Glue for ETL and data pipeline development
  • Hands-on experience optimizing Apache Spark and PySpark workloads
  • Proficiency in Python and SQL, and solid data modeling skills
  • Experience managing data storage solutions in Amazon S3
  • Strong understanding of AWS DevOps practices and automation frameworks
  • Proven ability to create scalable, high-performance data solutions

Responsibilities

  • Design and develop scalable ETL pipelines utilizing AWS Glue and PySpark
  • Automate data extraction, transformation, and loading processes
  • Build and maintain efficient data models and transformation frameworks
  • Optimize performance of Apache Spark workloads for efficiency and scalability
  • Manage the AWS Glue Catalog for structured and discoverable metadata
  • Implement AWS DevOps practices for streamlined operations
  • Ensure data security and compliance standards are met

Benefits

  • Comprehensive health insurance (medical, dental, vision)
  • 401(k) with company match and life/disability insurance
  • Generous PTO, paid holidays, and parental leave
  • Access to certifications, training programs, and advancement opportunities
  • Flexible hybrid work model for work-life balance
  • Wellness initiatives and employee recognition programs
Full Job Description
a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; }

Senior Data Engineer / Technical Lead - AWS Glue & Spark Optimization

Location: Remote
Experience: 8-12 years
Role Overview

We are seeking a highly skilled Senior Data Engineer / Technical Lead with strong expertise in AWS data services and Spark optimization.

In this role, you will design, optimize, and manage scalable data pipelines using AWS Glue, PySpark, and Amazon S3, while driving DevOps practices and ensuring high-performance data processing solutions that power analytics and business insights.
Key Responsibilities
Data Engineering & Pipeline Development
  • Design and develop scalable ETL pipelines using AWS Glue, PySpark, and AWS Glue Studio
  • Automate data extraction, transformation, and loading processes for improved data accessibility and usability
  • Build and maintain efficient data models and transformation frameworks
Spark & Performance Optimization
  • Optimize Apache Spark and PySpark workloads for performance, scalability, and cost efficiency
  • Implement tuning strategies for large-scale distributed data processing
AWS Platform & Data Management
  • Manage AWS Glue Catalog to maintain structured and discoverable metadata
  • Oversee Amazon S3 storage solutions ensuring secure, scalable, and efficient data management
  • Implement best practices for data lifecycle management and storage optimization
DevOps & Automation
  • Apply AWS DevOps practices to streamline CI/CD, deployment, and operations
  • Enable automation for pipeline orchestration, monitoring, and releases
  • Ensure integration with enterprise systems and tools
Governance, Security & Compliance
  • Ensure adherence to data security, privacy, and compliance standards
  • Implement access controls, auditing, and governance policies across data platforms
Collaboration & Leadership
  • Provide technical guidance and mentorship to engineering teams
  • Foster a culture of collaboration, knowledge sharing, and continuous improvement
  • Partner with stakeholders to translate business needs into scalable technical solutions
Documentation & Quality
  • Maintain comprehensive documentation of data processes, pipelines, and architectures
  • Support transparency, troubleshooting, and long-term knowledge retention
Required Qualifications
  • Strong experience in AWS Glue (ETL, Studio, Catalog) and data pipeline development
  • Hands-on expertise in Apache Spark / PySpark optimization
  • Proficiency in Python, SQL, and data modeling concepts
  • Experience managing data storage in Amazon S3
  • Solid understanding of AWS DevOps practices and automation frameworks
  • Proven ability to build scalable, high-performance data solutions
  • Strong analytical and problem-solving skills
Preferred Qualifications
  • Experience with AWS Data Pipeline or modern orchestration frameworks
  • Familiarity with CI/CD and configuration management tools
  • Knowledge of data governance and regulatory compliance frameworks
  • Strong background in root cause analysis and defect management
  • Experience working in enterprise-scale, hybrid environments
Leadership & Professional Skills
  • Strong communication and stakeholder engagement capabilities
  • Ability to drive execution excellence and innovation
  • Demonstrated accountability and results orientation
  • Collaborative mindset with ability to build trust and influence teams
  • Adaptability to drive change and continuous improvement
Compensation
  • Base Salary Range: $120,000 - $135,000 per year
  • Final compensation will be based on experience, skills, and location
  • Eligible for performance-based bonuses and incentive compensation
Benefits

Cognizant offers a comprehensive benefits package designed to support your well-being and career growth:
  • Health & Wellness: Medical, dental, and vision insurance
  • Financial Security: 401(k) with company match, life and disability insurance
  • Time Off: Generous PTO, paid holidays, and parental leave
  • Learning & Development: Access to certifications, training programs, and career advancement opportunities
  • Flexible Work Model: Hybrid setup supporting work-life balance
  • Employee Programs: Wellness initiatives, employee resource groups, and recognition programs

About Cognizant

TriZetto is Powering Integrated Healthcare Management. With technology solutions touching more than half the U.S. population today, TriZetto is uniquely positioned to drive the convergence of core benefit administration, care management and constituent engagement. TriZetto provides premier information technology solutions that enable payers and other constituents in the healthcare supply chain to improve the coordination of benefits and care for healthcare consumers.

Cognizant Careers

Join the vibrant team at Cognizant, a leading provider of information technology, consulting, and business process services. Cognizant is not just a company; it's a community where innovation, leadership, and diversity are valued and where every team member is encouraged to thrive.

Work You’ll Do

At Cognizant, we are dedicated to helping the world's leading companies build stronger businesses — not just for today but for the future. This commitment to growth and sustainability is what makes Cognizant a unique place to advance your career.

Explore Job Opportunities

Whether you're looking for an entry-level position or a more senior role, Cognizant offers a plethora of job opportunities that could be perfect for you. Our hiring process is designed to be transparent and engaging, ensuring that every candidate is able to showcase their skills and potential fully.

Innovative Work

Join a team where innovation is at the core of everything we do. Cognizant’s professionals are not just part of a company; they're part of a groundbreaking environment that combines technology, insights, and leadership to foster innovation across global industries.

Internship Programs

Kickstart your career with Cognizant’s internship programs. These opportunities allow you to gain hands-on experience, develop your professional skills, and network with experts in your field. An internship at Cognizant is a stepping stone to full-time employment and a chance to see how your studies apply in the real world.

Professional Growth and Development

Cognizant is committed to the professional growth of its employees. With access to cutting-edge training, leadership programs, and diversity initiatives, you can build a career that’s equipped for the demands of tomorrow’s business landscape.

Benefits and Culture

Our employees enjoy a range of benefits designed to support their physical, financial, and emotional well-being. Cognizant’s culture is built on a foundation of respect and inclusivity, where everyone’s contribution is valued. Join us and be part of a team that celebrates diversity and is driven to make a positive impact on the world.

Stay Connected

Join Our Team Search open positions that match your skills and interests. We look for passionate, curious, creative, and solution-driven team players. Ready to take the next step in your career journey? Explore the exciting employment and career opportunities waiting for you at Cognizant.

Keep Up to Date

Stay ahead with career tips, insider perspectives, and industry-leading insights you can put to use today—all from the people who work here.

Job Alert Emails

Personalize your subscription to receive job alerts, latest news, and insider tips tailored to your preferences. Discover the rewarding opportunities that await at Cognizant.

Networking and Interviews

Enhance your career prospects at Cognizant through our robust networking events. Prepare your resume, sharpen your interview skills, and connect with leaders across the industry to unlock new career paths. At Cognizant, we don’t just offer jobs; we offer career journeys. Be part of a team that’s committed to your long-term career growth and professional development. Join Cognizant today and be a part of our success story.
Learn more about Cognizant
Size
340,400 employees
Market Cap
$28.7 billion
Industry
Net Income
$1.3 billion
Founded
1994
5 Year Trend
+6.5%
Revenue
$16.6 billion
NASDAQ

Similar Jobs

More Jobs at Cognizant

More Information Technology Jobs

Find similar AWS Pyspark Tech Lead jobs: