IBM

AWS Databricks Lead

IBM$120K — $145K *
Enterprise Technology
Less than 5 years of experience
Job Overview by Ladders

Qualifications

  • 5-7 years of hands-on experience with AWS cloud services and Databricks features
  • Strong proficiency in Python and PySpark for data engineering
  • Knowledge of job cost optimization techniques
  • Experience with SQL development and data mart creation
  • Familiarity with CI/CD pipelines and tools like Terraform
  • Proficient in version control systems, especially GitHub
  • Understanding of Agile methodology and software development life cycle

Responsibilities

  • Collaborate with stakeholders to identify challenges and develop Databricks solutions
  • Research and evaluate new Databricks features and document best practices
  • Support Spark-related issues and conduct code reviews
  • Lead and mentor junior team members in development activities
  • Oversee project delivery for mid-sized teams and ensure quality
  • Participate in solution design discussions and create design documentation
  • Drive performance tuning and job cost optimization initiatives

Benefits

  • Opportunities for professional development and mentorship
  • Collaborative and Agile work environment
  • Engagement in innovative projects with cutting-edge technology
  • Potential for career advancement within the company
  • Access to ongoing training and upskilling opportunities
Full Job Description
Role description

AWS services and databrick

Location Hartford CT onsite

Proficient in leveraging Databricks features such as

oDelta Lake

oUnity Catalog

oScalable data processing frameworks

Strong understanding of AWS cloud services including

oAmazon S3

oAWS Lambda

oAmazon SNS

oAWS Step Functions

oAWS Glue

Skilled in job cost optimization techniques to

oMinimize runtimes

oImprove system performance

Experience with

oPython core libraries

oPySpark for data engineering tasks

Good analytical and problemsolving skills with expertise in

oSQL query development

oStored procedures

oCreating data marts and views in RDBMS

Strong understanding of

oETL concepts

oEndtoend data pipeline architecture

oData warehouse and reporting layers

Familiarity with CICD pipelines and deployment tools such as

oTerraform

Experience with version control systems

oGitHub or equivalent platforms

Understanding of

oAgile methodology

oSoftware Development Life Cycle SDLC

Excellent communication and presentation skills with the ability to collaborate effectively with stakeholders

Roles Responsibilities

Collaborate with business users and stakeholders to understand challenges and provide effective Databricksbased solutions

Research and evaluate new Databricks features conduct Proof of Concepts POCs and develop best practice documentation

Provide support for Sparkrelated issues and participate in code reviews offering recommendations for improvement

Lead and mentor junior team members guiding them in development and troubleshooting activities

Manage midsized teams and oversee delivery of work ensuring quality and timely completion

Take ownership of assigned tasks and contribute effectively in an Agile environment

Participate in solution design discussions and prepare designdocumentation artifacts based on requirements

Drive job cost optimization and performance tuning initiatives to enhance system efficiency

Support CICD processes and infrastructure automation to streamline deployments

Actively participate in Agile ceremonies and contribute to continuous improvement of team collaboration and productivity

About IBM

Sequent Computer Systems designs and manufactures multiprocessing computer systems based on Cache Coherent Non-Uniform Memory Access architecture, which are used primarily as application and database servers for commercial applications. Through a partnership with Oracle Corporation, Sequent became a dominant high-end UNIX platform in the late 1980s and early 1990s. Later it introduced a next-generation high-end platform for UNIX and Windows NT based on non-uniform memory access architecture, NUMA-Q. Sequent Computer Systems, Inc. was incorporated in 1983 and is based in Beaverton, Oregon. In July 1999 Sequent Computer Systems, Inc. was acquired by International Business Machines Corp.

IBM Careers

Joining IBM presents a prime opportunity to be part of our global team of professionals, leading the way in technological innovation and business solutions. At IBM, we offer more than just job opportunities; we provide a platform for growth, leadership development, and a chance to be at the forefront of industry innovation. Work You’ll Do At IBM, your work impacts global markets and helps reshape industries. As part of our team, you will contribute to projects that harness the power of cloud computing, AI, and blockchain technologies. With IBM, you are positioned to lead in the marketplace, leveraging our deep industry expertise and commitment to digital innovation. Transform Your Career IBM is not just a company; it's a culture of innovation and leadership. We are committed to diversity and providing an inclusive environment where all professionals can thrive. Joining IBM means being part of a team that values your unique skills and perspectives. IBM offers a variety of career paths, including full-time positions and internships, that allow you to explore your interests and develop new skills. Our professional development programs are designed to help you grow at every stage of your career, featuring robust training, certification support, and opportunities for advancement. Innovate with Us Engage in work that matters with a team of over 350,000 IBMers worldwide, driving progress in over 170 countries. At IBM, innovation isn’t just about technology, it’s about transforming how businesses operate and compete. Your work will deliver solutions that anticipate and fulfill the needs of our clients in an ever-evolving landscape. Be Part of a Great Team IBMers are diverse, talented, and globally connected. You’ll collaborate with thought leaders and industry experts who are shaping the future of technology. Our culture fosters creativity, teamwork, and the continuous pursuit of excellence. Networking and Professional Growth IBM is deeply invested in the professional growth of its employees. We encourage networking within the company to foster connections that can lead to greater innovation and career advancement. Our leadership is committed to providing every employee with the tools they need to succeed, from mentorship programs to diversity training. Explore Job Opportunities Whether you’re looking for an internship, a graduate role, or a leadership position, IBM offers a range of career opportunities. Our hiring process is designed to be transparent and fair, providing you with all the resources you need to succeed in your interview and resume preparation. Stay Connected Join Our Team Discover the various positions available that match your skills and interests. At IBM, we look for passionate, curious, and solution-driven team players. Explore our open positions and find where you can make a difference. Keep Up to Date Stay informed with career tips, insider perspectives, and industry-leading insights you can put to use today—all from the people who work here. Job Alert Emails Customize your subscription to receive job alerts, the latest news, and insider tips tailored to your preferences. See what exciting and rewarding opportunities await at IBM. Join IBM and be part of a legacy of leadership and innovation. Shape your future and build a career designed to last.
Learn more about IBM
Size
307,600 employees
Market Cap
$128 billion
Industry
Net Income
$5.5 billion
Founded
1911
5 Year Trend
-6.4%
Revenue
$73.6 billion
NASDAQ

Similar Jobs

More Jobs at IBM

More Enterprise Technology Jobs

Find similar AWS Databricks Lead jobs: