Data Engineer - Capital Markets

Sumitomo Mitsui Financial Group, Inc. • $97K — $156K *
New York, NY 10025In-Person
Enterprise Technology
Less than 5 years of experience
Today
Be an Early Applicant
Job Overview by Ladders

Qualifications

  • 4+ years of Databricks design and development experience
  • 3+ years in AWS cloud environments
  • Proficient with CI/CD tools and automation
  • Advanced Python (PySpark) and SQL coding skills
  • Strong grasp of Apache Spark, Delta Lake, and streaming concepts
  • Experience building production-grade ETL/ELT pipelines
  • Relevant industry certifications (Databricks or AWS) preferred

Responsibilities

  • Design and implement data pipelines using the Databricks Medallion Architecture
  • Develop and orchestrate complex data workflows with Databricks Jobs and Delta Live Tables
  • Automate the deployment of Databricks assets and workspace configurations
  • Ensure integration of Databricks with AWS services for secure data access
  • Optimize Spark clusters and Delta Lake storage management
  • Implement data governance and security controls using Unity Catalog
  • Collaborate with teams to provide clean data views for analytics

Benefits

  • Hybrid work model allowing flexible in-office and remote work
  • Opportunity to work with cutting-edge data technologies
  • Collaborative and supportive team environment
  • Access to training and professional development resources
  • Engagement with diverse teams in a financial trading context
Full Job Description


Role Description

We are seeking a Senior Databricks Engineer with deep AWS expertise to build, optimize, and maintain our enterprise data Lakehouse infrastructure. In this role, you will be responsible for designing high-performance data pipelines, implementing regulatory and real time reporting along with advanced analytics environments, and ensuring seamless integration between Databricks and core AWS services to support real-time financial trading data consumption.

Role Objectives

  • Lakehouse Architecture: Design and implement robust data pipelines using the Databricks Medallion Architecture (Bronze, Silver, Gold layers) to process structured and unstructured data.
  • Pipeline Automation: Develop, scale, and orchestrate complex data workflows utilizing Databricks Jobs and Delta Live Tables (DLT).
  • Data Ops & CI/CD Deployment: Standardize and automate the deployment of Databricks assets, workspace configurations, and code pipelines across Dev, QA, and Production environments.
  • AWS Integration: Ensure seamless data cataloging, storage, and movement across the AWS ecosystem, specifically integrating Databricks with Amazon S3, AWS Glue, and AWS IAM for secure access control.
  • Performance Optimization: Tune Spark clusters, optimize Delta Lake storage (e.g., Z-Ordering, partitioning), and manage compute costs within the AWS environment.
  • Data Governance & Security: Implement fine-grained data access controls, data lineage, and auditing using Unity Catalog or native cloud security controls.
  • Collaboration: Partner with Data Scientists, Risk Managers, and downstream analytics teams to deliver clean, business-ready data views for reporting and AI modeling.


Qualifications and Skills

  • 4+ years of hands-on experience in architectural design and development within the Databricks platform, alongside 3+ years working in an AWS cloud environment.
  • CI/CD & DevOps Tooling: Proven proficiency in automated deployments using Databricks Asset Bundles (DABs), Terraform (specifically the Databricks and AWS providers), and standard Git pipelines (e.g., GitHub Actions, GitLab CI/CD, or AWS CodePipeline).
  • Technical Proficiency: Advanced programming skills in Python (PySpark) and SQL for complex data manipulation and transformation.
  • Core Concepts: Strong understanding of Apache Spark internals, Delta Lake mechanics, and streaming data concepts (e.g., interacting with Amazon MSK or Kafka data streams).
  • Data Engineering Stack: Proven experience building production-grade ETL/ELT pipelines, handling data schema validation, and cleansing raw capture feeds.
  • Certifications: Databricks Certified Data Engineer Professional or AWS Certified Data Engineer - Professional is highly advantageous.


Additional Requirements

SMBC's employees participate in a Hybrid workforce model that provides employees with an opportunity to work from home, as well as, from an SMBC office. SMBC requires that employees live within a reasonable commuting distance of their office location. Prospective candidates will learn more about their specific hybrid work schedule during their interview process. Hybrid work may not be permitted for certain roles, including, for example, certain FINRA-registered roles for which in-office attendance for the entire workweek is required.

About Sumitomo Mitsui Financial Group, Inc.

Sumitomo Mitsui Financial Group, Inc. Careers

There has never been a more opportune time to join the dynamic team at Sumitomo Mitsui Financial Group, Inc. (SMFG)—a leading force in the financial services industry recognized for its leadership in innovation and diversity.

Explore Job Opportunities

Sumitomo Mitsui Financial Group, Inc. offers a plethora of job opportunities that cater to a variety of skills and interests. The company is renowned for its commitment to professional growth and leadership development, making it an ideal environment for ambitious individuals looking to advance their careers.

Experience Professional Growth

At SMFG, career advancement is not just a possibility but a priority. The company supports its team members with extensive training programs, including leadership development and diversity training, ensuring that every employee has the tools and knowledge necessary to succeed.

Join a Diverse and Inclusive Team

Diversity and inclusion are at the core of the company culture at Sumitomo Mitsui Financial Group, Inc. With a global team that values unique perspectives and fosters a collaborative and inclusive environment, SMFG is a place where everyone can thrive.

Internship Programs

For those starting their career journey, SMFG offers internship programs that provide a robust foundation in the financial sector. Interns gain invaluable experience, working alongside seasoned professionals and engaging in projects that offer real-world applications of their studies.

Benefits and Culture

Sumitomo Mitsui Financial Group, Inc. is dedicated to not only attracting but also retaining top talent by offering competitive benefits that enhance both personal and professional life. The company culture promotes work-life balance, employee well-being, and continuous learning.

Innovative Work Environment

Innovation is a key driver of SMFG’s success. Employees are encouraged to bring forward-thinking ideas to the table and are provided with the resources to transform these ideas into actionable solutions that drive the financial industry forward.

Networking and Career Development

Networking opportunities within SMFG are abundant. Employees are encouraged to connect with colleagues and industry leaders through various platforms and events, enhancing their professional network and opening doors to myriad career opportunities.

Apply for a Position

Sumitomo Mitsui Financial Group, Inc. is actively hiring and looking for talented individuals who are passionate, curious, and driven. Explore open positions that match your skills and interests on the SMFG careers page.

Stay Connected with SMFG Careers

Keep up to date with the latest career tips, industry insights, and company news from Sumitomo Mitsui Financial Group, Inc. Subscribe to receive updates and stay informed about new job openings and employment trends.

Prepare for Your Interview

Aspiring to join SMFG? Prepare your resume to reflect your best self and gear up for the interview process where you can showcase your skills and passion for finance and innovation.

Career Opportunities Await

At Sumitomo Mitsui Financial Group, Inc., the potential for professional development and personal growth is limitless. Discover the exciting and rewarding career opportunities that await at SMFG, where every position contributes to the company’s global success and leadership in the financial industry.
Learn more about Sumitomo Mitsui Financial Group, Inc.
Industry
Finance & Insurance
* Ladders Estimates

Similar Jobs

Ad banner background

Get Ready For Your
Next Interview

More Jobs at Sumitomo Mitsui Financial Group, Inc.

More Enterprise Technology Jobs

Find similar Data Engineer - Capital Markets jobs:

NationwideNew York, NY