Sr. Data Engineer (PySpark, Azure Data Factory)

ePATHUSA, Inc.

$100K — $130K *
Information Technology
8 - 10 years of experience
Job Overview by Ladders

Qualifications

  • 8-10 years of data engineering and BI development experience.
  • Expertise in PySpark for data transformation and pipeline development.
  • Extensive experience with Azure Data Services (Data Factory, Data Lake, Synapse, Microsoft Fabric, DataBricks).
  • Proficient in Power BI dashboard creation and data modeling.
  • Strong SQL skills focused on query performance and scalability.
  • Experience with dimensional modeling (Star and Snowflake schemas).
  • Familiar with machine learning, AI concepts, and Azure ML Services.

Responsibilities

  • Engage with business users to gather and translate requirements into data solutions.
  • Design and implement scalable data pipelines using PySpark and Azure Data Factory.
  • Develop and optimize BI models and visualizations in Power BI.
  • Lead migration of legacy BI systems to cloud platforms.
  • Collaborate with teams to align with architectural standards and objectives.
  • Oversee testing and deployments, ensuring quality and performance.
  • Maintain and enhance existing BI solutions for reliability and scalability.

Benefits

  • Medical, vision, and dental coverage.
  • Prescription drug coverage.
  • 401(K) matching plan.
  • Paid time off and parental leave.
  • Comprehensive health programs, including wellness initiatives and FSAs.
  • Life insurance.
  • Education reimbursement program.
  • Adoption assistance.
  • Scholarship and vocational grant programs.
  • Training opportunities.
  • Bonus eligibility.
Full Job Description
Job Description
As an Data Engineer at Calpine Inc., you will contribute to the BI development to optimize energy operations, enhance forecasting, and improve operational efficiency.This role is ideal for professionals with 10+ years of experience who are passionate comprehensive data and BI solutions development and solving real-world energy sector challenges using PySpark and Azure Data Factory.

Job Responsibilities

  • Engagewith business users to gather requirements and translate them intocomprehensive data and BI solutions.
  • Designand implement scalable data pipelines using PySparkand AzureData Factory, integrating with Azure Data Lake, SynapseAnalytics, DataBricks, and Microsoft Fabric.
  • Developand optimize data models and visualizations using Power BI,ensuring performance and usability across business units.
  • Leadmodernization efforts by migrating legacy BI systems to cloud-basedplatforms.
  • Collaboratewith cross-functional teams to ensure alignment with architecturalstandards and business objectives.
  • Overseesystem testing, user acceptance testing (UAT), and production deployments.
  • Maintainand enhance existing BI solutions, ensuring reliability and scalability.
  • Documentdata flows, transformation logic, and architectural decisions.
  • Communicateeffectively with stakeholders regarding upcoming changes and coordinatechange management activities.


Requirements

Job Requirements

  • Minimum 8-10years of experience in data engineering and BI development.
  • Provenexpertise in PySpark for data transformation and pipelinedevelopment.
  • Extensiveexperience with Azure Data Services, including DataFactory, Data Lake, Synapse,MicrosoftFabricand DataBricks.
  • Strongproficiency in Power BI dashboard development and datamodeling.
  • Solidunderstanding of cloud architecture, data governance, and performanceoptimization.
  • AdvancedSQL skills with a focus on query performance and scalability.
  • Experiencewith dimensional modeling (Star and Snowflake schemas).
  • Excellentcommunication and collaboration skills in cross-functional environments.
  • Abilityto manage multiple priorities in a dynamic and fast-paced setting.
  • Familiaritywith SSIS, SSRS, and SSAS(tabular andmulti-dimensional models).
  • Experiencewith DAX and MDX query languages.
  • Exposureto Power Apps, Data Flows
  • Knowledgeof machine learning (ML)and artificial intelligence(AI) concepts and their application in data analytics.
  • Have workedwith Azure ML Services, and understand OpenAI LLM Model.
  • Experienceimplementing role-based and row-level security in BI platforms.


Benefits

Benefits include:

  • Medical, vision and dental
  • Prescription drug coverage
  • Calpine's Retirement Savings Plan (401(K)) matching
  • Paid time off and paid parental leave
  • Comprehensive health programs, including our Wellness Program, telemedicine, Employee Assistance Program, Health Care and Dependents Care Flexible Spending Accounts (FSAs) and more
  • Life insurance
  • Education reimbursement program
  • Adoption assistance
  • Scholarship and vocational grant programs
  • Training and education opportunities
  • Bonus eligible

Similar Jobs

More Information Technology Jobs

Find similar Sr. Data Engineer (PySpark, Azure Data Factory) jobs: