Job DescriptionWe are looking for a talented
Data Engineer- Senior to join our team specializing in
Systems/Information Technology for our
Corporate organization in
Indianapolis, IN.
In this role, you will make an impact in the following ways:- Enable reliable data flow at scale by designing and automating deployments for distributed systems that ingest and transform diverse data sources, ensuring consistent and seamless data availability across the organization.
- Improve data trust and decision-making by building robust monitoring frameworks that quickly detect and resolve data quality and integrity issues before they impact analytics.
- Strengthen data governance and compliance by implementing clear standards for metadata, access control, and data retention, making data easier to discover, secure, and use responsibly.
- Accelerate analytics delivery by designing scalable, efficient data pipelines with built-in monitoring and alerting, reducing downtime and improving responsiveness to business needs.
- Enhance system performance and efficiency by creating optimized physical data models, indexing strategies, and table relationships that reduce query times and resource usage.
- Drive innovation through modern data architecture by leveraging cloud and distributed platforms (e.g., Data Lakes, NoSQL, Hadoop ecosystems) to support high-volume, high-velocity data processing.
- Increase productivity through automation by eliminating repetitive manual tasks in data preparation and integration, reducing errors and freeing up teams for higher-value work.
- Elevate team capability and delivery speed by mentoring teammates and applying Agile/DevOps practices, ensuring continuous improvement and successful execution of critical analytics initiatives
ResponsibilitiesTo be successful in this role you will need the following:- System Requirements Engineering - Uses appropriate methods and tools to translate stakeholder needs into verifiable requirements to which designs are developed; establishes acceptance criteria for the system of interest through analysis, allocation and negotiation; tracks the status of requirements throughout the system lifecycle; assesses the impact of changes to system requirements on project scope, schedule, and resources; creates and maintains information linkages to related artifacts.
- Collaborates - Building partnerships and working collaboratively with others to meet shared objectives.
- Communicates effectively - Developing and delivering multi-mode communications that convey a clear understanding of the unique needs of different audiences.
- Customer focus - Building strong customer relationships and delivering customer-centric solutions.
- Decision quality - Making good and timely decisions that keep the organization moving forward.
- Data Extraction - Performs data extract-transform-load (ETL) activities from variety of sources and transforms them for consumption by various downstream applications and users using appropriate tools and technologies.
- Programming - Creates, writes and tests computer code, test scripts, and build scripts using algorithmic analysis and design, industry standards and tools, version control, and build and test automation to meet business, technical, security, governance and compliance requirements.
- Quality Assurance Metrics - Applies the science of measurement to assess whether a solution meets its intended outcomes using the IT Operating Model (ITOM), including the SDLC standards, tools, metrics and key performance indicators, to deliver a quality product.
- Solution Documentation - Documents information and solution based on knowledge gained as part of product development activities; communicates to stakeholders with the goal of enabling improved productivity and effective knowledge transfer to others who were not originally part of the initial learning.
- Solution Validation Testing - Validates a configuration item change or solution using the Function's defined best practices, including the Systems Development Life Cycle (SDLC) standards, tools and metrics, to ensure that it works as designed and meets customer requirements.
- Data Quality - Identifies, understands and corrects flaws in data that supports effective information governance across operational business processes and decision making.
- Problem Solving - Solves problems and may mentor others on effective problem solving by using a systematic analysis process by leveraging industry standard methodologies to create problem traceability and protect the customer; determines the assignable cause; implements robust, data-based solutions; identifies the systemic root causes and ensures actions to prevent problem reoccurrence are implemented.
- Values differences - Recognizing the value that different perspectives and cultures bring to an organization.
Education, Licenses, Certifications: - College, university, or equivalent degree in relevant technical discipline, or relevant equivalent experience required.
- This position may require licensing for compliance with export controls or sanctions regulations.
Experience: Intermediate experience in a relevant discipline area is required. Knowledge of the latest technologies and trends in data engineering are highly preferred and includes:
- Familiarity analyzing complex business systems, industry requirements, and/or data regulations
- Background in processing and managing large data sets
- Design and development for a Big Data platform using open source and third-party tools
- SPARK, Scala/Java, Map-Reduce, Hive, Hbase, and Kafka or equivalent college coursework
- SQL query language
- Clustered compute cloud-based implementation experience
- Experience developing applications requiring large file movement for a Cloud-based environment and other data extraction tools and methods from a variety of sources
- Experience in building analytical solutions
Intermediate experiences in the following are preferred:
- Experience with IoT technology
- Experience in Agile software development
QualificationsCore Responsibilities / Activities: - Design and implement scalable and efficient data pipelines using Apache Spark and Databricks on Azure.
- Lead the complex transformation and integration of unstructured data sources into structured Delta Lake formats, applying software engineering best practices to ensure reliability, modularity, and reusability.
- Troubleshoot and optimize Spark jobs for performance, reliability, and cost-efficiency in a production environment.
- Drive continuous improvement of data engineering solutions by leveraging AI/ML and LLM-based techniques to enhance observability, performance optimization, and long-term maintainability.
Skill, Education, or Experience Requirements: - Minimum of 5 years of hands-on experience in data engineering with expertise in Azure Databricks and programming in Scala or Python.
- Proven experience in building and maintaining structured streaming pipelines using Spark.
- Strong knowledge of big data technologies, including Delta Lake, Apache Spark, Structured Streaming, and SQL.
- Experience with Git for version control and CI/CD pipeline management.
Nice to Have (Preferences): - Data Engineering Certification (e.g., Databricks Certified Data Engineer, Apache Spark Professional Data Engineer, or equivalent).
- Exposure to real-time data ingestion frameworks and cloud-native data services (e.g., Azure Event Hub, Azure Data Lake, AWS SQS, etc).
- Familiarity with data governance, access control (e.g., Unity Catalog or Immuta), and performance monitoring tools in cloud environments.
