Position summary

The Associate Data Engineer supports the day-to-day operations of Candid's cloud data platform. This role is responsible for maintaining ingestion and transformation pipelines built on Apache Iceberg, validating data outputs through schema and structural changes, and assisting with storage management, platform observability, and metadata operations. The Associate Data Engineer develops foundational skills across the modern data lakehouse stack while taking direct ownership of pipeline maintenance, documentation, and validation activities.

Position: Associate Data Engineer

Reporting to: Data Operations Manager

Supervises: N/A

Schedule: 35-hour work week, Monday through Friday

Compensation: $70,000 - $95,000 (this range is for the NYC area and will be adjusted for other localities; additionally, factors like skills and experience will be considered).

Location: Remote. In-person attendance is expected twice per year during our annual, weeklong all-staff summits. Additional in-person meeting participation is expected at least once per quarter for senior leaders and at least once per month for the executive team. Staff not located in the NYC area are expected to travel for these meetings.

Benefits: Health insurance (medical, dental, vision), retirement contribution with additional option for a match, paid life insurance and AD&D, paid leave time (PTO, compassionate leave, volunteer, holiday, parental), short-term and long-term disability, pre-tax transit, flexible spending accounts, supplemental insurance, summer hours, and Public Service Loan Forgiveness (PSLF) program eligible employer.

Responsibilities

• Pipeline Maintenance, Documentation, & Validation: Serve as the primary owner of ingestion pipelines and transformation table adjustments. Ensure continued, reliable data delivery and apply routine changes as business and schema needs evolve. Validate transformation outputs against expected results after schema or structural changes, documenting findings and escalating anomalies to the appropriate teams.
• Storage & Platform Support: Assist with scheduling compaction and cleanup jobs to maintain Iceberg table health and query performance. Support partition evolution and snapshot retention management to control storage growth.
• Observability & Metadata: Assist in implementing and maintaining CloudWatch metrics, alarms, and dashboards to ensure pipeline visibility. Contribute to tracking and reporting on platform performance metrics. Help maintain AWS Glue metadata refresh and statistics jobs that support query planning and optimization within the data platform.
• Schema Coordination: Assist with coordinating schema changes across ingestion and transformation layers to maintain consistency end to end. Collaborate with the Data Operations Engineer to communicate impacts and sequence changes safely.
• Infrastructure & Security: Support and maintain RBAC and ABAC (least privilege, standardized roles, and consistent tagging). Participate in access reviews and audits, documenting changes and escalating risks as needed.

Requirements

• 1- 3 years of experience in data engineering, analytics engineering, or a closely related technical role; internships and relevant academic project work considered.
• Solid SQL skills, including writing, reading, and debugging queries against relational or columnar data stores.
• Familiarity with cloud data concepts: object storage (Amazon S3), columnar file formats (i.e. Parquet), data-interchange formats (JSON, XML), or open table formats (i.e. Apache Iceberg).
• Experience with or exposure to distributed SQL query engines such as Trino or Starburst
• Familiarity with AWS services such as S3 and Glue.
• Exposure to Apache Airflow, SSIS or another workflow orchestration platform
• Experience writing or maintaining data pipelines in Python.
• Familiarity with on-prem relational data systems (i.e. Microsoft SQL Server).
• Strong attention to detail, especially around data validation and output accuracy.
• Strong analytical and problem-solving skills.
• Excellent written and verbal communication skills; ability to document findings clearly for both technical and non-technical audiences.
• Ability to work independently and collaboratively as part of a distributed team.
• Willingness to perform other duties and special projects as needed/requested.
• Sensitivity and respect for racial, gender, sexual orientation, and cultural differences.
• Champions and represents Candid's core values: We're driven, direct, accessible, curious, and inclusive.