Job Type
Full-time
Description
Independently lead high-impact data engineering workstreams from problem definition through measurable production outcomes. Build reliable pipelines, trusted data models, and governed AI workflows, owning design, delivery, operational readiness, and results within Engineering standards.
Combine deep Python and SQL engineering with coding agents, retrieval systems, and targeted small-model experimentation. Evaluate architectures across approved cloud platforms based on security, correctness, performance, recoverability, and cost. Help establish how intelligent systems access and protect enterprise data, turning proven approaches into reusable team patterns.
Representative work: Modernize warehouse and lakehouse pipelines; integrate partner data through APIs, files, and secure shares; design event-driven data flows; strengthen reconciliation and automated recovery; build permission-aware agents; and evaluate emerging defenses in isolated environments.
What You'll Do:
End-to-end ownership: Clarify problems and success measures; design solutions within established architecture; manage dependencies and risks; and drive delivery within assigned workstreams, following Change Management processes, through adoption, production support, troubleshooting, and improvement.
Data platforms and event-driven systems: Build secure batch, streaming, and message-driven pipelines and reusable data services. Evaluate compute placement, data movement, throughput, latency, delivery guarantees, recoverability, and operating cost. Apply FinOps discipline: monitor cloud compute and storage spend, identify wasteful patterns, and right-size workloads.
Correctness and resilience: Engineer idempotent, replayable pipelines with deduplication, schema-evolution handling, and late-arriving data support. Build reconciliation, control-total, checkpoint, and bounded automated recovery tooling with human escalation, supporting Finance and Operations sign-off.
Trusted analytics: Develop curated datasets, documented models, and semantic layers for internal and partner-facing reporting. Partner with business users and BI on definitions, validation, lineage, and controlled access.
Agentic coding: Delegate bounded tasks to approved coding agents, supply repository context, manage handoffs, and independently validate generated code and data. Retain human ownership of quality, security, and release decisions.
Governed AI workflows: Build retrieval and tool-using agents over approved data and MCP servers. Implement permission-aware access as defined by data owners briefed by Security, along with state management, evaluations, tracing, recovery, and approvals. Treat retrieved content as untrusted.
Defensive AI engineering: Build and test defensive AI controls in partnership with Security, including segmented agent trust zones, isolated execution, policy-enforced access, and auditable tool use. Work within Security-provided sandbox environments and governance to evaluate emerging AI security and adversarial testing techniques through controlled, authorized exercises, producing evidence of meaningful testing to support go-live decisions. Measure containment, detection, and effects on legitimate workflows.
Applied modeling: Prototype small-model or machine-learning applications for classification, extraction, entity matching, and data-quality triage. Compare reproducible evaluations against simpler approaches before recommending production use.
Production accountability: Own reliability, operational readiness, runbooks, and remediation within assigned workstreams, following Change Management processes; collaborate on?CI/CD, identity, networking, secrets, observability, capacity, incidents, and costs. Produce audit evidence, support control testing, and maintain rollback plans and appropriate human approvals.
Team leverage: Mentor through design and code reviews; convert proven approaches into reusable components and playbooks. Help develop validated engineering and security patterns.
Requirements
What We're Looking For:
• 5+ years of production data engineering experience, or equivalent scope, delivering reliable cloud-based solutions.
• Demonstrated success independently leading ambiguous work through sustained production use while mentoring engineers and improving team practices.
• Strong Python and advanced SQL, including automated testing, query-plan analysis, performance tuning, and transformation validation.
• Production experience with Snowflake, Microsoft Fabric, Azure, AWS, or a comparable platform, including orchestration and data modeling. Depth and transferable judgment matter more than familiarity with every listed product.
• Experience integrating APIs, files, databases, or data shares, including incremental processing, backfills, schema changes, reconciliation, and recovery.
• Demonstrable use of coding agents to delegate meaningful work, manage context and handoffs, verify changes, and deliver tested results.
• Hands-on experience building a retrieval, agent, or LLM-enabled workflow over real data; understanding of evaluation, prompt-injection risks, permission boundaries, and limitations.
• Practical knowledge of CI/CD, infrastructure as code, least privilege, secrets management, observability, and troubleshooting.
• Ability to translate ambiguous business needs into focused solutions and explain tradeoffs to non-technical users.
• Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related field.
Preferred Qualifications:
• Advanced Snowflake engineering, including secure sharing, Snowpark, performance optimization, and cost governance.
• Microsoft Fabric, Azure Data Factory, Synapse Analytics, Dataverse, Power BI semantic models, or warehouse-to-lakehouse modernization.
• Event streaming and enterprise message queues, including Azure Service Bus and Azure Event Hubs alongside Apache Kafka, IBM MQ, and comparable platforms ; partitioning, ordering, replay, idempotent consumers, backpressure, and failure recovery.
• AWS data services, secure cross-cloud integration, infrastructure automation, or private-network architectures.
• Production agent systems, MCP integrations, persistent state, evaluation, tracing, and recovery.
• Agent security, sandboxing, adversarial evaluation, or deception engineering; familiarity with emerging AI security research is a plus, not a prerequisite.
• Small-model fine-tuning, LoRA/PEFT, quantization, self-hosted inference, or reproducible model evaluation.
• Banking, fintech, payments, or another regulated environment.
• Relevant certifications, or a master's or doctoral degree in a related field.
