Director - Data & Systems Engineering- Job Description

LocationBoston, MA (Hybrid)Reports ToCIOTeam Size10 Engineers Across Data, Software, AI, and Platform EngineeringRole TypeHands-on player-coach; Director + Principal Architect profilePrimary StackPython, Snowflake, FastAPI, Redis, PostgreSQL, Databricks Azure, APIs, Distributed Systems, AI/LLM Platforms

Director - Data & Software Engineering, is responsible for leading BETM's core Data Engineering and Software Engineering functions. This role owns the architecture, delivery, and operational excellence of enterprise data platforms, APls, backend services, software applications, and engineering practices that support trading, risk management, asset management, analytics, operations, and internal business platforms.

The role is primarily focused on data platform leadership and software engineering execution. The successful candidate will guide engineering teams, establish technical standards, drive modernization, oversee delivery of business-critical systems, and ensure platforms are scalable, secure, reliable, maintainable, and cost-effective. This leader must be hands-on enough to review architecture, challenge design decisions, understand code-level tradeoffs, and coach engineers toward pragmatic, high-quality solutions.

While this is not intended to be a dedicated Al Architect role, the Director must have a strong working understanding of Al Engineering and collaborate closely with the Al Architect to ensure BETM's data, API, and software platforms are ready for Al-enabled use cases. This includes supporting RAG patterns, semantic layers, governed data access, vector/search integrations, model-facing APls, observability, evaluation workflows, and responsible Al controls in partnership with the Al Architect and broader technology leadership.

Key Responsibilities

• Lead and develop engineers across Data Engineering, Software Engineering, API development, platform services, and production support.
• Own the strategic direction, architecture, delivery, and operational health of BETM's enterprise data platforms and major software systems.

• Architect and develop latest trading and asset management platforms for management of client's assets and trade in the ISO markets.
• Establish engineering standards, development practices, documentation expectations, delivery discipline, code quality, testing practices, and operational readiness expectations.
• Define architecture standards for data platforms, APls, backend services, business applications, distributed systems, and cloud-native engineering solutions.
• Provide technical leadership across Python, SQL, FastAPI, REST APls, PostgreSQL, Redis, Snowflake, Azure services, event-driven patterns, and distributed systems.
• Lead design and modernization of software applications, internal platforms, data-backed products, and legacy systems while balancing delivery urgency with long-term maintainability.
• Own data engineering strategy, including ingestion pipelines, curated datasets, analytical data models, semantic layers, metadata, lineage, governance, data quality, testing, monitoring, and documentation.
• Drive Snowflake architecture, performance optimization, workload management, cost governance, access patterns, and platform reliability.
• Oversee software engineering delivery across APls, backend services, integrations, internal applications, automation workflows, and business-facing platforms.
• Partner with the Al Architect to ensure data, API, and software platforms support Al Engineering patterns such as RAG, semantic search, embeddings, vector databases, Al evaluation workflows, and model-facing service integration.
• Collaborate with the Al Architect on Al governance, data access controls, auditability, observability, hallucination-risk mitigation, and responsible Al practices where platform and data engineering decisions are involved.
• Partner with stakeholders across trading, risk, asset management, operations, analytics, IT, and executive leadership to translate business objectives into actionable engineering roadmaps and delivery plans.
• Manage prioritization, capacity planning, staffing plans, delivery commitments, stakeholder expectations, and tradeoff decisions across multiple engineering workstreams.
• Own production supports routines, incident management, monitoring, alerting, runbooks, operational readiness, reliability improvements, and continuous improvement through automation.
• Drive Cl/CD, secure deployment practices, automated testing, observability, service-level expectations, engineering KPls, and platform health metrics.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or equivalent practical experience.

• 15+ years of experience in software engineering, data engineering, platform engineering, or architecture.
• 7+ years of engineering leadership experience managing teams of 10+ engineers across data, software, API, platform, or application engineering functions.
• Deep hands-on expertise in Python, Snowflake, Databricks, REST API design, FastAPI or comparable API frameworks, distributed systems, and modern software architecture principles.
• Strong production experience with Snowflake, including data modeling, workload design, query performance, governance, cost optimization, operational monitoring, and platform reliability.
• Strong experience with PostgreSQL, Redis, Azure cloud services, DevOps practices, Cl/CD pipelines, automated testing, observability, and production operations.
• Proven experience designing and operating scalable data platforms, data pipelines, curated datasets, semantic layers, data quality frameworks, metadata/lineage practices, and governed analytical systems.
• Proven experience leading software engineering teams delivering APls, backend services, integrations, internal applications, automation platforms, and business-critical systems.
• Strong architectural judgment with the ability to review solution designs, evaluate technical tradeoffs, challenge assumptions, and guide teams toward maintainable and cost-effective implementation choices.
• Working understanding of Al Engineering concepts, including LLMs, RAG, embeddings, vector databases, semantic search, model-facing APls, Al evaluation, observability, and responsible Al controls.
• Ability to collaborate effectively with an Al Architect and translate Al platform needs into data, API, security, integration, and software engineering requirements.
• Experience supporting high-availability, business-critical platforms requiring reliability, scalability, security, auditability, monitoring, and disciplined production support.
• Strong communication skills with the ability to engage executives, business stakeholders, architects, product owners, analysts, and engineering teams.
• Preferred experience in energy trading, commodities, utilities, financial services, or other data-intensive industries.
• Preferred experience with Snowflake Cortex, Databricks, Azure OpenAI, Azure Al Services, LangGraph, Semantic Kernel, MCP, or comparable Al/data platform ecosystems, with emphasis on integration and platform readiness rather than pure Al research.

Salary Range: $200,000.00 - $240,000.00