Role SummaryWe are seeking a Senior Data Platform Engineer to support the administration, automation, governance, security, and optimization of our enterprise Databricks Lakehouse platform on AWS.
This is a hands-on technical role responsible for maintaining and enhancing the Databricks platform, implementing infrastructure automation, enforcing compute and security standards, supporting cost-governance initiatives, and enabling reliable Data Engineering and Data Science workloads.
The ideal candidate will have strong hands-on experience with Databricks platform administration, AWS cloud services, Terraform-based infrastructure automation, and DevOps practices. The individual will evaluate emerging Databricks capabilities through proof-of-concept implementations and contribute to reusable platform patterns and standards.
This role will collaborate closely with Data Engineering, Data Science, Cloud Platform, Security, and Governance teams to ensure the Databricks environment remains secure, reliable, scalable, and cost-efficient.
Key ResponsibilitiesDatabricks Platform Administration- Support and maintain enterprise Databricks workspace architecture, including networking, security, and compute governance.
- Implement and maintain cluster policies, job policies, compute guardrails, and platform governance standards.
- Manage and optimize Databricks compute environments including job clusters, interactive clusters, and serverless compute.
- Implement Unity Catalog for centralized data governance and access management.
- Manage workspace configurations, platform integrations, and operational reliability.
AWS Cloud Architecture for DatabricksDesign and maintain secure and scalable Databricks infrastructure leveraging AWS services including:
- IAM roles and cross-account access architecture
- VPC networking, subnets, and secure connectivity
- PrivateLink and secure endpoint configuration
- S3 data lake architecture
- AWS KMS encryption and key management
- CloudWatch monitoring and logging integration
Ensure the platform aligns with enterprise security, compliance, and networking standards.
Infrastructure as Code & DevOps Automation- Build and maintain Databricks infrastructure using Terraform.
- Implement infrastructure-as-code frameworks for workspace provisioning, cluster policies, and platform configurations.
- Develop automation using:
- Databricks CLI
- Databricks REST APIs
- Python and shell scripting
- Design and maintain CI/CD pipelines for Databricks workloads and platform infrastructure.
- Integrate deployments with Git-based development workflows (GitHub, GitLab, Azure DevOps).
- Enable standardized environment promotion frameworks across Dev, Test, and Production environments.
Platform Cost Governance & Optimization- Monitor and analyze DBU consumption and compute utilization across workloads.
- Provide recommendations on cost optimizations
- Implement guardrails such as:
- Cluster policies
- Budget alerts
- Resource tagging
- Usage monitoring dashboards
- Identify and implement cost optimization strategies across:
- Compute configurations
- Job scheduling patterns
- Serverless vs classic compute usage
- Storage lifecycle management
- Build dashboards for cost transparency.
Data Platform EnablementProvide technical guidance and platform best practices for data engineering and data science workloads, including:
Data Engineering
- Delta Lake architecture and performance optimization
- Medallion architecture (Bronze/Silver/Gold)
- Batch and streaming data ingestion
- Large-scale Spark workloads
Data Science & AI
- MLflow experimentation and model lifecycle management
- Feature Store usage
- Machine learning pipelines and model deployment frameworks.
Innovation & Proof of Concept DevelopmentEvaluate and operationalize new Databricks capabilities by building Proof of Concepts (POCs) and evaluate POCs and support adoption into platform standards.
Areas of focus include:
- Databricks Serverless Compute
- Delta Live Tables
- Auto Loader
- Lakehouse AI capabilities
- Databricks Feature Store
- Vector Search and LLM integrations
- Lakehouse monitoring and observability frameworks
Support and maintain reference architectures and reusable design patterns based on POC outcomes.
Required Technical SkillsDatabricks Platform- Databricks workspace administration
- Cluster and job policy design
- Unity Catalog governance
- Delta Lake architecture
- Databricks REST APIs and CLI automation
- Performance optimization and troubleshooting
AWS Cloud PlatformStrong hands-on experience with:
- AWS IAM
- VPC networking and security
- S3 data lake architecture
- AWS KMS encryption
- CloudWatch monitoring
- PrivateLink connectivity
- Cross-account access architecture
- Enterprise cloud security practices
Infrastructure & Automation- Terraform for Databricks and AWS infrastructure
- Infrastructure as Code design patterns
- Databricks CLI and API automation
- Python / Bash scripting
- Platform automation frameworks
DevOps & DataOps- CI/CD pipeline development
- Git-based version control workflows
- Automated deployment and release processes
- Monitoring, logging, and observability frameworks
- Incident management and root cause analysis
Data Engineering & Analytics Skills- Apache Spark / PySpark
- SQL optimization
- Delta Lake performance tuning
- Streaming frameworks
- Data pipeline orchestration
- Data modeling and ETL best practices
Preferred Qualifications- Databricks Certified Data Engineer / Databricks Platform Architect
- AWS Certified Solutions Architect
- Experience designing enterprise-scale Lakehouse platforms
- Experience supporting large-scale ML workloads
- Experience implementing data governance frameworks
This is a hybrid role. Tuesday through Thursday are in-office days at BJ's Club Support Center in Marlborough, MA and Monday and Friday are remote days.
In accordance with the Pay Transparency requirements, the following represents a good faith estimate of the compensation range for this position. At BJ's Wholesale Club, we carefully consider a wide range of non-discriminatory factors when determining salary. Actual salaries will vary depending on factors including but not limited to location, education, experience, and qualifications. The pay range for this position is $100,000.00 - $131,500.00
We recognize the growing role of AI tools, including ChatGPT, and value familiarity with them. That said, we want to hear from your authentic self. Your application should reflect your own skills, experiences, and insights rather than AI-generated responses.
