If you have a passion for tackling business challenges through the use of both structured and unstructured data at scale, Tiger Analytics wants to connect with you. We are looking for a skilled and proactive Databricks Architect to take a pivotal role in designing and deploying resilient data solutions that address our clients' complex business issues.
Responsibilities- Design and implement scalable, end-to-end data solutions for ETL, real-time streaming, and machine learning workloads on the Databricks platform
- Provide technical leadership and strategic guidance on Databricks adoption, including developing reference architectures and best practices.
- Ensure optimal performance by tuning Spark jobs, optimizing data pipelines, and resolving complex technical issues.
- Architect, develop, and debug PySpark jobs and data models, integrating Databricks with cloud services such as AWS S3, EMR, and Lambda
- Implement and enforce data governance, security, and access control policies (e.g., encryption, auditing) on the Databricks environment
Requirements- 12+ years of experience in Architecture, design and data engineering development of large-scale data ecosystems.
- Databricks Design and Architecture: Design and architect scalable and secure Databricks solutions platform, optimizing performance, cost, and reliability.
- Hands on Databricks/Dataops Architect that can help reduce consumption, and improve performance issues. Can help build monitoring tools.
- Good exposure to AWS. Strong comm and leadership skills and Self driven and work independently.
- Experience in Performance initiatives and drive recommendations (identify tuning opportunities, creation of performance environment etc.
- Experience with hadoop environment.
- Core Skills:- Databricks platform knowledge, Unity Catalog migration implementation exposure, Spark Streaming pipeline design exposure
BenefitsSignificant career development opportunities exist as the company grows. The position offers a unique opportunity to be part of a small, fast-growing, challenging, and entrepreneurial environment, with a high degree of individual responsibility.
