Job Summary
We are seeking a Senior Kafka Administrator to provide hands-on administration and operational support for enterprise streaming and data platforms across on-premises and cloud environments. The role involves managing Apache Kafka clusters across Cloudera, Confluent Platform, and AWS Managed Streaming for Apache Kafka (MSK), supporting platform reliability, and contributing to migration initiatives. The ideal candidate will have strong experience in Kafka administration, AWS MSK, Terraform, GitLab CI/CD, security configuration, and infrastructure modernization, along with the ability to collaborate effectively with platform engineers, SREs, and application teams.
Key Responsibilities
• Administer and operate Kafka clusters across Cloudera, Confluent Platform, and AWS MSK in collaboration with the platform team.
• Provide day-to-day operational support, including incident response, on-call participation, root cause analysis, and continuous reliability improvements.
• Manage Kafka topics, partitions, replication, retention, quotas, access control lists (ACLs), and consumer groups.
• Tune Kafka brokers, producers, and consumers to optimize throughput, latency, and reliability.
• Perform patching, upgrades, capacity planning, and cluster health and performance troubleshooting.
• Configure and maintain Kafka security, including TLS/SSL, SASL, mutual TLS (mTLS), IAM for AWS MSK, RBAC, ACLs, and encryption at rest and in transit.
• Support migration and workload-transition initiatives from on-premises Cloudera and Confluent environments to AWS MSK.
• Contribute to migration strategies using tools and approaches such as MirrorMaker 2, Confluent Replicator, and cluster linking.
• Validate topic parity, offsets, throughput, and data integrity before and after migrations.
• Help define rollback plans and conduct migration dry runs in lower environments to minimize downtime and prevent data loss.
• Build and maintain Kafka and AWS MSK infrastructure using Terraform.
• Manage automated deployments and configuration through GitLab CI/CD pipelines.
• Implement GitOps practices for topic, ACL, and cluster configuration management.
• Participate in code reviews and enforce version control and environment promotion practices across development, testing, and production.
• Collaborate with producer and consumer application teams, platform engineers, SREs, and other stakeholders to maintain reliable streaming services.
• Support related data platform modernization initiatives as required.
Required Qualifications
• Strong hands-on experience administering and operating Apache Kafka clusters in enterprise environments.
• Experience supporting Kafka across on-premises platforms, including Cloudera and Confluent Platform, and cloud environments using AWS MSK.
• Experience managing Kafka topics, partitions, replication, retention policies, quotas, ACLs, and consumer groups.
• Experience with Kafka performance tuning, cluster monitoring, incident management, and root cause analysis.
• Knowledge of Kafka security mechanisms, including TLS/SSL, SASL, mTLS, IAM, RBAC, and encryption practices.
• Experience with Terraform for infrastructure provisioning and configuration management.
• Experience developing or maintaining GitLab CI/CD pipelines for automated infrastructure deployment.
• Understanding of GitOps practices, version control, code reviews, and environment promotion.
• Experience supporting Kafka migration strategies and validating data integrity during workload transitions.
• Strong troubleshooting, communication, collaboration, and documentation skills.
• Ability to participate in on-call support and work collaboratively within a cross-functional platform engineering team.
