Kafka Engineers/AWS/Terraform

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Dice, so some details may not match this job. The employer's own page has the real description.

Overview

The Kafka Engineer with expertise in AWS and Terraform will be responsible for designing, implementing, and managing scalable Kafka architecture solutions. This role encompasses ensuring high availability and performance of Kafka clusters, leveraging AWS services to facilitate cloud integration, and utilizing Terraform for infrastructure as code deployment. The ideal candidate will work closely with cross-functional teams to analyze, improve, and maintain data streaming capabilities while adhering to best practices in security and data governance.

Job Responsibilities

• Design, deploy, and manage Kafka clusters in both on-premise and AWS environments.
• Implement and maintain infrastructure as code using Terraform for efficient deployment and scaling.
• Collaborate with data engineering and application development teams to optimize data streaming workflows.
• Monitor and troubleshoot Kafka performance issues and ensure the reliability of data pipelines.
• Establish security best practices for data governance and access control in Kafka environments.
• Integrate Kafka with other AWS services for seamless data processing and analytics.
• Document system architecture, configuration changes, and operational procedures.
• Participate in on-call rotations for support and maintenance of production-level systems.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Proven experience managing Kafka clusters and streaming data pipelines.
• Strong familiarity with AWS services, particularly with services like EC2, S3, and Lambda.
• Hands-on experience with Terraform or similar infrastructure as code tools.
• Proficiency with containerization technologies such as Docker and orchestration platforms like Kubernetes.
• Solid understanding of messaging systems and data streaming technologies.
• Excellent analytical and problem-solving skills.
• Effective communication and teamwork abilities.

Technologies & Tools

This position will require a solid understanding of technologies such as Apache Kafka and its ecosystem, AWS cloud services, and Terraform for managing infrastructure. Candidates should also be skilled in using monitoring tools for Kafka performance, like Prometheus or Grafana, and familiar with data processing frameworks that integrate seamlessly with Kafka, such as Apache Flink or Spark. Knowledge of CI/CD pipelines is also beneficial for efficient deployments and testing.

Ideal Candidates

The ideal candidate for this position possesses a combination of technical expertise and a collaborative mindset. They are detail-oriented, capable of independently managing complex projects, and demonstrate a proactive approach to problem-solving. Successful candidates will be strong communicators, comfortable working in cross-functional teams, and committed to continuous learning and adapting to new technologies in the evolving field of data engineering.