Sr. Kafka Glue DevOps Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Irvine Technology Corporation, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Sr. Kafka Glue DevOps Engineer at Irvine Technology Corporation will be responsible for designing, implementing, and managing scalable Kafka and Glue solutions to support our data processing initiatives. The ideal candidate will collaborate with cross-functional teams to ensure continuous integration and delivery of data pipelines, while maintaining high availability and reliability of data services. This role demands a deep understanding of cloud-based solutions, data streaming, and the ability to troubleshoot complex issues in a fast-paced environment.

Job Responsibilities

• Design and architect scalable data pipelines using Apache Kafka and AWS Glue.
• Implement and manage CI/CD processes for data applications ensuring rapid deployment and integration.
• Monitor and optimize Kafka performance for high throughput and low latency.
• Collaborate with data engineers and data scientists to integrate data sources and endpoints.
• Troubleshoot data pipeline failures, identify root cause issues and develop solutions.
• Ensure data quality and integrity across all data processing workflows.
• Document infrastructure and processes to maintain knowledge sharing within the team.
• Stay updated with emerging technologies and best practices in data engineering and DevOps.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Minimum of 5 years of experience in a DevOps or engineering role focused on data pipelines.
• Proficiency in Apache Kafka and AWS Glue, with hands-on experience in stream processing.
• Strong knowledge of CI/CD tools such as Jenkins, GitLab, or CircleCI.
• Experience with cloud platforms, particularly AWS services (Lambda, S3, EC2, etc.).
• Understanding of containerization technologies like Docker and orchestration tools such as Kubernetes.
• Excellent problem-solving skills and the ability to work under pressure.
• Strong written and verbal communication skills.

Technologies & Tools

This position utilizes a range of technologies and tools to manage and develop data solutions, including but not limited to:

• Apache Kafka for real-time data streaming and processing.
• AWS Glue for ETL processes and data cataloging.
• Docker and Kubernetes for containerization and orchestration.
• Jenkins, GitLab for CI/CD automation tasks.
• AWS services such as S3, EC2, and Kinesis for data management and processing.

Ideal Candidates

The ideal candidate for the Sr. Kafka Glue DevOps Engineer role will possess not only the requisite technical skills but also demonstrate a strong aptitude for collaboration and innovative thinking. They should be proactive problem solvers with the ability to adapt to evolving technologies and practices. A commitment to continuous learning and improvement, along with excellent interpersonal skills, will make them a vital asset to the team. Candidates should also exhibit a strong passion for data engineering and an enthusiasm for contributing to scalable data architecture solutions.