Software Engineer - Data Infrastructure - Kafka

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Canonical, so some details may not match this job. The employer's own page has the real description.

Overview

The Software Engineer - Data Infrastructure - Kafka at Canonical will play a pivotal role in designing, building, and maintaining robust data pipelines and infrastructure solutions that leverage Apache Kafka. This position involves collaborating with cross-functional teams to optimize data flow and ensure data integrity across various applications. The successful candidate will work on high-performance systems that support real-time analytics and data processing, contributing to the overall data strategy while focusing on scalability and reliability.

Job Responsibilities

• Design and implement data pipelines using Apache Kafka for processing and streaming data.
• Collaborate with data science and engineering teams to optimize data architecture and streamline workflows.
• Monitor and troubleshoot data infrastructure performance, ensuring high availability and reliability.
• Develop reusable components and libraries to support data ingestion and transformation processes.
• Write and maintain technical documentation to facilitate team cohesion and onboarding.
• Evaluate and integrate new technologies to enhance existing systems and workflows.
• Participate in code reviews and contribute to team best practices to improve code quality.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• Proven experience in software development with a focus on data engineering or data infrastructure roles.
• Strong knowledge of Apache Kafka and its ecosystem.
• Proficiency in programming languages such as Java, Scala, or Python.
• Experience with data modeling and database technologies (SQL and NoSQL).
• Familiarity with cloud platforms (AWS, Google Cloud, Azure) and containerization tools (Docker, Kubernetes).
• Excellent problem-solving skills and the ability to work in a fast-paced, collaborative environment.

Technologies & Tools

The role will involve working primarily with Apache Kafka for data streaming, alongside various tools and technologies such as Confluent, Apache Flink, Spark, and cloud services (AWS, GCP, or Azure). Proficiency in database management systems like PostgreSQL or MongoDB, as well as familiarity with container orchestration tools like Docker and Kubernetes, will also be essential for optimizing data infrastructure.

Ideal Candidates

The ideal candidate for this position will possess strong analytical thinking and problem-solving capabilities, with a passion for big data and real-time processing. They will be detail-oriented and exhibit strong communication skills, enabling effective collaboration with different teams. An inherent curiosity towards exploring new technologies and a commitment to delivering high-quality work will further distinguish a successful candidate in this dynamic environment.