Software Engineer - Data Infrastructure - Kafka

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Canonical, so some details may not match this job. The employer's own page has the real description.

Overview

The Software Engineer - Data Infrastructure - Kafka at Canonical will play a pivotal role in designing, implementing, and maintaining robust data pipelines utilizing Kafka to support scalable data processing and analytics. This position requires a deep understanding of both data infrastructure and the Kafka ecosystem, as well as a commitment to optimizing data flows and ensuring reliability. The successful candidate will collaborate with cross-functional teams to define requirements, enhance existing systems, and contribute to strategic initiatives that drive data-driven decision-making across the organization.

Job Responsibilities

• Design and implement scalable data architectures leveraging Apache Kafka.
• Maintain and optimize data pipelines to ensure high availability and performance.
• Collaborate with data scientists, analysts, and engineering teams to translate requirements into technical solutions.
• Monitor system performance and troubleshoot issues related to data flow and integrity.
• Develop and maintain comprehensive documentation for data processes and systems.
• Identify opportunities for automation and optimization within data processing workflows.
• Stay updated on industry trends and best practices related to data infrastructure and streaming technologies.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 5+ years of experience in software engineering with a focus on data infrastructure.
• Proficiency in Apache Kafka including setup, configuration, and monitoring.
• Strong programming skills in Java, Scala, or Python.
• Experience with distributed data processing frameworks such as Apache Spark or Flink.
• Familiarity with data modeling, ETL processes, and database technologies (SQL/NoSQL).
• Solid understanding of cloud services and containerization technologies (e.g., Docker, Kubernetes).

Technologies & Tools

The Software Engineer - Data Infrastructure - Kafka will work extensively with a variety of technologies, including but not limited to: Apache Kafka for streaming data, Apache Spark for large-scale data processing, Docker for containerization, Kubernetes for orchestration, and cloud platforms such as AWS or Azure for deployment. Familiarity with SQL and NoSQL databases, along with data visualization tools, will also be advantageous in this role.

Ideal Candidates

Ideal candidates for this position will possess a robust analytical mindset and a strong passion for data-driven solutions. They should be self-motivated, able to work independently as well as collaboratively within a team setting. Excellent problem-solving skills, attention to detail, and strong communication abilities are essential. Furthermore, a proactive attitude towards learning new technologies and a commitment to best practices in data management will greatly enhance a candidate’s fit for this role.