Software Engineer - Data Infrastructure - Kafka

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Canonical, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Software Engineer - Data Infrastructure - Kafka at Canonical will play a pivotal role in designing, developing, and deploying robust data infrastructure solutions leveraging Kafka technology. This position requires an individual who is passionate about constructing scalable and high-performance data pipelines, contributing to the enhancement of data accessibility and integrity across diverse applications. The ideal candidate will collaborate with cross-functional teams to ensure that the data infrastructure supports the company's commitment to delivering high-quality products and services.

Job Responsibilities

• Design, implement, and maintain scalable data pipelines and architecture using Kafka.
• Collaborate with product and engineering teams to gather data requirements and translate them into technical specifications.
• Build and optimize real-time data processing applications to support analytical workloads.
• Monitor and analyze system performance metrics, ensuring high availability and reliability of the data infrastructure.
• Troubleshoot and resolve issues related to data ingestion, processing, and storage.
• Document architecture, data flows, and system changes to maintain a comprehensive knowledge base.
• Stay updated with industry trends and best practices to continuously improve the existing data infrastructure.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred.
• Proven experience with Apache Kafka and related data streaming technologies.
• Strong programming skills in languages such as Java, Scala, or Python.
• Experience with data modeling and query languages (SQL and NoSQL databases).
• Solid understanding of distributed systems and microservices architecture.
• Knowledge of cloud platforms, containerization, and orchestration tools (Docker, Kubernetes).
• Familiarity with DevOps tools and practices for CI/CD and automated testing.

Technologies & Tools

This position primarily utilizes Apache Kafka for building data pipelines and streaming applications. Other relevant technologies include SQL and NoSQL databases for data storage, cloud platforms such as AWS or Azure for deployment, and containerization tools like Docker and Kubernetes for managing microservices and infrastructure. Familiarity with monitoring tools and logging frameworks is also essential for ensuring the reliability of data systems.

Ideal Candidates

Ideal candidates for this position possess strong analytical and problem-solving skills, along with a deep understanding of data engineering principles. They are proactive, detail-oriented, and demonstrate a genuine enthusiasm for working with cutting-edge technologies. Excellent communication skills are essential for effective collaboration with team members from diverse backgrounds. A commitment to continuous learning and improvement in the fast-evolving field of data infrastructure is highly valued.