Software Engineer - Data Infrastructure - Kafka

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Canonical, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Software Engineer - Data Infrastructure - Kafka at Canonical will play a critical role in designing, implementing, and maintaining scalable data infrastructure solutions utilizing Kafka. This position requires a solid understanding of distributed systems and expertise in data processing frameworks, ensuring high data reliability and availability. The ideal candidate will collaborate with cross-functional teams to enhance our data pipeline capabilities and contribute to building innovative data-driven applications that meet our clients' needs.

Job Responsibilities

• Design, develop, and maintain robust data pipeline architectures using Apache Kafka.
• Collaborate with data engineers, data scientists, and software developers to integrate data solutions into existing systems.
• Monitor and optimize data flow, ensuring data integrity and low-latency processing.
• Implement best practices for data governance, security, and compliance.
• Conduct performance tuning of systems and SQL queries in a high-traffic environment.
• Participate in peer code reviews and offer constructive feedback to others.
• Design and maintain comprehensive documentation of data processes and standards.
• Stay current with industry trends and emerging technologies to continuously improve data infrastructure.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• Proven experience with Apache Kafka and data streaming technologies.
• Strong programming skills in languages such as Java, Scala, or Python.
• Experience with data processing frameworks like Apache Spark or Flink.
• Understanding of SQL and NoSQL databases, including MongoDB, Cassandra, or similar technologies.
• Familiarity with containerization technologies (Docker, Kubernetes).
• Experience with cloud-based platforms (AWS, GCP, Azure) is a plus.
• Excellent problem-solving skills and the ability to work independently as well as in a team environment.

Technologies & Tools

The Software Engineer - Data Infrastructure - Kafka position primarily utilizes Apache Kafka for data streaming, alongside a suite of tools including Apache Spark for data processing, SQL and NoSQL databases for data storage, and cloud infrastructure resources for deployment and scaling. Familiarity with Docker and Kubernetes for container orchestration is also beneficial as part of our technology stack.

Ideal Candidates

Ideal candidates for this position will demonstrate a strong analytical mindset, an eagerness to tackle challenging data problems, and a proactive approach to continuous improvement. They should possess a collaborative spirit, able to work effectively within diverse teams while maintaining independence in their task management. A passion for data engineering and an understanding of its impact on business decisions are essential for success in this role at Canonical.