Software Engineer - Data Infrastructure - Kafka

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Canonical, so some details may not match this job. The employer's own page has the real description.

Job Overview

As a Software Engineer specializing in Data Infrastructure at Canonical, you will play a vital role in designing, building, and maintaining the data architecture, focusing particularly on Kafka as a distributed streaming platform. You will collaborate with cross-functional teams to ensure seamless data flow and integrity, optimize performance, and contribute to projects that enhance the overall data infrastructure. Your expertise in deploying and managing a scalable and resilient data pipeline will enable the organization to support critical business applications and deliver actionable insights.

Job Responsibilities

• Design, develop, and maintain robust data pipeline architectures that ingest, process, and store large datasets.
• Implement and manage Kafka to ensure reliable, real-time event streaming and messaging solutions.
• Collaborate with data scientists and analysts to define data requirements and optimize data models.
• Monitor system performance and troubleshoot issues to ensure maximum uptime and data integrity.
• Develop and enforce data governance standards and best practices across the organization.
• Participate in code reviews and provide constructive feedback to ensure adherence to quality standards.
• Continuously evaluate and improve systems to enhance reliability, scalability, and maintainability.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field; or equivalent experience.
• Proven experience as a Software Engineer with a focus on data infrastructure and streaming technologies.
• Strong proficiency with Apache Kafka and associated ecosystem tools (Confluent, Kafka Streams, etc.).
• Experience with programming languages such as Java, Scala, or Python.
• In-depth understanding of database systems, including SQL and NoSQL databases.
• Familiarity with cloud platforms (AWS, Azure, or GCP) and their data services.
• Solid understanding of data governance, ETL processes, and data modeling principles.
• Ability to work effectively in a collaborative, team-oriented environment.

Technologies & Tools

The Software Engineer in this role will be working with a variety of technologies, including but not limited to Apache Kafka, Confluent Platform, Apache Spark, and cloud-based services from AWS or GCP. Proficiency in programming languages such as Java, Scala, or Python will be key in developing scalable data solutions. Familiarity with data visualization tools and orchestration platforms, such as Apache NiFi or Airflow, is also beneficial.

Ideal Candidates

Ideal candidates for this position will possess a strong analytical mindset with a passion for problem-solving in complex systems. They will exhibit excellent communication skills, enabling effective collaboration with peers and stakeholders. A proactive approach to continuous learning and adaptation to new technologies is essential, alongside a natural tendency for innovation and creativity in designing data solutions. Candidates should thrive in a dynamic, fast-paced environment, demonstrating resilience and commitment to project goals.