Software Engineer - Data Infrastructure - Kafka

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Canonical, so some details may not match this job. The employer's own page has the real description.

Overview

Canonical is seeking a dedicated and skilled Software Engineer specializing in Data Infrastructure, particularly focusing on Kafka. The successful candidate will play a crucial role in designing, implementing, and maintaining scalable data pipelines and architectures that enhance our data ecosystem. The engineer will collaborate within cross-functional teams to ensure high performance and reliability of the data infrastructure. This role requires a balance between technical expertise in distributed systems and a capacity for strategic problem-solving in data management.

Job Responsibilities

• Design, build, and maintain scalable and reliable data infrastructure using Kafka.
• Collaborate with data scientists and business analysts to define data requirements and optimize existing data pipelines.
• Implement best practices for data management, including data quality, security, and access control.
• Monitor and troubleshoot data pipelines to ensure seamless data flow and performance.
• Evaluate and adopt new technologies that enhance data processing and analysis capabilities.
• Provide support for data infrastructure-related issues and perform root cause analysis.
• Participate in code reviews and contribute to the development of best practices for the team.
• Document data processes and systems to enable knowledge sharing within the team.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field; Master's degree is a plus.
• Proven experience in software engineering, particularly with data infrastructure and distributed systems.
• Strong proficiency with Kafka and related ecosystems (e.g., Kafka Connect, KSQL).
• Experience with programming languages such as Java, Scala, or Python.
• Solid understanding of data modeling, ETL processes, and data warehousing concepts.
• Familiarity with containerization technologies such as Docker and orchestration platforms like Kubernetes.
• Knowledge of cloud technologies and services (e.g., AWS, Google Cloud, Azure) is preferred.
• Excellent problem-solving skills and the ability to work under pressure.

Technologies & Tools

This role will utilize a variety of modern technologies and tools that include, but are not limited to, Apache Kafka, Apache Spark, AWS/Azure services, Docker, Kubernetes, and SQL/NoSQL databases. The engineer will also leverage tools for monitoring and logging to ensure that all data systems operate efficiently and effectively.

Ideal Candidates

The ideal candidate for this position possesses a deep technical knowledge of data systems and a passion for engineering innovative data solutions. They are analytical thinkers who thrive in a collaborative environment and possess strong communication skills to effectively convey complex technical concepts to non-technical stakeholders. A proactive approach to problem-solving and a commitment to continuous learning and improvement are essential qualities for success in this role.