Lead Data Engineer (Java, Scala, AWS, Spark, SQL)

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Optomi, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Lead Data Engineer at Optomi will play a pivotal role in designing and implementing robust data pipelines and infrastructure, leveraging a variety of technologies including Java, Scala, AWS, Spark, and SQL. The successful candidate will lead a team of data engineers, ensuring high-quality data solutions that support the organization's analytical and business needs. This position involves close collaboration with cross-functional teams to develop scalable data architectures and facilitate efficient data modelling and transformation processes.

Job Responsibilities

• Design, develop, and maintain scalable data architectures and pipelines using Java, Scala, and Spark.
• Collaborate with data scientists and analysts to understand data requirements and deliver tailored solutions.
• Implement best practices for data processing and ensure data quality across all data pipelines.
• Lead and mentor a team of data engineers, providing guidance on technical challenges and career development.
• Utilize AWS services for data storage, processing, and orchestration to enhance cloud-based data solutions.
• Monitor and troubleshoot data processing workflows to ensure optimal performance and reliability.
• Stay updated with industry trends and emerging technologies to continuously improve data engineering practices.
• Facilitate communication between stakeholders to ensure alignment on data strategy and initiatives.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field; master’s degree is preferred.
• 5+ years of experience in data engineering, with significant expertise in Java and Scala.
• Hands-on experience with AWS services: S3, Redshift, Lambda, and Glue.
• Proficiency in Spark for big data processing and SQL for data manipulation.
• Solid understanding of data warehousing concepts and data modeling techniques.
• Experience with ETL processes and tools.
• Strong analytical and problem-solving skills, with an ability to communicate complex technical concepts to non-technical stakeholders.
• Previous leadership experience or team management is a plus.

Technologies & Tools

The Lead Data Engineer role at Optomi employs a diverse technological stack that includes programming languages such as Java and Scala, along with data processing frameworks like Apache Spark. AWS cloud services, specifically Amazon S3, Redshift, Lambda, and Glue, are utilized for data storage, processing, and orchestration. The position also requires proficiency in SQL for database management and interaction.

Ideal Candidates

Ideal candidates for the Lead Data Engineer position are those who demonstrate strong technical proficiency, along with a proactive approach to problem-solving and innovation. They should possess excellent leadership skills, fostering collaboration and knowledge sharing within their team. A passion for data, an analytical mindset, and the ability to navigate complex projects while maintaining attention to detail will make them stand out. Furthermore, candidates should exhibit strong communication skills to effectively convey technical information to diverse stakeholders.