Senior Scala Data Engineer - (Apache Spark, Flink, Kafka, AWS) - 100% Remote - W2 Contract

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Optomi, so some details may not match this job. The employer's own page has the real description.

Overview

Optomi is seeking a highly skilled Senior Scala Data Engineer with expertise in Apache Spark, Flink, Kafka, and AWS to join our innovative and collaborative team on a 100% remote, W2 contract basis. In this role, you will be responsible for designing, implementing, and maintaining robust data processing frameworks that facilitate real-time data analysis and pipeline construction. You will work closely with cross-functional teams to optimize data workflows, ensuring reliable and efficient data delivery to support various business intelligence and analytics initiatives.

Job Responsibilities

• Design and develop large-scale data processing solutions using Scala, Apache Spark, and Apache Flink.
• Implement real-time data streaming architectures with Apache Kafka to ensure continuous data flow.
• Collaborate with data scientists and analysts to understand their data needs and translate requirements into scalable data models.
• Optimize existing data pipelines, ensuring high performance and reliability for production workloads.
• Maintain and enhance cloud-based data storage and processing infrastructures using AWS services such as S3, EMR, and Redshift.
• Perform data quality checks and troubleshoot data-related issues, ensuring accuracy and consistency in data delivery.
• Document system design, processes, and architecture for future reference and compliance purposes.
• Stay updated with the latest industry trends and technologies related to data engineering and analytics.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field; Master's degree preferred.
• 5+ years of experience in data engineering, with a focus on Scala and big data technologies.
• Proficiency in Apache Spark, Apache Flink, and Apache Kafka for data processing and streaming.
• Extensive experience with AWS services, including but not limited to S3, EMR, Lambda, and Redshift.
• Strong programming skills in Scala and familiarity with functional and object-oriented programming paradigms.
• Knowledge of data warehousing concepts, ETL processes, and data modeling techniques.
• Experience with CI/CD pipelines and tools such as Jenkins or GitLab CI.
• Excellent problem-solving skills and an analytical mindset.

Technologies & Tools

This position will leverage a range of cutting-edge technologies and tools including Scala, Apache Spark, Apache Flink, Apache Kafka, and AWS services such as S3 and Redshift. Familiarity with data engineering frameworks and tools as well as cloud infrastructure management will be essential for success in this role.

Ideal Candidates

The ideal candidates for the Senior Scala Data Engineer position will exhibit a strong passion for data engineering and analytics, with a keen interest in building scalable data solutions. They will possess exceptional analytical and problem-solving skills, with the ability to work collaboratively in a team-oriented environment. Candidates should be proactive learners, eager to adapt to new technologies and methodologies, and demonstrate effective communication skills to articulate technical concepts to both technical and non-technical stakeholders.