Sr Big Data Engineer Airflow and Oozie (GCP)

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Rackspace Technology, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Senior Big Data Engineer specializing in Airflow and Oozie on Google Cloud Platform (GCP) at Rackspace Technology will be responsible for designing, developing, and implementing scalable data pipelines to manage large datasets. This role entails collaborating with cross-functional teams to ensure the smooth integration of data across various platforms, enhancing data processing capabilities, and optimizing workflow orchestration. The candidate will leverage their expertise to drive big data solutions that support strategic business initiatives and deliver high-quality, actionable insights.

Job Responsibilities

• Design and implement data processing frameworks using Apache Airflow and Oozie on GCP.
• Optimize data pipelines for performance and reliability, ensuring efficient data flow.
• Collaborate with data architects and analysts to understand data requirements and implement solutions that meet business needs.
• Conduct code reviews and mentor junior engineers to enhance team capabilities and performance.
• Monitor and troubleshoot data workflows and pipelines, ensuring minimal downtime and continuous data availability.
• Manage data ingestion, transformation, and validation processes to maintain data integrity.
• Stay updated on industry trends and best practices to introduce innovative big data solutions.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field; Master’s degree preferred.
• 5+ years of experience in big data technologies, with a focus on Apache Airflow and Oozie.
• Strong knowledge of Google Cloud Platform services, including BigQuery, Dataflow, and Dataproc.
• Proficiency in programming languages such as Python, Java, or Scala.
• Experience with data modeling, ETL processes, and relational and non-relational databases.
• Strong problem-solving skills and the ability to work collaboratively in a fast-paced environment.
• Excellent communication skills, with a focus on technical documentation and stakeholder interaction.

Technologies & Tools

This position will utilize a variety of technologies and tools, primarily focusing on Apache Airflow and Oozie for workflow orchestration. Experience with Google Cloud Platform technologies, such as BigQuery, Dataflow, and Dataproc, will be leveraged for data storage and processing. Additionally, familiarity with databases (both SQL and NoSQL), cloud services, and programming frameworks is essential to effectively manage large datasets and build robust data pipelines.

Ideal Candidates

The ideal candidate for the Senior Big Data Engineer role will possess a strong analytical mindset and a collaborative spirit. They will be proactive in identifying problems and developing innovative solutions while maintaining composure in high-pressure situations. A successful candidate will demonstrate a passion for big data technologies, coupled with excellent communication skills, enabling effective interactions with various stakeholders. Adaptability and a commitment to continuous learning will further enhance their potential for success in this position.