Sr Big Data Engineer Airflow and Oozie (Google Cloud Platform)

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Dice, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Sr Big Data Engineer specializing in Airflow and Oozie on Google Cloud Platform at Dice is a pivotal role responsible for designing, implementing, and maintaining large-scale data processing pipelines. This position will require collaboration with cross-functional teams to ensure data integrity and availability while optimizing workflow through efficient orchestration techniques. The ideal candidate will demonstrate excellence in managing complex data environments, with a strong emphasis on the use of Airflow and Oozie for workflow automation in cloud-based infrastructures.

Job Responsibilities

• Design, develop, and maintain big data pipelines utilizing Airflow and Oozie within Google Cloud Platform.
• Implement data ingestion, transformation, and storage solutions to facilitate data processing across various applications.
• Collaborate with data scientists and analysts to understand data requirements and optimize existing workflows.
• Monitor and troubleshoot data pipelines to ensure timely and accurate data delivery.
• Establish best practices and standards for data engineering processes and workflows.
• Provide technical guidance and mentoring to junior data engineers and team members.
• Stay abreast of emerging trends in big data technologies and practices, and assess their applicability for the organization.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field; Master's degree preferred.
• 5+ years of experience in big data engineering or related fields.
• Strong proficiency in Apache Airflow and Apache Oozie.
• Experience with Google Cloud Platform services such as BigQuery, Dataflow, and Cloud Storage.
• Solid understanding of data modeling, ETL processes, and data warehousing concepts.
• Proficient in programming languages such as Python, Java, or Scala.
• Strong analytical and problem-solving skills.

Technologies & Tools

The Sr Big Data Engineer will work extensively with a variety of technologies and tools, including but not limited to Apache Airflow, Apache Oozie, Google Cloud Platform services such as BigQuery, Google Cloud Storage, and Dataflow, as well as database technologies like SQL and NoSQL databases. Familiarity with containerization technologies such as Docker and orchestration platforms like Kubernetes may also be beneficial.

Ideal Candidates

Ideal candidates for this position will possess a strong combination of technical expertise and interpersonal skills. They should exhibit a proactive mindset with a passion for leveraging big data technologies to solve complex challenges. Strong communicators who can translate technical concepts to non-technical stakeholders will excel in this role. Additionally, candidates must demonstrate a commitment to continuous improvement and innovation within data engineering practices.