Data Engineer - Databricks

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by 11:59, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Data Engineer - Databricks at 11:59 will play a crucial role in designing and implementing scalable data pipelines and architectures to support analytics and reporting initiatives. The ideal candidate will work collaboratively with data scientists, analysts, and other stakeholders to ensure data integrity and accessibility across the organization. Responsibilities also include optimizing the data flow and taking an active part in evaluating new data technologies, resulting in improved data solutions and overall performance.

Job Responsibilities

• Design and develop data pipelines using Databricks to ensure seamless data integration from various sources.
• Collaborate with data scientists and analysts to define data requirements and access needs.
• Implement and maintain data quality frameworks and best practices.
• Monitor and optimize data workflows for performance and scalability.
• Evaluate and integrate new data technologies and tools into existing workflows.
• Document data models and maintain comprehensive technical documentation.
• Ensure compliance with data governance and security policies.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 3+ years of experience in data engineering, with a focus on Databricks or similar platforms.
• Strong programming skills in Python, SQL, or Scala.
• Proficiency in cloud data services, particularly Azure or AWS.
• Experience in building and optimizing ETL processes.
• Familiarity with big data technologies such as Spark, Hadoop, or Kafka.
• Excellent analytical and problem-solving skills.

Technologies & Tools

The Data Engineer - Databricks will predominantly work with Databricks, Apache Spark, and various cloud platforms like AWS or Azure. Additional tools may include SQL databases, ETL tools, cluster management systems, and data visualization software to support data processing and analytical tasks.

Ideal Candidates

The ideal candidate for the Data Engineer - Databricks position will possess strong analytical capabilities and a profound interest in data-driven decision-making. Exceptional attention to detail, combined with effective communication skills and a collaborative spirit, are essential for interacting with diverse teams. The candidate should also demonstrate a passion for learning new technologies and embracing innovative solutions to improve data systems.