Databricks Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Illumination Works, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Databricks Engineer at Illumination Works plays a pivotal role in designing, developing, and maintaining large-scale data processing solutions leveraging the Databricks platform. This position requires a thorough understanding of data engineering principles, strong analytical skills, and the ability to collaborate effectively within cross-functional teams. The successful candidate will be responsible for optimizing data pipelines, ensuring data quality, and enabling data-driven decision-making across the organization.

Job Responsibilities

• Design and implement scalable data pipelines using Databricks, ensuring efficient processing of large datasets.
• Develop and optimize ETL processes, utilizing Spark and Delta Lake capabilities.
• Work collaboratively with data scientists and analysts to facilitate data accessibility and usability.
• Implement and monitor data quality checks to maintain high standards of integrity and accuracy.
• Participate in architectural discussions and promote best practices in data engineering.
• Provide support for data operations, including troubleshooting and performance tuning of data workflows.
• Document data pipeline processes and maintain updated technical documentation for future reference.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• 3+ years of experience in data engineering or related roles, preferably with Databricks.
• Strong proficiency in Apache Spark and hands-on experience implementing Spark applications.
• Experience with SQL and scripting languages such as Python or Scala.
• Familiarity with cloud platforms such as AWS, Azure, or GCP.
• Understanding of data warehousing concepts and tools.
• Strong analytical and problem-solving skills, with the ability to troubleshoot complex data issues.

Technologies & Tools

The Databricks Engineer will primarily work with the Databricks platform, leveraging its capabilities in Apache Spark and Delta Lake for data processing tasks. The role also involves using various tools and technologies, including SQL, Python, and cloud services such as AWS or Azure, to develop and maintain efficient data pipelines. Familiarity with associated data tools and technologies is essential for effective collaboration and performance in this position.

Ideal Candidates

The ideal candidate for the Databricks Engineer position at Illumination Works possesses a strong passion for data engineering and is driven by a desire to solve complex data challenges. They exhibit a collaborative mindset, thriving in team-oriented environments while also demonstrating independence in their work. Strong communication skills are essential, as the role requires interfacing with various stakeholders across departments. A commitment to quality and continuous improvement, combined with a curiosity for exploring new technologies and methodologies, will further enhance their fit for this role.