Senior Data Engineer / Databricks / Terraform

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Motion Recruitment, so some details may not match this job. The employer's own page has the real description.

Overview

As a Senior Data Engineer at Motion Recruitment, you will play a pivotal role in designing, implementing, and maintaining data pipelines that result in actionable insights across the organization. You will leverage Databricks and Terraform to build robust data solutions, facilitate data management strategies, and collaborate with cross-functional teams to ensure data quality and accessibility. Your expertise will be instrumental in driving data-driven decision-making processes that support business objectives.

Job Responsibilities

• Design and develop scalable data pipelines using Databricks to support analytics and data science initiatives.
• Utilize Terraform for infrastructure as code (IaC) practices, automating the deployment and management of data solutions.
• Collaborate with data scientists, analysts, and other stakeholders to define data requirements and translate them into technical specifications.
• Ensure data ingestion processes are efficient, reliable, and well-documented, maintaining code quality and best practices.
• Monitor and optimize data pipelines for performance, ensuring timely processing and availability of data.
• Implement data governance practices to ensure compliance, security, and quality of data across various platforms.
• Conduct code reviews and mentor junior data engineers, fostering a culture of continuous improvement and learning.
• Stay abreast of industry trends, tools, and techniques to continuously enhance our data engineering capabilities.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, or a related field; Master’s degree is a plus.
• Minimum of 5 years of experience in data engineering, with a focus on cloud-based technologies.
• Proficient in utilizing Databricks and Terraform in a production environment.
• Strong programming skills in languages such as Python, SQL, or Scala.
• Experience with data warehousing solutions, ETL processes, and data architecture best practices.
• Familiarity with data visualization tools and analytics platforms.
• Excellent problem-solving skills and the ability to work collaboratively in a team environment.
• Strong communication skills, both verbal and written.

Technologies & Tools

The Senior Data Engineer will work primarily with Databricks for data processing and analytics, utilizing Terraform for infrastructure management. Familiarity with AWS or Azure cloud services is essential, along with exposure to a range of database technologies including SQL and NoSQL databases. Additional tools that may be utilized include Apache Spark, Kafka, and various data visualization platforms.

Ideal Candidates

The ideal candidate for this position is a self-motivated and detail-oriented professional who thrives in a dynamic environment. They should possess a strong analytical mindset, with a passion for data and an eagerness to solve complex problems. Excellent interpersonal skills and the ability to convey technical concepts to non-technical stakeholders are essential. A proactive attitude towards continuous learning and adaptation to new technologies will also contribute to success in this role.