Data Platform Engineer with Azure Databricks and Unity Catalog

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by GeorgiaTEK Systems Inc., so some details may not match this job. The employer's own page has the real description.

Job Overview

The Data Platform Engineer at GeorgiaTEK Systems Inc. will play a crucial role in designing and implementing robust data solutions utilizing Azure Databricks and Unity Catalog. The ideal candidate will collaborate with cross-functional teams to architect data pipelines, optimize data workflows, and ensure the seamless integration of data technologies. Additionally, this role involves the management of data governance and compliance initiatives, ensuring that best practices are upheld throughout all stages of data processing.

Job Responsibilities

• Design and implement scalable data architectures using Azure Databricks and Unity Catalog.
• Develop and optimize ETL pipelines for efficient data ingest, transformation, and storage.
• Collaborate with data scientists and analysts to understand data requirements and deliver solutions that meet business needs.
• Manage data governance policies and ensure compliance with relevant data regulations.
• Monitor and troubleshoot data workflows to maintain system integrity and performance.
• Document data architecture, processes, and best practices for future reference and team onboarding.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Minimum of 3 years of experience in data engineering or data platform roles.
• Proficiency in Azure Databricks, with significant experience in developing and managing data solutions.
• Hands-on experience with Unity Catalog for data governance and data management.
• Strong programming skills in languages such as Python or Scala.
• Familiarity with SQL and data querying techniques.
• Understanding of data warehousing concepts and data lake architectures.

Technologies & Tools

This position primarily utilizes Azure Databricks for data processing and analytics, alongside Unity Catalog for managing data governance. The successful candidate will also engage with various data visualization tools, Azure data storage solutions including Data Lake, and may incorporate additional technologies such as Apache Spark and data orchestration tools like Apache Airflow.

Ideal Candidates

The ideal candidate for the Data Platform Engineer position will possess strong analytical skills, a problem-solving mindset, and an exceptional attention to detail. A successful applicant will be proactive in learning new technologies and best practices in data engineering. Excellent communication and collaboration skills are essential, as the role requires working closely with various teams to deliver comprehensive data solutions. Moreover, candidates should demonstrate a commitment to data quality and governance, ensuring integrity throughout the data lifecycle.