Azure Databricks Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Trident Consulting, so some details may not match this job. The employer's own page has the real description.

Job Overview

Trident Consulting is seeking an experienced Azure Databricks Engineer to join our dynamic team. The successful candidate will be responsible for designing, implementing, and optimizing data solutions on the Azure platform using Databricks. This role involves collaborating with data scientists and analysts to create efficient data pipelines, ensuring system performance and data integrity while adhering to industry best practices. The ideal candidate will have a solid background in big data technologies and be passionate about delivering high-quality data solutions that drive business outcomes.

Job Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Data Factory and Databricks.
• Collaborate with data scientists and analysts to understand data requirements and deliver robust data solutions.
• Optimize and troubleshoot existing data workflows for efficiency and performance improvements.
• Conduct data modeling and transformation using Apache Spark within Databricks.
• Implement best practices for data governance, security, and compliance.
• Monitor performance metrics and provide recommendations for enhancements.
• Document system architecture, data flow diagrams, and standard operating procedures of developed solutions.
• Engage with stakeholders to ensure data delivery aligns with business objectives and timelines.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, or a related field.
• Minimum of 3 years’ experience in data engineering or similar roles.
• Extensive experience with Azure services, particularly Azure Databricks and Azure Data Factory.
• Strong proficiency in Spark, Python, and SQL for data processing and analytics.
• Experience with data modeling, ETL processes, and data warehousing concepts.
• Familiarity with cloud computing architecture and best practices.
• Excellent problem-solving skills and attention to detail.
• Capacity to work collaboratively in a team-oriented environment while managing multiple priorities.

Technologies & Tools

The Azure Databricks Engineer will primarily work with a range of technologies and tools including Azure Databricks, Azure Data Factory, Apache Spark, Python, SQL, and Azure SQL Database. Familiarity with data visualization tools such as Power BI, as well as big data processing frameworks and cloud-based data storage solutions, will be advantageous in this role.

Ideal Candidates

The ideal candidates for this position possess a strong analytical mindset, exceptional technical abilities, and a solid understanding of data architecture. They will have a proactive approach to problem-solving and a proven track record of delivering results in fast-paced environments. A commitment to continuous learning and staying updated with the latest technologies in data engineering is essential. Candidates must also demonstrate effective communication skills and the ability to collaborate across various teams to achieve shared goals.