Azure databricks Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Genpact, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Azure Databricks Engineer at Genpact will play a crucial role in designing, implementing, and optimizing data pipelines and analytics solutions leveraging Azure Databricks. This position requires significant collaboration with data scientists, analysts, and business stakeholders to deliver business insights from complex data sets. The engineer will be responsible for ensuring data integrity, performance of applications, and the overall quality of data solutions. Furthermore, the candidate will engage in the migration of data from on-premise environments to the Azure cloud, ensuring best practices are followed in data management, governance, and security.

Job Responsibilities

• Design and implement scalable data pipelines using Azure Databricks.
• Collaborate with cross-functional teams to gather and analyze business requirements.
• Optimize existing ETL processes to improve performance and reduce operational costs.
• Utilize Spark and Databricks notebooks for data processing and exploratory data analysis.
• Monitor and troubleshoot data workflows to ensure data quality and consistency.
• Participate in code review processes and contribute to best practices for data engineering.
• Support data migration from on-premise solutions to Azure cloud environments.
• Document data architecture, pipelines, and processes for ongoing support and future reference.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Proven experience working with Azure Data Services (e.g., Azure SQL, Azure Data Lake, Azure Databricks).
• Strong programming skills in Scala, Python, or SQL.
• Experience with Apache Spark and big data processing frameworks.
• Knowledge of data warehousing concepts and methodologies.
• Familiarity with data governance and data quality best practices.
• Excellent problem-solving skills and ability to work in a collaborative environment.
• Strong communication skills to convey complex technical concepts to non-technical stakeholders.

Technologies & Tools

In this role, the Azure Databricks Engineer will utilize a variety of technologies and tools, including Azure Databricks for big data analytics, Azure Data Lake Storage for data storage, Apache Spark for data processing, and various Azure services for orchestration and management. Additionally, proficiency in using version control systems, such as Git, and familiarity with CI/CD practices will be essential for maintaining code quality and facilitating deployments.

Ideal Candidates

The ideal candidate for the Azure Databricks Engineer position will demonstrate a strong analytical mindset combined with a proactive attitude towards problem-solving. They should possess a deep desire to continuously learn about new technologies and trends in the data engineering field. Successful candidates will exhibit excellent teamwork skills, often collaborating with various stakeholders to deliver effective solutions. Additionally, the ability to articulate complex technical information clearly and a keen attention to detail will be vital for success in this role.