Azure databricks Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Genpact, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Azure Databricks Engineer at Genpact will be responsible for designing, developing, and optimizing scalable data pipelines using Azure Databricks. This role demands strong collaboration with cross-functional teams to ensure data accuracy and accessibility, providing actionable insights for informed decision-making. The successful candidate will leverage their expertise in big data technologies, cloud solutions, and data engineering best practices to deliver high-quality data solutions that support the business's strategic initiatives.

Job Responsibilities

• Design and implement robust data pipelines using Azure Databricks to facilitate data ingestion, transformation, and analysis.
• Collaborate with data architects and data scientists to define and refine data models and ensure integration with analytical applications.
• Optimize existing data workflows for performance, reliability, and cost-efficiency within the Azure ecosystem.
• Conduct performance tuning and troubleshooting of Spark jobs and Databricks clusters.
• Maintain comprehensive documentation of data engineering processes, including data lineage and governance practices.
• Implement data security and compliance measures in alignment with organizational policies and industry standards.
• Monitor and manage operational data integrity and quality using Azure tools and frameworks.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, or a related field.
• Proven experience in data engineering, particularly with Azure Databricks and Spark.
• In-depth knowledge of Azure services, including Azure Data Lake Storage, Azure SQL Database, and Azure Data Factory.
• Familiarity with programming languages such as Python, Scala, or SQL.
• Strong understanding of data modeling concepts and ETL/ELT processes.
• Experience with version control systems (e.g., Git) and agile development methodologies.
• Excellent problem-solving skills with attention to detail and ability to work under tight deadlines.

Technologies & Tools

The Azure Databricks Engineer role utilizes a range of technologies and tools to effectively manage and process data. Key technologies include Azure Databricks for data analytics and processing, Azure Data Lake Storage for scalable storage solutions, and Azure SQL Database for reliable data management. Familiarity with Apache Spark is crucial for running large-scale data processing jobs. Additional tools may include Azure Data Factory for data orchestration and monitoring tools to ensure data integrity and performance.

Ideal Candidates

Ideal candidates for the Azure Databricks Engineer position will possess strong analytical and technical skills, coupled with a proactive approach to problem-solving. They should demonstrate exceptional communication abilities and the capability to collaborate effectively within diverse teams. An enthusiasm for leveraging data to drive business value, along with a commitment to maintaining high standards in data quality and governance, will make a candidate stand out. Additionally, adaptability to changing technologies and a continuous learning mindset are vital traits that align with our dynamic work environment at Genpact.