Data Platform Engineer with Databricks

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Indus Valley Consultants, so some details may not match this job. The employer's own page has the real description.

Job Overview

Indus Valley Consultants is seeking a highly skilled Data Platform Engineer with expertise in Databricks to join our dynamic team. In this role, you will be responsible for designing, developing, and optimizing data pipelines and platforms to ensure seamless data flow and insights generation. You will collaborate closely with data scientists and analysts, driving innovative data solutions that leverage the full capabilities of the Databricks platform while adhering to industry best practices.

Job Responsibilities

• Design, implement, and manage scalable data pipelines and data warehouses using Databricks and Apache Spark.
• Collaborate with cross-functional teams to gather data requirements and translate them into technical specifications.
• Ensure data quality and integrity by implementing effective testing and monitoring strategies.
• Optimize performance and cost efficiency of Databricks workloads and query execution.
• Develop and maintain documentation for data architecture, engineering processes, and data governance policies.
• Stay updated with the latest trends and technologies in data engineering and Databricks for continual improvement of services.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, Mathematics, or related field; Master's degree preferred.
• Proven experience with Databricks, Apache Spark, and data engineering concepts.
• Solid understanding of data modeling, ETL processes, and data warehousing principles.
• Proficiency in programming languages such as Python, Scala, or SQL.
• Experience with cloud services (AWS, Azure, or GCP) and Big Data technologies.
• Strong problem-solving skills and ability to work independently and in teams.

Technologies & Tools

The Data Platform Engineer will utilize a variety of technologies and tools essential for data processing and analytics. This includes Databricks for data analytics and machine learning, Apache Spark for big data processing, SQL for database management, and various cloud services such as AWS or Azure for data storage and orchestration purposes. Familiarity with tools like Airflow or similar orchestration tools is also beneficial.

Ideal Candidates

The ideal candidate for the Data Platform Engineer position should possess a strong analytical mindset and a proactive approach to problem-solving. They should be detail-oriented and have a passion for data engineering and analytics. Excellent communication skills are crucial as the role requires collaboration with various stakeholders. A desire to stay informed about industry advancements and a commitment to continuous learning will also make someone stand out in this position.