DevOps Engineer - Databricks Expert

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Aroha Technologies, Inc, so some details may not match this job. The employer's own page has the real description.

Overview

The DevOps Engineer - Databricks Expert at Aroha Technologies, Inc. is responsible for implementing and maintaining complex data pipelines and workflows within the Databricks environment. This role involves collaborating with cross-functional teams to ensure seamless integration of software development and operational practices, optimizing the architecture, and ensuring the efficient utilization of resources. The ideal candidate will be pivotal in enhancing our cloud-based analytics capabilities while continuously improving system reliability and deployment speed.

Job Responsibilities

• Design, implement, and maintain CI/CD pipelines for Databricks environments.
• Collaborate with data engineers and data scientists to optimize data solutions and improve performance.
• Monitor and troubleshoot existing Databricks jobs to ensure performance and reliability.
• Automate routine tasks to enhance operational efficiency and reduce manual workload.
• Manage and optimize large-scale data processing and analytics pipelines in Azure Data Lake and Databricks.
• Provide expertise in SQL and Spark for advanced analytics and data transformation activities.
• Develop and maintain infrastructure as code (IaC) solutions for Databricks deployments.
• Ensure best practices for security, data governance, and compliance are implemented and maintained.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Minimum of 3 years experience in a DevOps or data engineering role.
• Proficient with Databricks, Apache Spark, and cloud platforms (preferably Azure).
• Strong experience in managing CI/CD pipelines.
• Solid understanding of containerization technologies like Docker and orchestration tools such as Kubernetes.
• Excellent scripting skills, preferably in Python, Scala, or similar languages.
• Demonstrated experience with SQL and NoSQL databases.
• Familiarity with infrastructure automation tools (e.g., Terraform, Ansible).

Technologies & Tools

The DevOps Engineer - Databricks Expert will utilize a variety of cutting-edge tools and technologies, including Databricks, Apache Spark, Azure Data Lake, Terraform, Jenkins, Docker, Kubernetes, and an array of monitoring and performance-tuning tools. Familiarity with modern data analytics frameworks and cloud services will be essential to successfully fulfill the responsibilities of this position.

Ideal Candidates

The ideal candidate for this position should demonstrate strong analytical and problem-solving skills, with a commitment to continuous improvement. An effective communicator, the candidate should be able to collaborate well within a team and engage with cross-functional stakeholders. A passion for data engineering and a drive to optimize processes would make a candidate stand out. Additionally, adaptability and a willingness to learn new technologies and methodologies in the dynamic field of data analytics are highly valued traits.