Principal Databricks DevOps Engineer - Remote

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by ChabezTech LLC, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Principal Databricks DevOps Engineer at ChabezTech LLC will play a crucial role in enhancing and maintaining our data engineering platform. This role encompasses designing, implementing, and managing infrastructure solutions and CI/CD pipelines within the Databricks environment. The ideal candidate will be responsible for collaborating with cross-functional teams to streamline operations, optimize workflows, and ensure the efficiency and reliability of data processes, ultimately driving data-driven decision-making across the organization.

Job Responsibilities

• Design, implement, and manage scalable and robust Databricks architecture to support large-scale data processing workloads.
• Develop and maintain CI/CD pipelines for data applications and services deployed in the Databricks environment.
• Collaborate with data engineers, data scientists, and other stakeholders to understand data requirements and provide technical solutions.
• Monitor and optimize performance of Databricks jobs, ensuring minimal downtime and efficient resource utilization.
• Implement security best practices to safeguard data while adhering to compliance requirements.
• Provide technical leadership and mentoring to junior team members in DevOps methodologies and practices.
• Conduct root-cause analysis on production issues and implement corrective actions to prevent future occurrences.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• A minimum of 5 years of experience in DevOps, with a strong background in Databricks and cloud technologies (e.g., AWS, Azure).
• Proficiency in scripting languages such as Python, Bash, or PowerShell.
• Extensive experience with CI/CD tools like Jenkins, GitLab CI, or similar.
• Solid understanding of containerization technologies such as Docker and orchestration platforms like Kubernetes.
• Experience with infrastructure as code (IaC) tools, including Terraform or CloudFormation.
• Strong troubleshooting and problem-solving skills, with a proactive approach to optimizing processes.

Technologies & Tools

The Principal Databricks DevOps Engineer will utilize various technologies and tools including Databricks, Apache Spark, AWS or Azure services, Git version control, Jenkins for CI/CD, Docker for containerization, Kubernetes for orchestration, and Terraform for infrastructure management. Familiarity with monitoring tools such as Datadog or Prometheus will also be essential for ensuring the health and performance of our data platforms.

Ideal Candidates

The ideal candidates for this position will possess a combination of technical expertise and strong interpersonal skills. They should be proactive problem solvers who thrive in a collaborative environment and have a strong commitment to delivering high-quality solutions. Successful candidates will demonstrate a passion for data-driven technology and a continuous desire for learning and professional growth in a fast-paced setting. Strong communication skills and the ability to convey complex technical concepts to diverse teams are also vital for success in this role.