Databricks Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by LightFeather, so some details may not match this job. The employer's own page has the real description.

Job Overview

LightFeather is seeking a skilled Databricks Engineer who will be responsible for designing, implementing, and optimizing data pipelines and analytics solutions within the Databricks ecosystem. The ideal candidate will work closely with data scientists, analysts, and other stakeholders to ensure the seamless integration of large data sets into analytical frameworks that facilitate effective business decision-making. The role requires a strong understanding of both the technical and analytical aspects of data engineering, with a focus on performance optimization and best practices in data management.

Job Responsibilities

• Develop and maintain robust data pipelines using Databricks and Apache Spark.
• Collaborate with data scientists and analysts to gather requirements and transform them into technical specifications.
• Optimize data storage and processing workflows for performance and cost efficiency.
• Implement data governance, security, and auditing processes in compliance with organizational policies.
• Monitor and troubleshoot data workflows, ensuring high availability and reliability.
• Create and maintain documentation for data architecture, data lineage, and system integrations.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, or related field.
• 3+ years of experience in data engineering, with a focus on Databricks and Apache Spark.
• Proficiency in SQL and data modeling principles.
• Strong experience with cloud platforms, preferably AWS or Azure.
• Familiarity with ETL processes and tools.
• Knowledge of Python or Scala for data manipulation and automation.
• Excellent problem-solving skills and attention to detail.

Technologies & Tools

The Databricks Engineer position at LightFeather primarily utilizes Databricks as a unified analytics platform, Apache Spark for big data processing, and cloud services like AWS or Azure. Familiarity with data warehousing solutions, data visualization tools (such as Power BI or Tableau), and ETL frameworks will also be beneficial as part of the engineering stack.

Ideal Candidates

Successful candidates for the Databricks Engineer position will possess a blend of technical expertise and interpersonal skills. They should demonstrate a proactive attitude towards problem-solving and continuously seek ways to enhance data processes. A strong team player, the candidate should be able to communicate effectively across different departments, advocate for best practices in data management, and adapt quickly to evolving technologies and data challenges.