Databricks Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by LightFeather, so some details may not match this job. The employer's own page has the real description.

Job Overview

LightFeather is seeking a skilled Databricks Engineer to join our dynamic team. In this role, you will be responsible for designing, developing, and implementing scalable and efficient data solutions on the Databricks platform. You will work closely with data scientists, analysts, and other stakeholders to understand data requirements and ensure that data pipelines meet the organization's analytics needs. Additionally, you will optimize existing data workflows and ensure data integrity across various systems, thus contributing to enhanced decision-making processes within the organization.

Job Responsibilities

• Design and implement data pipelines utilizing Databricks and Apache Spark.
• Collaborate with cross-functional teams to gather and define data requirements.
• Optimize data workflows to improve performance and reduce costs.
• Integrate data from various sources and ensure data integrity.
• Monitor and troubleshoot data pipeline issues to maintain smooth operations.
• Develop and maintain documentation for data models, pipelines, and procedures.
• Conduct thorough testing and validation of data solutions before deployment.
• Stay updated with the Databricks ecosystem and implement best practices.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Minimum of 3 years of experience working with Databricks and Apache Spark.
• Proficiency in SQL and experience with relational databases.
• Knowledge of cloud platforms such as AWS, Azure, or Google Cloud.
• Familiarity with data modeling concepts and ETL processes.
• Strong programming skills in Python or Scala.
• Excellent problem-solving and analytical skills.
• Experience with data visualization tools (e.g., Tableau, Power BI) is a plus.

Technologies & Tools

The Databricks Engineer will primarily work with Databricks, Apache Spark, and various cloud services such as AWS or Azure. Additionally, proficiency in SQL, Python or Scala programming languages, and familiarity with data orchestration tools will be required. The role may also involve utilizing data storage solutions, such as data lakes and various relational databases, along with data visualization platforms.

Ideal Candidates

The ideal candidate will exhibit strong analytical and technical skills, with a proven track record in data engineering within cloud environments. They should possess excellent communication abilities to facilitate collaboration with various teams. A proactive mindset, attention to detail, and a passion for data-driven decision making will also characterize a successful candidate. Furthermore, adaptability to changing technologies and a commitment to continuous improvement are essential attributes for this role.