Databricks Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by LightFeather, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Databricks Engineer at LightFeather will be responsible for designing, implementing, and maintaining data processing pipelines using Databricks. This role requires a deep understanding of big data technology, cloud computing, and data analytics to enhance the overall data ecosystem. The engineer will collaborate with software developers, data scientists, and business stakeholders to ensure effective data solutions that meet organizational needs and help drive data-driven decision-making.

Job Responsibilities

• Design and implement robust and scalable ETL pipelines using Databricks and Apache Spark.
• Collaborate with cross-functional teams to gather requirements and define data strategies.
• Optimize performance of data processing workflows to ensure efficiency and cost-effectiveness.
• Monitor and troubleshoot data ingestion, transformation, and storage processes.
• Maintain data quality and integrity across data lakes and warehouses.
• Provide technical guidance and support for Databricks-related queries and issues.
• Document data engineering processes and maintain architectural diagrams.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, and/or a related field.
• 3+ years of experience in data engineering or related roles, specifically utilizing Databricks and Apache Spark.
• Proficient in programming languages such as Python and SQL.
• Familiarity with cloud platforms, especially AWS, Azure, or GCP.
• Strong understanding of data modeling, ETL processes, and data warehousing.
• Experience with data visualization tools such as Tableau or Power BI is a plus.
• Excellent problem-solving skills and the ability to work in a fast-paced environment.

Technologies & Tools

The Databricks Engineer will utilize a variety of technologies and tools to ensure seamless data processing and analytics. Key technologies include Databricks, Apache Spark, cloud platforms such as AWS, Azure, or GCP, as well as programming languages such as Python and SQL. Additional tools may include Apache Kafka for stream processing, as well as data visualization software like Tableau or Power BI for reporting and insights.

Ideal Candidates

The ideal candidate for the Databricks Engineer position will possess strong analytical and technical capabilities, accompanied by a proactive problem-solving attitude. They should be detail-oriented, capable of managing multiple tasks simultaneously while maintaining a high standard of quality. Excellent communication skills are essential for collaborating effectively with diverse teams and stakeholders, alongside a commitment to continuous improvement and learning within the dynamic field of data engineering.