Databricks Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by LightFeather, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Databricks Engineer at LightFeather will play a pivotal role in designing, implementing, and maintaining data solutions using the Databricks platform. The successful candidate will collaborate with cross-functional teams to develop frameworks that enhance data processing and analytics. This position necessitates a deep understanding of big data technologies and methodologies in order to provide scalable data pipelines, efficient data lakes, and robust analytical models. The Databricks Engineer will also ensure the integrity and security of data across all environments.

Job Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks and Apache Spark.
• Collaborate with data scientists and analysts to develop and optimize data models and analytics solutions.
• Implement data governance practices to ensure data quality and compliance.
• Monitor and troubleshoot performance issues within the Databricks environment.
• Conduct code reviews and provide mentorship to junior team members.
• Document data architecture and processes for team reference and scalability.
• Stay updated on industry best practices and emerging technologies in big data and analytics.

Qualifications

• A bachelor’s degree in Computer Science, Engineering, Information Technology, or related field.
• 3+ years of experience in data engineering, with a proven record of using Databricks and Apache Spark.
• Strong proficiency in programming languages such as Python, Scala, or Java.
• Experience with SQL and NoSQL databases.
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud.
• Understanding of data warehousing and ETL processes.
• Excellent problem-solving skills with attention to detail.
• Strong communication skills and the ability to work collaboratively in a team environment.

Technologies & Tools

The Databricks Engineer will work with an array of technologies and tools, including but not limited to Databricks, Apache Spark, Python, SQL, Jupyter notebooks, Airflow for scheduling tasks, and various cloud services such as AWS or Azure. Familiarity with DevOps tools and CI/CD practices will also be beneficial for facilitating smooth deployment processes.

Ideal Candidates

Ideal candidates for the Databricks Engineer position will possess a keen analytical mindset, strong technical expertise, and an innovative approach to problem-solving. They should be proactive in seeking continuous improvement and optimizing data workflows. A candidate who thrives in fast-paced environments and can adapt to changing priorities while maintaining a focus on delivering high-quality work will be a great fit. Strong interpersonal skills are essential to effectively collaborate with diverse teams and communicate technical concepts to non-technical stakeholders.