Databricks Data Engineer

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Smart IT Frame LLC, so some details may not match this job. The employer's own page has the real description.

Job Overview

The Databricks Data Engineer at Smart IT Frame LLC will play a pivotal role in designing, building, and maintaining robust data pipelines that integrate various data sources into a centralized Databricks environment. The successful candidate will ensure the efficiency and effectiveness of data processing while collaborating with cross-functional teams to drive data-driven decision-making. This position demands a strong grasp of data engineering principles, cloud technologies, and best practices in data management to support business intelligence and analytics initiatives.

Job Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Optimize data storage and dataflow processes for efficiency and reliability.
• Work collaboratively with data scientists, analysts, and business stakeholders to understand data requirements and deliver practical solutions.
• Implement data quality checks and monitor data pipeline performance for continuous improvement.
• Develop and maintain documentation regarding data architecture and data flows.
• Utilize ETL tools to extract, transform, and load data from various sources into the Databricks environment.
• Stay informed about emerging technologies and best practices in data engineering and analytics.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, or a related field.
• 3+ years of experience in data engineering or a related field.
• Extensive experience with Databricks, Apache Spark, and cloud platforms (AWS, Azure, or GCP).
• Proficiency in programming languages such as Python, Scala, or SQL.
• Hands-on experience with ETL processes and tools (e.g., Apache Airflow, Informatica).
• Familiarity with data warehousing concepts and technologies.
• Strong problem-solving skills and the ability to work under tight deadlines.

Technologies & Tools

The Databricks Data Engineer role utilizes a variety of technologies and tools, including but not limited to Databricks, Apache Spark, AWS/Azure/GCP cloud services, SQL databases, and ETL tools such as Apache Airflow and Informatica. Familiarity with version control systems such as Git, as well as containerization tools like Docker, is also beneficial for effective collaboration and deployment.

Ideal Candidates

The ideal candidates for the Databricks Data Engineer position will exhibit strong analytical and problem-solving skills, with a passion for working with data to create actionable insights. They should have excellent communication abilities to effectively collaborate across teams and articulate complex technical concepts to non-technical stakeholders. A proactive attitude, attention to detail, and a commitment to continuous learning and improvement will make candidates especially suited for success in this role.