100% Remote // DataBricks Engineer (PySpark, Python)

This job was posted over 30 days ago, so it has most likely closed. See newer jobs like it.

Written by AI from the job title and company, not by Talent Groups, so some details may not match this job. The employer's own page has the real description.

Job Overview

The DataBricks Engineer (PySpark, Python) at Talent Groups will play a critical role in the design, development, and implementation of data processing solutions using the DataBricks platform. The selected candidate will collaborate with data scientists, analysts, and other engineers to build data pipelines and optimize existing workflows for scalability and performance. The role demands strong programming skills in Python, a deep understanding of PySpark, and the ability to translate business requirements into technical specifications while ensuring data integrity and quality.

Job Responsibilities

• Design and develop scalable data processing solutions using DataBricks and PySpark.
• Collaborate with cross-functional teams to understand data requirements and translate them into efficient data models.
• Optimize existing data pipelines for improved performance and reliability.
• Implement data validation and quality checks to ensure the integrity of data as it moves through processing stages.
• Conduct performance tuning and debugging of data pipelines to enhance efficiency.
• Document all processes and maintain best practices for data management and workflows.
• Provide technical support and troubleshooting for data-related issues as needed.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field.
• A minimum of 3 years of experience in data engineering or a related field.
• Strong programming skills in Python and experience with the PySpark framework.
• Proficient in ETL processes and data warehousing concepts.
• Experience with data modeling and SQL-based technologies.
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud is a plus.
• Excellent problem-solving abilities and attention to detail.
• Strong communication skills, with the ability to work effectively in a remote team environment.

Technologies & Tools

The DataBricks Engineer will work extensively with technologies and tools such as DataBricks, PySpark, Apache Spark, Python, SQL, and cloud services including AWS, Azure, or Google Cloud. Familiarity with version control systems like Git and data visualization tools is also beneficial.

Ideal Candidates

The ideal candidate is a proactive and analytical thinker with a passion for turning raw data into actionable insights. They possess a strong technical background and a commitment to continuous improvement and learning. A successful DataBricks Engineer will be someone who thrives in a collaborative remote environment, demonstrating both initiative and adaptability as they engage with various stakeholders to deliver high-quality data solutions.