Job Description

Job Summary

We are seeking a Data Engineer with 5+ years of experience in designing, developing, and optimizing scalable data pipelines and cloud-based data platforms. The ideal candidate will have strong expertise in Python, SQL, Databricks, Apache Spark, and ETL/ELT development, along with hands-on experience in at least one cloud platform (AWS, Azure, or Google Cloud Platform).

Key Responsibilities

Design, develop, and maintain scalable ETL/ELT data pipelines.
Build and optimize data ingestion, transformation, and integration workflows using Databricks and Apache Spark.
Develop and maintain data lakes and cloud-based data warehouses.
Create scalable batch and real-time data processing solutions.
Optimize data pipelines for performance, reliability, and scalability.
Develop reusable data engineering frameworks and automation solutions.
Collaborate with data analysts, data scientists, and business stakeholders to deliver high-quality data solutions.
Implement data quality, governance, monitoring, and security best practices.
Troubleshoot production issues and continuously improve data platform performance.
Follow DevOps and CI/CD best practices for data engineering projects.

Required Qualifications

Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
5+ years of experience as a Data Engineer.
Strong programming skills in Python.
Advanced SQL proficiency.
Hands-on experience with Databricks.
Strong experience with Apache Spark (PySpark preferred).
Experience building and maintaining ETL/ELT pipelines.
Experience with Apache Airflow or similar workflow orchestration tools.
Experience with data warehousing technologies such as Snowflake, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics.
Hands-on experience with at least one cloud platform (AWS, Azure, or Google Cloud Platform).
Experience with Git, Docker, and CI/CD pipelines.
Knowledge of relational and NoSQL databases.

Preferred Qualifications