Job Description

Mandatory Technical Skills

Databricks Lakehouse Platform
Delta Lake, Delta Live Tables (DLT)
Unity Catalog
Databricks Workflows
Auto Loader
PySpark, Spark SQL, Python
AWS (S3, IAM, Glue, Redshift, EMR, Lambda)
Apache Airflow
CDC & Data Migration Frameworks
Data Governance & Security
CI/CD, GitHub, Jenkins

JD:

Lead enterprise-scale Databricks Lakehouse Architecture design and implementation on AWS.
Drive large-scale data platform modernisation and cloud transformation initiatives.
Architect scalable Medallion Architecture (Bronze, Silver, Gold) data platforms.
Lead migration of legacy EMR, NiFi, Redshift, and ETL workloads to Databricks.
Design high-performance batch and real-time data processing solutions.
Build robust ingestion frameworks using Auto Loader, Delta Lake, and Structured Streaming.
Define enterprise data governance and security standards using Unity Catalog.
Architect metadata-driven and reusable PySpark-based ETL/ELT frameworks.
Establish best practices for performance tuning, scalability, and cost optimisation.
Design and implement data quality, lineage, and observability frameworks.
Drive adoption of CI/CD, DevOps, Infrastructure as Code, and automation practices.
Collaborate with business, analytics, and engineering teams to define target-state architectures.
Conduct architecture reviews and provide technical leadership across multiple projects.
Mentor architects and senior engineers on Databricks and AWS best practices.
Design secure and scalable solutions leveraging AWS services (S3, Glue, Athena, Lambda, IAM, Redshift).
Implement and govern enterprise-wide data access, compliance, and security controls.
Evaluate and adopt latest Databricks capabilities such as DLT, Lakehouse Federation, Serverless, and MLflow.
Enable AI/ML, Generative AI, and advanced analytics use cases on the Lakehouse platform.
Create architecture roadmaps, migration strategies, standards, and governance frameworks.
Act as the primary technical advisor for customer leadership on data strategy and platform evolution.