Senior Azure Databricks Architect

Job Summary

We are seeking a Senior Azure Databricks Architect / Data & AI Architect to lead enterprise Data and AI platform modernization. You will design scalable Lakehouse architectures, engineer real-time pipelines, establish data governance, and deploy modern AI solutions natively on Microsoft Azure.

Key Responsibilities

• Lakehouse Architecture: Design scalable Azure Databricks Lakehouse, streaming, and data engineering architectures using Delta Lake and Apache Iceberg interoperability.
• Pipeline Modernization: Build declarative data pipelines utilizing Lakeflow Designer, Lakeflow Connect, and Lakeflow Declarative Pipelines.
• Governance & Security: Implement enterprise governance, automated lineage, and RBAC/ABAC access controls via Unity Catalog across Azure storage.
• AI & Analytics: Architect GenAI, RAG, and agentic workflows using Unity AI Gateway, Vector Search, Databricks Genie / Genie Code, and Agent Bricks.
• Architecture Governance: Lead technical design reviews, optimize Azure compute/storage costs (FinOps), and establish engineering standards.

Required Technical Skills

• Databricks Platform: Lakeflow (Connect / Designer / Declarative Pipelines), Unity Catalog, Delta Lake / Iceberg, Genie Code, Agent Bricks, Managed Agent Memory, Unity AI Gateway, Vector Search, MLflow.
• Azure Cloud: Azure Databricks, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory (ADF), Microsoft Entra ID (SCIM), Azure Key Vault, VNet injection / Private Link.
• Engineering & Streaming: PySpark, Apache Spark Structured Streaming, Python, SQL, Databricks Asset Bundles (DABs), CI/CD.

Qualifications

• 10+ years in Data Engineering, Cloud Architecture, or Enterprise Solution Architecture.
• 5+ years hands-on Databricks architecture experience, including modern platform features.
• Strong production experience building and securing platforms natively on Microsoft Azure.
• Relevant certifications preferred (Databricks Certified Solutions Architect / Data Engineer Professional, Azure Solutions Architect Expert).