Platform Architect with Databricks experience and Retail experience

Role: Platform Architect with Databricks experience and Retail experience

Location:- Remote in US

Duration: Long term Contract

Job description:

JD:

Key Roles & Responsibilities:

1. Azure Databricks Service Management & Optimization:

• Design & Deployment: Architect and oversee the deployment of Azure Databricks workspaces, including VNet Injection, secure cluster connectivity (no public IP), and private endpoints.

• Configuration Management: Define and manage Databricks cluster policies, instance types, Databricks Runtime (DBR) versions, and auto-scaling configurations.

• Performance Tuning: Monitor and optimize Databricks workspace and job performance, advising on cluster sizing, Spark configurations, and Delta Lake optimizations.

• Capacity Planning: Forecast and plan for future Databricks resource needs based on platform usage trends.

• Cost Optimization: Identify and implement strategies to optimize Azure Databricks costs (e.g., proper cluster sizing, auto-termination, spot instances, reserved instances).

• Troubleshooting: Provide expert-level troubleshooting for complex Databricks issues, spanning cluster failures, job errors, performance bottlenecks, and connectivity problems.

2. Data Governance & Unity Catalog:

Unity Catalog Design: Architect and implement the Unity Catalog metastore, including the setup of catalogs, schemas, volumes, and external locations.

Fine-Grained Access Control (FGAC): Define and enforce data access policies within Unity Catalog, securing tables, views, and managed objects at row and column levels where required.

Data Lineage & Audit: Leverage Unity Catalog capabilities for data lineage tracking and support auditing requirements for data access.

Managed Tables & Volumes: Advise and govern the usage of managed and external tables/volumes within Unity Catalog.

3. Security & Access Management (Entra ID, SCIM, Key Vault, ACLs):

*Azure Entra ID Integration: Design and implement seamless integration of Azure Databricks with Azure Entra ID (formerly Azure Active Directory) for Single Sign-On (SSO) and user/group synchronization.

* SCIM (System for Cross-domain Identity Management): Configure and manage SCIM provisioning for automated user and group lifecycle management between Entra ID and Databricks.

* Azure Key Vault Integration: Architect and implement secure secret management for Databricks using Azure Key Vault-backed secret scopes for notebooks, jobs, and applications. Advise on credential passthrough configurations.

* Access Control Lists (ACLs): Define and enforce ACLs on Azure Data Lake Storage Gen2 (ADLS Gen2) for Databricks service principals and user access. Implement table-level, view-level, and object-level ACLs within Unity Catalog and non-Unity Catalog environments.

* Compliance & Auditing: Ensure Databricks configurations meet organizational security and compliance standards. Assist in auditing access logs and activity.

4. Networking & Connectivity:

* Secure Network Design: Architect and implement secure network connectivity for Azure Databricks, including VNet peering, private endpoints, and network security groups (NSGs).

* Firewall Rules: Define and manage firewall rules to control inbound and outbound traffic for Databricks workspaces, ensuring secure communication with other Azure services and on-premises resources.

* DNS & Routing: Ensure proper DNS resolution and routing for Databricks components and integrated services.

5. Data Integration & Orchestration (Azure Data Factory - ADF):

* ADF Integration Patterns: Design and implement robust integration patterns between Azure Data Factory (ADF) and Azure Databricks, including notebook activity execution, data movement, and orchestrating complex data pipelines.

* Automated Deployments: Work with DevOps teams to define and implement CI/CD pipelines for Databricks assets, notebooks, jobs, and configurations using ADF or Azure DevOps.

* API Management: Leverage Databricks REST APIs for automation and integration with other enterprise systems as needed.

6. Azure Storage (ADLS Gen2) Integration:

* Storage Account Design: Advise on the design and optimal configuration of Azure Data Lake Storage Gen2 accounts for Databricks workloads, including hierarchical namespaces, access tiers, and lifecycle management policies.

* Mount Points & Credential Passthrough: Guide and troubleshoot secure access patterns to ADLS Gen2 from Databricks, including mount points (legacy) and direct access via Entra ID credential passthrough or service principals.

7. Operations, Monitoring & Reliability:

* Monitoring & Alerting: Implement comprehensive monitoring for Databricks workspaces using Azure Monitor, Log Analytics, and Databricks native logging, setting up alerts for critical events (e.g., cluster failures, job errors, resource utilization).

* Disaster Recovery & Business Continuity: Contribute to and validate disaster recovery and business continuity plans for the Databricks platform.

* Patching & Upgrades: Plan and oversee Databricks Runtime (DBR) upgrades and other platform-level maintenance activities.

* Service Health: Monitor Azure service health and ensure proactive communication of any known issues impacting Databricks.

8. Documentation & Best Practices:

* Architectural Documentation: Create and maintain comprehensive architectural diagrams, design documents, and operational runbooks for the Databricks platform.

* Best Practices: Define and evangelize best practices for Databricks usage, security, cost optimization, and performance to data engineering and data science teams.

* Knowledge Transfer: Provide training and guidance to internal teams on Databricks capabilities, new features, and platform usage.

9. Collaboration & Stakeholder Management:

* Collaborate closely with data engineers, data scientists, security teams, network teams, and operations teams to align on architectural decisions and resolve issues.

* Act as a subject matter expert for all Databricks-related queries from internal teams.

* Participate in architectural review boards and provide technical leadership.

Key Skills & Qualifications:

• Deep Expertise in Azure Services: Azure Databricks, Azure Data Factory, Azure Data Lake Storage Gen2, Azure Key Vault, Azure Networking (VNets, NSGs, Private Endpoints), Azure Entra ID (formerly Azure AD), Azure Monitor, Log Analytics.

• Databricks Specifics: Unity Catalog, Delta Lake, Spark architecture, Databricks Runtime, Workspaces, Clusters, Jobs, Notebooks, Repos, Databricks APIs.

• Security: Strong understanding of cloud security principles, identity and access management (IAM), role-based access control (RBAC), data encryption, network security. Experience with SCIM.

• Networking: Solid understanding of TCP/IP, VPNs, routing, firewalls, and secure network design in Azure.

• Scripting/Automation: Proficiency in Python, Scala, SQL, and PowerShell/Azure CLI for automation and management tasks.

• CI/CD: Experience with CI/CD pipelines for cloud resources and Databricks artifacts (e.g., Azure DevOps, GitHub Actions).

• Problem-Solving: Excellent analytical and problem-solving skills, with the ability to troubleshoot complex, multi-service issues.

• Communication: Strong verbal and written communication skills to articulate complex technical concepts to both technical and non-technical audiences.

• Certifications (Preferred): Microsoft Certified: Azure Solutions Architect Expert, Databricks Certified Data Engineer Professional/Associate.

Authorization: Applicants must be legally authorized to work in the United States.

Regards,

_______________________

Parthasarathy K

Lead Recruiter

Work: 972-474-8787 Ext: 306,Direct: 972-737-8607

[email protected]

Themesoft Inc |Themesoft Jobs