Senior Databricks Solution Architect

Senior Databricks Solution Architect

Azure | Lakehouse Architecture | Security | AI/BI | Agentic AI

Location: Houston, Texas

Company: Frame Data & AI

Role Type: Full-Time | Client-Facing | Hands-On Architecture

About Frame

Frame is a Houston-based Data & AI consulting firm focused on engineering modern data platforms and intelligent solutions for Energy, Industrial, and complex enterprise environments.

We combine senior-level consulting expertise, hands-on engineering, and modern delivery approaches to help clients establish trusted data foundations and move AI solutions into production.

The Opportunity

We are seeking a Senior Databricks Solution Architect who can take ownership of the complete architecture lifecycle, from initial client discovery and solution design through infrastructure deployment, data integration, governance, performance optimization, and production readiness.

This is not a traditional data engineering role.

You will be responsible for designing Azure Databricks environments from scratch, making critical architectural decisions, establishing secure and scalable foundations, and enabling advanced analytics and AI capabilities.

The right candidate can lead technical discovery with a client, whiteboard the architecture, defend design decisions, estimate the implementation, and guide engineering teams through delivery.

Key Responsibilities

1. Azure & Databricks Platform Architecture

• Architect and implement Azure Databricks environments from the ground up, including greenfield deployments.
• Design Azure landing zones, subscriptions, resource groups, networking, storage, identity, and platform connectivity.
• Establish development, testing, and production environments with appropriate isolation and governance.
• Design workspace architecture, compute strategies, serverless adoption, and workload isolation.
• Define architectural standards for scalability, reliability, availability, and disaster recovery.
• Develop architecture diagrams, technical design documents, deployment patterns, and implementation roadmaps.

2. Security, Infrastructure & Governance

• Architect secure Azure Databricks deployments using Private Link, network isolation, secure connectivity, and appropriate network controls.
• Design identity and access management using Microsoft Entra ID, service principals, managed identities, and role-based access controls.
• Implement Unity Catalog governance, including catalogs, schemas, permissions, data lineage, and controlled data access.
• Configure storage credentials, external locations, managed storage, and secure data access patterns.
• Design encryption, secrets management, audit logging, and compliance controls.
• Establish row-level security, column masking, and data classification strategies.
• Design secure connectivity between Azure Databricks, enterprise applications, cloud services, and on-premises environments.

3. Lakehouse & Medallion Architecture

• Architect enterprise Lakehouse solutions using Delta Lake and Bronze, Silver, and Gold data layers.
• Design data ingestion, cleansing, standardization, transformation, and consumption patterns.
• Establish data modeling strategies supporting analytics, operational reporting, and AI workloads.
• Design batch, streaming, and near-real-time data architectures.
• Implement and optimize Lakeflow pipelines, Spark workloads, and Delta Lake tables.
• Establish data quality, observability, schema evolution, and pipeline recovery standards.
• Architect data lifecycle management, retention, and performance optimization.

4. Enterprise Integration & Data Engineering Architecture

• Design integrations across ERP, SAP, operational systems, APIs, relational databases, and enterprise data platforms.
• Establish ingestion patterns using Lakeflow Connect, Auto Loader, CDC, Kafka, and other appropriate technologies.
• Design connectivity across Azure services, on-premises systems, and hybrid cloud environments.
• Evaluate migration strategies from legacy data warehouses and data platforms into Databricks.
• Define reusable integration patterns, orchestration standards, and error-handling mechanisms.
• Understand operational technology data patterns, including SCADA, historians, IoT, and industrial telemetry.

5. Databricks AI/BI & Genie Spaces

• Architect governed analytics solutions using Databricks SQL, AI/BI Dashboards, and Genie Spaces.
• Design semantic data structures and curated Gold-layer datasets for business consumption.
• Configure Genie Spaces with trusted datasets, instructions, business definitions, example queries, and appropriate access controls.
• Establish data quality and semantic consistency to improve natural-language analytics accuracy.
• Integrate Databricks AI/BI capabilities with existing enterprise reporting and analytics ecosystems.
• Evaluate AI/BI accuracy, adoption, performance, and governance requirements.

6. AI, ML & Agentic Architecture

• Design data and platform foundations supporting enterprise AI, machine learning, RAG, and agentic applications.
• Understand MLflow, model serving, vector search, and model governance.
• Architect secure access to governed enterprise data for AI agents and applications.
• Evaluate Databricks-native capabilities alongside Azure AI services.
• Understand Lakebase and its role in operational application architectures.
• Design appropriate patterns for AI workloads that require transactional state, governed analytics, and operational integrations.

7. DevOps, Platform Engineering & Operational Excellence

• Implement Infrastructure as Code using Terraform and automated deployment practices.
• Establish CI/CD using Databricks Declarative Automation Bundles, Azure DevOps, or GitHub Actions.
• Define platform monitoring, alerting, logging, and operational support models.
• Optimize Spark workloads, SQL warehouses, compute utilization, and Databricks consumption costs.
• Implement environment promotion, version control, release management, and rollback practices.
• Establish production readiness, operational documentation, and handover standards.

8. Consulting, Presales & Technical Leadership

• Lead client discovery workshops to assess business objectives, existing technology, architectural requirements, and constraints.
• Translate business requirements into practical, secure, and scalable technical architectures.
• Develop solution estimates, resource plans, delivery approaches, and implementation timelines.
• Present architectural recommendations and tradeoffs to CIOs, IT leaders, enterprise architects, and engineering teams.
• Lead technical design reviews and guide onshore and nearshore delivery engineers.
• Identify architectural risks early and communicate mitigation strategies.
• Take accountability for architecture quality, delivery feasibility, and client outcomes.

Required Qualifications

• 8+ years of relevant technology experience, including substantial hands-on data platform and cloud architecture responsibilities.
• 3+ years of hands-on Databricks implementation and architecture experience.
• Demonstrated experience designing and deploying at least one production Databricks platform from initial architecture through implementation.
• Strong Azure infrastructure knowledge, including networking, identity, security, storage, and deployment architecture.
• Hands-on experience with Unity Catalog and enterprise data governance.
• Strong understanding of Delta Lake, Spark, SQL, PySpark, and medallion architecture.
• Experience architecting data ingestion, enterprise integration, orchestration, and data processing solutions.
• Experience with Terraform, CI/CD, and automated platform deployments.
• Working knowledge of Databricks SQL, AI/BI Dashboards, and Genie Spaces.
• Ability to produce detailed architecture diagrams, explain technical tradeoffs, and guide implementation teams.
• Demonstrated client-facing consulting, solution design, and technical leadership capabilities.
• Ability to work directly with Houston-based clients and travel as project delivery requires.

Preferred Qualifications

• Databricks Certified Data Engineer Professional or Databricks Certified Data Analyst Associate.
• Databricks architecture, platform administration, or other relevant advanced credentials.
• Microsoft Azure Solutions Architect Expert certification.
• Experience deploying and optimizing Genie Spaces in production.
• Experience with Databricks AI/ML, MLflow, Vector Search, and agentic AI architectures.
• Knowledge of Lakebase and operational data application patterns.
• Experience with SAP, industrial data platforms, SCADA, historians, or IoT systems.
• Background in Oil & Gas, Energy, Utilities, Manufacturing, or Industrial operations.
• Experience leading nearshore engineering teams.

What Success Looks Like

Within the first 90 days, you should be able to:

• Independently lead Databricks architecture discovery and technical design sessions.
• Produce complete Azure Databricks reference architectures, including security, infrastructure, governance, integration, and data consumption.
• Define a credible deployment plan with resources, dependencies, effort estimates, and delivery milestones.
• Guide engineers through implementation of a secure, production-ready Databricks environment.
• Demonstrate how governed data can support AI/BI Dashboards, Genie Spaces, and emerging AI applications.

We are looking for an architect who has built these environments, not someone who has only consumed the services they provide.