Lead Azure Databricks Platform Engineer / Architect

Salary: £? - ? per year

Requirements:

• We need deep hands-on Azure Databricks implementation and troubleshooting experience.
• We need experience with Databricks Serverless and compute/workload optimisation.
• We need strong knowledge of Azure identity, networking, security, secrets, monitoring and private connectivity.
• We need experience with Databricks SQL, Delta Lake and performance optimisation.
• We need strong Python, PySpark and SQL skills.
• We need experience with jobs/workflows, incremental processing, CDC and data quality.
• We need experience designing REST/API and external data integration patterns.
• We need FinOps experience covering cost attribution, tagging, budgets, monitoring and operational support.
• We need significant experience delivering enterprise Azure Databricks platforms in production.
• We need the ability to move between architecture, implementation, debugging and optimisation without relying entirely on specialist engineering teams.
• We need a strong understanding of platform security, data governance, operational support and controlled delivery in regulated or complex enterprises.
• We need experience working collaboratively with data engineers, data scientists, architects, security teams, platform teams and business stakeholders.
• We need clear communication skills and the ability to document standards, patterns, decisions and operational guidance.
• POSIT/RStudio migration or consolidation experience would be highly desirable.
• Experience migrating analytical or data science workloads, including converting and migrating R development/libraries to Databricks, would be highly desirable.
• AI/ML, LLM integration, model lifecycle, RAG/vector retrieval or model-serving experience would be highly desirable.
• Large-scale enterprise platform transformation and regulated-industry experience would be highly desirable.
• Strong cost optimisation and FinOps delivery experience across Azure and Databricks would be highly desirable.

Responsibilities:

• We will assess existing workloads and determine suitability for Serverless, classic, job or interactive compute based on duration, utilisation, SLA, concurrency, performance and cost.
• We will enable and configure Serverless for appropriate jobs, SQL workloads, notebooks, analytical processing and data pipelines.
• We will establish workload-placement guidance, including when Serverless is not economical for predictable, heavy or continuously running workloads.
• We will implement compute policies, autoscaling, quotas, budget controls and operational guardrails.
• We will measure cost and performance outcomes, identify idle or oversized compute, and recommend optimisation actions.
• We will define and embed a practical FinOps operating model covering ownership, accountability, projects, environments, teams, applications and cost centres.
• We will implement mandatory tagging and integrate validation into CI/CD so non-compliant resources are prevented from being provisioned.
• We will provide granular cost attribution by workspace, project, application, workload, job and team/user where technically appropriate.
• We will implement budget policies, thresholds, proactive alerts and usage reporting to prevent uncontrolled spend.
• We will use platform usage and billing data to identify idle compute, inefficient workloads, unnecessary storage/data movement and cost anomalies.
• We will enhance the Databricks Discovery Zone to support migration from POSIT/RStudio.
• We will enable application deployment, secure API integrations, external data ingestion, LLM integration, scheduling, BI connectivity, local IDE-based development and operational reporting.
• We will define reusable onboarding and migration patterns that reduce technology sprawl while improving security, supportability and delivery speed.
• We will design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake.
• We will implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls.
• We will design reusable integration patterns for REST APIs, SaaS platforms, databases, files, object storage, document repositories, enterprise applications and public or external data providers.
• We will implement secure authentication and credential handling for external and internal integrations.
• We will build end-to-end data flows from source through governed ingestion and curated layers to BI, ML or application consumption.

Technologies:

• AI
• API
• Architect
• Azure
• CI/CD
• Databricks
• Support
• LLM
• Python
• PySpark
• RAG
• REST
• SQL
• Security
• Serverless
• Cloud

More:

We are hiring a highly experienced, hands-on Azure Databricks Platform Engineer / Architect for a 6-month contract in London/Hybrid, inside IR35 at £500 per day. The role combines architecture with direct implementation to enhance and optimise our enterprise data platform, with a focus on Databricks Serverless, FinOps and platform controls, and the Databricks Discovery Zone to support workloads currently delivered through POSIT/RStudio. We work in a complex enterprise environment and need someone who can configure, develop, troubleshoot and optimise Databricks in practice, while collaborating closely with engineering, architecture, security and business stakeholders.

last updated 33 week of 2026