Databricks Lead Engineer - South East London

Salary: £130,000 - 149,500 per year

Requirements:

• At least two years of recent, hands-on Databricks experience
• Evidence of delivering Databricks solutions into production environments
• An active Databricks Data Engineering, Machine Learning or Generative AI certification
• Strong experience implementing Medallion Architecture across Bronze, Silver and Gold layers
• Hands-on Unity Catalog experience covering catalog design, access controls, lineage and secure data sharing
• Experience developing Declarative Pipelines with Expectations, including failure handling and observability
• Knowledge of Auto Loader, Lakeflow Connect and batch, streaming or incremental ingestion patterns
• Production-level PySpark and SQL development skills
• Strong knowledge of Delta Lake, including MERGE, schema evolution, OPTIMIZE, ZORDER and VACUUM
• Experience with Databricks Workflows, orchestration and CI/CD
• Understanding of monitoring, alerting, replay, backfill and operational support strategies
• Ability to mentor engineers and act as the technical escalation point
• Experience with RBAC, data masking, Photon, cluster optimisation or cost management would be beneficial
• Exposure to Mosaic AI, Vector Search, model serving, RAG or Lakebase would be advantageous

Responsibilities:

• Lead the delivery of a modern enterprise data platform for a leading financial services organisation
• Take technical ownership of Databricks lakehouse solutions, covering ingestion through to curated serving layers
• Implement Medallion Architecture, Unity Catalog, Declarative Pipelines and scalable Databricks-native ingestion patterns
• Mentor engineers and drive production-quality delivery
• Shape engineering standards, reusable patterns and delivery practices

Technologies:

• AI
• CI/CD
• Databricks
• Support
• Machine Learning
• Model Serving
• PySpark
• RAG
• RBAC
• SQL
• Unity
• GameDev

More:

We are offering up to £575 per day inside IR35 on an initial 6-month contract with potential extension. The role is hybrid with three days per week on-site in London and an ASAP start. We are providing technical leadership of a strategic Databricks implementation, with the opportunity to shape engineering standards, reusable patterns and delivery practices for a leading financial services organisation.

last updated 38 week of 2026