Databricks Data Engineer

Salary: £9,600 - 14,400 per year

Requirements:

• 8–10 years of hands-on data engineering experience.
• Strong commercial experience with Azure Databricks and Azure Data Factory.
• Advanced Spark development using Python or Scala, alongside strong SQL skills.
• Experience with Azure Data Lake Storage Gen2 and Azure SQL or PostgreSQL.
• Strong understanding of Azure analytics services and Azure identity, including SPN, SAMI and UAMI.
• Experience designing data lake, data warehouse, data mesh and service-oriented integration solutions.
• Previous ETL development experience using Informatica, SSIS, Talend or similar.
• Experience delivering solutions within an Agile SDLC environment.
• Strong Git, GitLab or GitHub experience, including branching, pull requests and CI/CD.
• Hands-on scripting experience with Linux or PowerShell.
• Exposure to Docker and automated testing practices.
• Experience with pipeline orchestration tools such as Apache Airflow, Autosys or Control-M, with Airflow preferred.
• Excellent written and verbal English communication skills.
• Kafka or other streaming technologies.
• Terraform, ARM templates, YAML or Puppet.
• GitLab CI/CD pipeline creation.
• Azure Functions and Logic Apps.
• Test-driven development.
• Previous experience delivering data solutions within an HR or employee lifecycle environment

Responsibilities:

• Design and develop scalable data pipelines and analytics solutions using Azure Databricks and Azure Data Factory.
• Build and improve data products supporting HR and employee lifecycle processes.
• Translate business and technical requirements into production-ready solutions.
• Develop data lake, data warehouse and data mesh solutions using Azure services.
• Work with large, complex and multi-format datasets.
• Apply automated testing, CI/CD and DevOps practices throughout the development lifecycle.
• Build observability into solutions to monitor production health and support incident resolution.
• Ensure solutions meet security, reliability, compliance and performance standards.
• Collaborate with engineering, product, HR and other cross-functional teams.
• Contribute to technical decisions with long-term scalability and sustainability in mind.

Technologies:

• Airflow
• ARM
• Azure
• CI/CD
• Data Warehouse
• Databricks
• DevOps
• Docker
• ETL
• Git
• GitHub
• GitLab
• Informatica
• Support
• Kafka
• Linux
• PostgreSQL
• PowerShell
• Puppet
• Python
• SQL
• Scala
• Security
• Spark
• SSIS
• Talend
• Terraform
• Cloud

More:

We are hiring an Azure Databricks Data Engineer in London on a hybrid working pattern, with 3–4 days per week onsite. This is an inside IR35 contract role within an HR technology environment, where we focus on building reliable, scalable and production-quality data solutions across the employee lifecycle. Our team works closely with engineering, product, HR and other cross-functional partners, with a strong emphasis on automation, testing, security, observability and long-term sustainability.

last updated 32 week of 2026