Senior Azure Data Engineer (Databricks)

Salary: £65,000 - 105,000 per year

Requirements:

• Proven hands-on experience with the Databricks platform, including orchestration and enterprise-scale Azure data engineering solutions.
• Strong technical skills in Python, PySpark, and SQL, with experience working with distributed data processing frameworks.
• Experience delivering end-to-end data pipelines across ingestion, transformation, and serving layers within modern Data Lakehouse architectures.
• Experience with schema design, GDPR-compliant data solutions, DevOps tooling, and CI/CD processes.
• A collaborative approach to engineering with experience using AI-enabled tools responsibly to improve productivity while maintaining appropriate human oversight, governance, and solution quality.
• Bonus: development experience using Scala or Java.
• Bonus: familiarity with Cloudera, Hadoop, Hive, Spark, and related distributed data ecosystems.
• Bonus: experience working with sensitive data and applying data privacy regulations, including GDPR.
• Bonus: curiosity to explore new technologies, including responsible AI and automation, to continuously improve engineering practices.
• Bonus: passion for collaboration, knowledge sharing, and continuous learning within high-performing engineering teams.

Responsibilities:

• Design and develop robust data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services.
• Build real-time, event-driven solutions using technologies such as Kafka and Azure Event Hubs, applying AI-enabled engineering approaches where appropriate to improve delivery efficiency.
• Develop and maintain CI/CD pipelines using Azure DevOps, Jenkins, GitHub Actions, and modern DevOps practices to support reliable deployments.
• Collaborate with clients and multidisciplinary teams to translate business requirements into scalable, production-ready data solutions.
• Champion clean code, data lifecycle optimisation, engineering best practices, and the responsible use of approved AI tools and automation to improve collaboration and delivery.

Technologies:

• AI
• Azure
• CI/CD
• Databricks
• DevOps
• GitHub
• Hadoop
• Hive
• Support
• Java
• Jenkins
• Kafka
• Python
• PySpark
• SQL
• Scala
• Spark
• Unity
• Cloud
• GameDev

More:

We are Capco, a world-class consulting environment in London offering a permanent Senior Azure Data Engineer (Databricks) role within our Technology & Engineering practice. We deliver high-impact technology solutions for Tier 1 financial institutions and help shape the future of digital transformation across Financial Services and Energy. We foster a collaborative, flat, entrepreneurial culture with continuous learning, training, industry certifications, and a people-first benefits package that includes bonus, pension, health and life insurance, mental health support, family-friendly leave, backup care, flexible holiday options, healthcare access, and a range of extra perks. We are committed to accessible recruitment and an inclusive #BeYourselfAtWork culture that values individuality and collaboration.

last updated 34 week of 2026