Senior Data Engineer

Salary: £60,000 - 100,000 per year

Requirements:

• Degree or equivalent, typically requiring 7+ years of relevant experience
• Bachelors degree level or above
• 6+ years experience in data engineering or platform engineering, including at least 2 years at senior or lead level
• 4+ years experience building and maintaining CI/CD pipelines, deployment workflows, and associated governance processes
• Advanced SQL and Python, with hands-on experience building data pipelines and transformations at scale
• Hands-on experience with a modern cloud data stack, including Snowflake, dbt, Airflow, and Databricks, on a cloud platform (Azure preferred)
• Demonstrated experience delivering data lineage and traceability within a regulated environment (e.g. SOX, financial services, healthcare, or similar)
• Experience contributing to platform migrations, re-architecture, or modernisation programmes
• Proven ability to present technical concepts to non-technical and senior stakeholders, including Finance, and influence decisions through practical recommendations
• Experience maintaining Infrastructure as Code for cloud-based data platforms
• Exposure to containerised or distributed systems (AKS advantageous), including deployment, scaling, and operational considerations
• Experience operating Airflow at scale, including DAG reliability, scheduling, and failure handling
• Strong version control, branching strategies, and release process experience
• Experience with test-driven development and automated testing within CI/CD workflows
• Ability to define and enforce standardised engineering patterns across a shared codebase
• Experience with operational reliability, including monitoring, structured logging, alerting, and systematic failure recovery
• PySpark experience desirable
• Experience supporting Airflow in a self-managed environment desirable
• Experience in a business building a data platform from the ground up desirable

Responsibilities:

• Take technical ownership of platform components, ensuring quality and consistency
• Establish, apply, and evolve engineering best practices, design patterns, and coding standards
• Define reusable patterns and frameworks across the data platform, including layered/medallion architecture approaches
• Identify opportunities to improve performance, cost efficiency, scalability, and reliability
• Contribute to the target architecture and long-term platform direction
• Design and build robust, scalable data pipelines using Snowflake, dbt, Airflow, ADF, and Python
• Apply strong practices in test-driven development, monitoring, logging, and observability
• Own more complex technical solutions and problem-solving
• Support production issues and drive root-cause improvements rather than quick fixes
• Ensure data lineage and traceability across the data lifecycle so any figure can be traced back to source
• Ensure sensitive data is appropriately protected within the data governance framework
• Work with IT teams to ensure solutions meet governance and compliance requirements
• Understand and manage the implications of production changes within a controlled release process
• Contribute to and maintain Infrastructure as Code for the platform in collaboration with the Infrastructure team
• Work within and help mature CI/CD pipelines, deployment workflows, and governance processes
• Continuously improve how the team builds, tests, and deploys data pipelines
• Bring forward ideas to improve the platform, tooling, and developer experience
• Evaluate emerging technologies and contribute to adoption decisions
• Support platform evolution, including re-architecture and migration activities
• Support other Data Engineers by sharing best practices and technical approaches
• Collaborate closely with Analytics & Data Insights, Software Engineering, and IT/Infrastructure
• Present technical work and recommendations to a range of audiences, including company-wide forums
• Build strong working relationships with business stakeholders, including Finance, and translate technical concepts into clear, practical terms

Technologies:

• Airflow
• Azure
• CI/CD
• Cloud
• Databricks
• Support
• Python
• PySpark
• SQL
• Security
• Snowflake
• dbt

More:

We are the technical owner of the ClarusONE data platform, a fast-growing environment where this hands-on engineering role will help shape reliability, scalability, security, and quality. We work within a data governance framework to protect sensitive data and ensure traceability end to end, partnering closely with Analytics, Data Science, Software Engineering, IT/Infrastructure, and Finance. This is a hybrid role based in the UK, with an expectation of two days per week onsite. Our location is London, and this is a full-time position.

last updated 36 week of 2026