Data Engineering Lead – AWS & AI - Manchester

Salary: £70,000 - 95,000 per year

Requirements:

• You are an experienced Data Engineer or Data Engineering Lead who has built and operated complete data platforms, not just individual components.
• You combine strong technical knowledge with the confidence to make architectural decisions, lead change, and determine when existing technology should be evolved or replaced.
• Your experience includes batch processing, data warehousing, ELT, and orchestration.
• You have experience with modern streaming technologies such as Flink or comparable platforms.
• You have production AWS data engineering experience.
• You have hands-on experience with AWS services including S3, Lambda, Glue, Athena, Kinesis, RDS, IAM, and CloudWatch.
• You have experience with lakehouse architectures and open table formats such as Iceberg.
• You have experience with medallion-style data layering and dimensional modelling.
• You have strong SQL and Python development skills.
• You have infrastructure-as-code experience using Terraform.
• You have experience with CI/CD, automated testing, and modern engineering practices.
• You have experience with SQL Server, SSIS/SSDT, and Power BI.
• You have experience with semantic models, shared metrics, and data glossaries.
• You have experience with data governance, retention, lineage, classification, and auditability.
• You have experience managing, mentoring, or developing Data Engineers.
• You are comfortable using advanced AI coding assistants such as Claude Code or equivalent tools while maintaining appropriate quality and security controls.
• Desirable experience includes financial services or another regulated environment.
• Desirable experience includes PySpark for large-scale batch or streaming transformations.
• Desirable experience includes migrating SQL Server and SSIS estates to lakehouse or metadata-driven architectures.
• Desirable experience includes Customer Data Platforms or event-driven systems.
• Desirable experience includes data-quality tools such as dbt tests or Great Expectations.
• Desirable experience includes MCP servers or governed natural-language data interfaces.
• Desirable experience includes financial crime, fraud, or regulatory-reporting data.
• Desirable experience includes Power BI administration and semantic modelling.

Responsibilities:

• Own the reliability, security, and evolution of our data platform.
• Modernise our existing SQL Server, SSIS/SSDT, and Power BI environment.
• Develop scalable batch and real-time data-processing capabilities.
• Lead the design of a Customer Data Platform.
• Build and evolve a modern AWS lakehouse architecture.
• Create governed semantic models, consistent metrics, and clear data definitions.
• Enable secure, self-service data analysis through modern AI tools.
• Automate manual, repetitive, and fragile data processes.
• Support BI, Data Science, Machine Learning, and Applied AI teams through a unified platform.
• Own data governance, platform security, monitoring, and operating costs.
• Manage, develop, and help grow our Data Engineering team.
• Work closely with technical leaders and senior stakeholders across the organisation.

Technologies:

• AI
• AWS
• Lambda
• Business Intelligence
• CI/CD
• Claude Code
• CloudWatch
• Flink
• IAM
• Support
• Machine Learning
• MCP
• Power BI
• Python
• PySpark
• SQL
• Security
• SSIS
• Terraform
• dbt
• Cloud

More:

We are an established financial services organisation looking for a hands-on Data Engineering Lead to take ownership of our existing data estate and deliver a modern platform supporting Business Intelligence, Data Science, and Applied AI. This is an influential position combining technical leadership, platform ownership, and line management. You will modernise our data capabilities, introduce scalable batch and real-time processing, and eliminate manual or fragile processes through intelligent automation. We offer a hybrid working pattern based in Manchester, with the opportunity to take genuine ownership of a business-critical data platform and shape how data and AI are used across our organisation. You will have the freedom to introduce modern technologies, develop a talented engineering team, and build a platform capable of supporting future analytics, AI products, and strategic decision-making.

last updated 39 week of 2026