Data Engineer

Salary: £50,000 - 50,000 per year

Requirements:

• Experience developing and managing PySpark workflows within AWS environments.
• Knowledge of AWS Lambda and event-driven solutions to support automated data processing.
• Experience provisioning and supporting cloud infrastructure using Amazon EC2 and automation tools.
• Understanding of Agile ways of working and experience contributing to CI/CD delivery practices.
• Ability to optimise data processing, storage, and queries to improve performance and data quality.
• Experience designing and automating ETL pipelines using PySpark and Spark SQL.
• Strong full stack development background with technologies including .NET 4.8, C#, JavaScript, C++, and Java.

Responsibilities:

• Help deliver modern data engineering solutions that support analytical reporting, business intelligence, and strategic decision-making.
• Design, build, maintain, and optimise scalable data products used by analysts and data scientists.
• Develop and orchestrate PySpark-based ETL pipelines using Amazon EMR, AWS Lambda, and EC2.
• Process large volumes of structured and unstructured data efficiently in a hybrid cloud environment.
• Collaborate with stakeholders to translate reporting requirements into robust technical solutions.
• Maintain high standards of data quality, performance, and security.
• Contribute to Agile delivery practices and support CI/CD automation.
• Maintain technical documentation, conduct code reviews, and share knowledge with other engineers and data professionals.

Technologies:

• AWS
• Lambda
• Business Intelligence
• C#
• CI/CD
• Cloud
• EC2
• ETL
• Support
• Java
• JavaScript
• PySpark
• SQL
• Security
• Spark
• ASP.NET

More:

We are DWP Digital, part of the UKs largest government department, and we use fresh ideas and leading-edge technology to build and maintain digital solutions used by nearly every person in the UK at key moments in their lives. Our Management Information team works in a hybrid model, spending 40% of time working from home and 60% collaborating face to face in a hub. We offer competitive pay of up to £51,528, a generous civil service pension with employer contributions of 28.97%, and a leave package starting at 26 days rising to 31 days, plus up to 3 extra flexi days each month. Our benefits also include volunteering and charitable giving time, discounts and savings, interest-free loans for bikes or season tickets, and professional development, coaching, mentoring, and career progression opportunities. We have an award-winning, inclusive culture and have been recognised for diversity, leadership, and being a great place to work in digital.

last updated 33 week of 2026