AI Quality Engineer

Salary: £70,000 - 95,000 per year

Requirements:

• Strong test automation engineering experience.
• Experience with Python or a comparable programming language.
• Experience with a modern automation framework such as Playwright, pytest or similar.
• Strong understanding of CI/CD.
• An analytical and questioning approach to testing and quality.
• The ability to investigate test failures and inconsistent metrics and identify underlying causes.
• Strong communication skills, with the ability to explain complex technical issues to different audiences.
• A collaborative approach to working with engineers and product teams.
• Evidence that you can learn and become competent in technically challenging areas quickly.
• Genuine interest in the challenges involved in testing non-deterministic AI output.
• At a senior level, experience with LLM evaluation, rubric design, LLM-as-judge approaches, AI regression testing, evaluation harnesses, and improving quality practices across teams is beneficial.

Responsibilities:

• Enhance and run layered evaluation frameworks, including automated checks, LLM-as-judge scoring, defined rubrics and targeted human review.
• Maintain versioned golden datasets to ensure evaluations remain reproducible and auditable.
• Monitor key quality signals across investigative outputs, including groundedness, hallucination rate, entity resolution and source quality.
• Run and improve a tiered CI testing model using Python, Playwright and pytest.
• Maintain test coverage across six authenticated personas, three browsers and multi-tenant environments.
• Investigate flaky tests, inconsistent results and changes in evaluation metrics, identifying and addressing root causes.
• Work collaboratively with engineering and product teams to improve quality and confidence in releases.
• Apply your own judgement when interpreting results rather than relying solely on whether a test has passed or failed.

Technologies:

• AI
• CI/CD
• Support
• LLM
• Playwright
• Python
• pytest
• Claude Code

More:

We are offering a hybrid role based in Central London, with two days per week in the office, and a flexible salary depending on experience. You will join a business at the forefront of AI-powered investigative technology as our platform scales, helping us maintain confidence in the quality of AI-generated output across complex product releases and regulatory challenges. We offer employee shares and equity, private health insurance and life insurance, unlimited holidays, and an annual professional development fund to support learning and development.

last updated 37 week of 2026