Principal AI Data Engineer

Salary: £? - ? per year

Requirements:

• Strong expertise with Databricks, including Delta Lake, Unity Catalog, DLT and MLflow.
• Deep knowledge of large language models including GPT, Llama, Claude and Mistral.
• Experience building and deploying GenAI, RAG and Agentic AI solutions in enterprise environments.
• Strong programming skills in Python, SQL and/or Scala.
• Experience with Azure cloud technologies including Azure OpenAI and Azure AI Foundry.
• Knowledge of Spark Structured Streaming, Autoloader and modern data engineering practices.
• Experience with ETL/ELT, data modelling and scalable data architectures.
• Familiarity with Azure Data Factory, Qlik Replicate and data ingestion pipelines.

Responsibilities:

• Develop AI, GenAI and Agentic AI prototypes using Azure AI Foundry, Copilot Studio, Databricks Mosaic AI and MLflow.
• Build, optimise and evaluate retrieval-augmented generation (RAG) solutions using prompt engineering and embedding models.
• Design and deploy AI agents using frameworks including LangChain, LangGraph and AutoGen.
• Deliver production-ready GenAI applications for commercial teams within Trading & Supply.
• Build scalable AI workflows on Databricks using Mosaic AI, MLflow, Genie and AgentBricks.
• Apply DevOps best practices through CI/CD pipelines, GitHub workflows and automated testing.
• Design evaluation frameworks to benchmark, test and improve GenAI system performance and reliability.

Technologies:

• Agentic AI
• AI
• AI Agents
• Azure
• CI/CD
• Cloud
• Copilot
• Databricks
• DevOps
• ETL
• Embedding Models
• GitHub
• Marketing
• MLflow
• Python
• Qlik
• RAG
• SQL
• Scala
• Spark
• Unity
• GameDev

More:

We are Certain Advantage, hiring for a Principal AI Data Engineer based in London. This is an initial contract role until the end of the year, and it is hybrid with 3 days per week onsite in London. We work across the UK to find the best people in Finance, Marketing, IT and Engineering, and we go the extra mile to help people make an impact and grow their careers.

last updated 32 week of 2026