Software Engineer – Data, Lakehouse & AI Platform

Salary: £60,000 - 100,000 per year

Requirements:

• Bachelors or masters degree in a relevant discipline, or equivalent practical experience, with evidence of strong quantitative skills or data engineering expertise.
• Strong hands-on programming experience in Python or Java.
• Good working knowledge of SQL, including troubleshooting, optimisation and data analysis.
• Ability to learn new tools, internal platforms and delivery workflows quickly.
• Familiarity with software engineering fundamentals, including version control, testing, release discipline and CI/CD practices.
• Stronger ownership of technical design across multiple datasets or pipeline domains.
• Experience guiding implementation standards, code quality and engineering practices within a team.
• Ability to lead delivery for a workstream, manage dependencies and support less experienced engineers.
• Understanding of temporal data modelling, including the handling of historical state and change over time.
• Knowledge of schema design, schema evolution and data compatibility considerations.
• Understanding of partitioning, clustering and other techniques used to improve data performance at scale.
• Ability to make sensible design choices across normalised and denormalised models, and between natural and surrogate keys.
• Practical approach to data quality, reconciliation and root-cause analysis.
• Experience building or supporting production data pipelines in a collaborative engineering environment.
• Experience working with distributed data processing frameworks such as Apache Spark.
• Working knowledge of common data formats such as JSON, Avro and Parquet.

Responsibilities:

• Build, enhance and support batch and streaming data pipelines on our Lakehouse and AI data platform.
• Refactor or modernise existing data flows where needed to improve reliability, performance and maintainability.
• Ensure data pipelines are production-ready, well tested and operationally supportable.
• Develop raw, refined and curated datasets that support analytics, reporting and AI use cases.
• Apply sound data modelling principles to represent business entities, relationships and historical change accurately.
• Work with consumers to shape data products that are usable, well documented and aligned to business needs.
• Implement controls to validate completeness, accuracy and consistency of data across pipelines and datasets.
• Use reconciliation approaches to build confidence in production outputs and investigate breaks where they arise.
• Contribute to clear standards for testing, monitoring and issue resolution.
• Work closely with engineers, platform teams and data consumers to deliver agreed outcomes to time and quality expectations.
• Communicate clearly on progress, risks, dependencies and design choices.
• Take a broader role in technical leadership, task breakdown and support for junior engineers.

Technologies:

• AI
• CI/CD
• Databricks
• Support
• JSON
• Java
• Kafka
• LESS
• Python
• SQL
• Snowflake
• Spark
• Hadoop
• Kubernetes

More:

We are building the data foundations that support our AI and analytics capabilities within a modern Lakehouse and AI data platform. This delivery-focused engineering role is centered on scalable data systems, reliable production pipelines, and high-quality curated datasets that support analytics, operational decision-making and emerging AI use cases. We work with a modern and evolving data stack, including technologies such as ANSI SQL, Apache Spark, Kafka, Snowflake, Apache Iceberg, Databricks and related deployment approaches. We value pragmatic engineers who can work closely with platform teams and stakeholders, and we offer a collaborative environment with opportunities for professional growth, training, development, wellness and personal finance offerings, and firmwide networks. We are headquartered in New York with offices around the world, and we are committed to diversity, inclusion and reasonable accommodations during the recruiting process.

last updated 36 week of 2026