Lead Software Engineer - Python, Databricks, AWS

Salary: £62,000 - 102,000 per year

Requirements:

• Proven experience in data management, ETL/ELT pipeline development, and large-scale data processing.
• Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning.
• Hands-on experience with data lake platforms such as Databricks, Apache Spark, or similar.
• Experience with AWS cloud services such as S3, ECS, SNS/SQS, Lambda, and related tools.
• Strong understanding of data quality, security, and lineage best practices.
• Experience with cloud-based data warehouse migration and modernization.
• Proficiency in CI/CD, continuous delivery methods, the full Software Development Life Cycle, and Agile methodologies.
• In-depth knowledge of the financial services industry and its IT systems.
• Excellent problem-solving, troubleshooting, and analytical skills, with the ability to investigate data issues, identify root causes, and implement solutions.
• Strong communication and documentation abilities, with the ability to collaborate effectively with business and technical stakeholders.
• Knowledge of regulatory reporting and financial data aggregation techniques is preferred.
• Experience with data orchestration tools such as Airflow and Step Functions is preferred.
• Databricks or AWS certifications are preferred.
• Demonstrated proficiency in cloud-native development, including cloud, artificial intelligence, and machine learning, is preferred.
• Experience with Parquet, JSON, CSV, Avro, and Delta Lake file formats is preferred.

Responsibilities:

• Execute creative, data-driven software solutions, including design, development, and technical troubleshooting, with the ability to think beyond routine approaches to solve technical problems.
• Design, develop, and maintain scalable data pipelines and processing workflows using Python, PySpark, SQL, and Databricks on AWS, processing and transforming large-scale financial datasets for analytics and reporting.
• Develop fact and dimension data models for reporting and analytics.
• Write secure, high-quality production code, and review and debug code written by others.
• Ensure data quality, consistency, security, and lineage throughout all stages of data processing and transformation, implementing monitoring and alerting mechanisms to maintain pipeline reliability.
• Support data migration and modernization initiatives, transitioning legacy systems to cloud-based data warehouses.
• Lead communities of practice across Software Engineering to promote awareness and adoption of new technologies.
• Foster a team culture of diversity, opportunity, inclusion, and respect.
• Mentor and guide Associate-level engineers, supporting their technical development and ensuring consistent delivery standards across the team.
• Collaborate with business stakeholders to develop data management strategies, transforming data into insights that drive strategic decisions.
• Document data flows, logic, and transformation rules to maintain transparency and facilitate knowledge sharing.
• Lead and participate in the full Software Development Life Cycle, including requirements gathering, design, development, testing, deployment, and maintenance.
• Identify and automate remediation of recurring issues to improve operational stability, acting as both Production Support and SRE function for data pipelines and platform services.
• Utilise AI tools to accelerate development and testing of data pipelines.

Technologies:

• AI
• Airflow
• AWS
• Lambda
• CI/CD
• Cloud
• Data Warehouse
• Databricks
• ETL
• Support
• JSON
• Machine Learning
• Marketing
• Python
• PySpark
• SQL
• Security
• Spark
• Claude Code
• Copilot
• GitHub
• Jenkins
• Sonar

More:

We are JPMorgan Chase, a global leader in financial services, providing strategic advice and products to prominent corporations, governments, wealthy individuals, and institutional investors. Our External Regulatory Financial Control (ERFC)/Strategic Data team uses cutting-edge technology to drive data-driven decision-making and enhance business performance. Our corporate functions span finance, risk, human resources, and marketing, and we value diversity, inclusion, and equal opportunity. We are committed to building trusted, long-term partnerships and supporting our clients and employees. This is a full-time role.

last updated 36 week of 2026