Senior Data Engineer

Salary: £? - ? per year

Requirements:

• Strong commercial experience as a Data Engineer or Senior Data Engineer
• Expert-level Python development skills
• Extensive hands-on experience with Databricks in enterprise environments
• Strong Apache Spark and PySpark expertise
• Advanced SQL development and optimisation skills
• Experience designing and building large-scale ETL and data integration solutions
• Strong understanding of distributed computing concepts and data processing frameworks
• Experience with modern cloud platforms, including Azure, AWS or GCP
• Knowledge of software engineering principles, testing and code quality practices
• Experience working in Agile development environments
• Excellent problem-solving and troubleshooting capabilities
• Desirable: experience in investment banking, financial services or other highly regulated environments
• Desirable: exposure to risk, trading, treasury or regulatory data domains
• Desirable: experience with Kafka, streaming technologies or real-time data processing
• Desirable: Databricks certifications
• Desirable: experience with Airflow, Databricks Workflows or similar orchestration tools
• Desirable: CI/CD pipeline development and DevOps practices
• Desirable: experience with Docker and Kubernetes
• Desirable: experience supporting data science or machine learning workloads

Responsibilities:

• Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark
• Build and optimise ETL/ELT solutions that process large-scale datasets
• Develop high-quality, reusable code and engineering frameworks
• Deliver robust, performant and maintainable data solutions
• Troubleshoot and resolve complex production and performance issues
• Enhance and modernise enterprise data platforms
• Develop and support Databricks-based data solutions and Lakehouse architectures
• Use Delta Lake and Spark to deliver scalable data processing capabilities
• Optimise distributed data processing workloads for performance and reliability
• Implement data quality controls and monitoring solutions
• Contribute to platform stability, scalability and operational excellence
• Work with business and technology stakeholders to understand requirements and deliver effective solutions
• Participate in Agile delivery activities, including sprint planning, refinement and estimation
• Support strategic data initiatives across multiple business areas
• Collaborate with architects, engineers and product teams to deliver high-quality outcomes
• Identify and resolve technical risks and implementation challenges
• Promote engineering best practices, including testing, code reviews and version control
• Support CI/CD adoption and deployment automation
• Contribute to coding standards and engineering governance
• Improve development and support processes
• Mentor junior engineers and share knowledge within the team

Technologies:

• Airflow
• AWS
• Azure
• CI/CD
• Cloud
• Databricks
• DevOps
• Docker
• ETL
• GCP
• Support
• Kafka
• Kubernetes
• Machine Learning
• Python
• PySpark
• SQL
• Spark

More:

We are recruiting for a global financial institutions growing data engineering function, which delivers critical data platforms and analytics capabilities across the business. This hands-on contract role offers the opportunity to work on complex data challenges in a modern cloud environment alongside engineers, architects and business stakeholders. The contract is for six months, with a very likely extension, at £850 per day inside IR35. The role is hybrid, based near Liverpool Street in London, with three days per week in the office. GCS is acting as an Employment Business in relation to this vacancy.

last updated 40 week of 2026