Software Engineer II - Databricks

Salary: £62,000 - 102,000 per year

Requirements:

• Experience building data pipelines on Databricks and/or Apache Spark in production.
• Strong coding skills in Python (PySpark) and SQL; Scala is a plus.
• Hands-on experience with Delta Lake, including MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE, and VACUUM.
• Experience with orchestration and scheduling tools such as Databricks Workflows, Airflow, and Azure Data Factory.
• Familiarity with cloud data platforms such as AWS, Azure, or GCP, and storage services such as S3, ADLS, or GCS.
• Solid understanding of data engineering fundamentals, including data modeling, ETL/ELT patterns, reliability, observability, and performance tuning.
• Hands-on experience using enterprise-authorized AI-assisted software development tools, with the ability to critically evaluate and validate AI-generated outputs.
• Understanding of responsible AI use in engineering workflows, including data sensitivity, secure handling of inputs and outputs, and adherence to resiliency and security expectations.
• Preferred: Experience with streaming technologies such as Structured Streaming, Kafka, Event Hubs, or Kinesis.
• Preferred: Experience implementing data quality frameworks such as Great Expectations or DQ, and data testing in CI.
• Preferred: Exposure to Unity Catalog or similar governance and fine-grained permissions tools.

Responsibilities:

• Design and implement batch and streaming data pipelines using Databricks, Spark, Delta Lake, and orchestration tools.
• Develop and optimize Spark jobs and SQL transformations for performance, reliability, and cost efficiency.
• Build and maintain curated data models, data quality checks, and automated testing.
• Implement CI/CD for notebooks and code using Git-based workflows, and automate deployments across environments.
• Manage and tune Databricks clusters, jobs, and configurations; monitor production workloads and resolve incidents.
• Integrate multiple data sources, including cloud storage, relational databases, APIs, and event streams, and implement robust ingestion patterns.
• Apply data governance and security best practices, including access controls, secrets management, lineage, metadata, and auditing.
• Create clear documentation for pipelines, data contracts, and operational runbooks.
• Use enterprise-authorized AI coding assist tools to improve code quality, delivery speed, and productivity, while validating outputs through peer review, automated testing, and secure coding standards.
• Apply knowledge of Software Development Life Cycle tools, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.

Technologies:

• AI
• Airflow
• AWS
• Azure
• CI/CD
• Cloud
• Databricks
• ETL
• GCP
• Git
• Support
• Kafka
• Python
• PySpark
• SQL
• Scala
• Security
• Spark
• Unity
• GameDev

More:

We are JPMorgan Chase within our Corporate Investment Bank, Payments Technology team, and we are looking for a Databricks Engineer to help design, build, and operate scalable data pipelines and analytics workloads on the Databricks Lakehouse platform. You will partner with data analysts, data scientists, and application teams to deliver trusted datasets, performant ETL/ELT pipelines, and well-governed data products. We are part of J.P. Morgan, a global leader in financial services and a leading business within banking, markets, securities services, and payments. We value diversity and inclusion, offer equal opportunity, and support reasonable accommodations. This is a full-time role.

last updated 34 week of 2026