Software Engineer II - Databricks

Salary: £100,000 - 100,000 per year

Requirements:

• We have experience building data pipelines on Databricks and/or Apache Spark in production.
• We have strong coding skills in Python (PySpark) and SQL; Scala is a plus.
• We have hands-on experience with Delta Lake, including MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE, and VACUUM.
• We have experience with orchestration and scheduling tools such as Databricks Workflows, Airflow, and Azure Data Factory.
• We are familiar with cloud data platforms and storage such as AWS, Azure, GCP, S3, ADLS, and GCS.
• We have a solid understanding of data engineering fundamentals, including data modeling, ETL/ELT patterns, reliability, observability, and performance tuning.
• We have hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment and can critically evaluate and validate AI-generated outputs.
• We understand responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs and outputs, and adherence to resiliency and security expectations.
• Preferred: We have experience with streaming technologies such as Structured Streaming, Kafka, Event Hubs, or Kinesis.
• Preferred: We have experience implementing data quality frameworks such as Great Expectations or DQ and data testing in CI.
• Preferred: We have exposure to Unity Catalog or similar tools for governance and fine-grained permissions.

Responsibilities:

• We design and implement batch and streaming data pipelines using Databricks, Spark, Delta Lake, and orchestrators such as Workflows, Airflow, and ADF.
• We develop and optimize Spark jobs and SQL transformations for performance, reliability, and cost efficiency.
• We build and maintain curated data models, data quality checks, and automated testing.
• We implement CI/CD for notebooks and code using Git-based workflows and automate deployments across environments.
• We manage and tune Databricks clusters, jobs, and configurations, monitor production workloads, and resolve incidents.
• We integrate multiple data sources, including cloud storage, relational databases, APIs, and event streams, and implement robust ingestion patterns.
• We apply data governance and security best practices, including access controls, secrets management, lineage and metadata, and auditing.
• We create clear documentation for pipelines, data contracts, and operational runbooks.
• We leverage enterprise-authorized AI coding assist tools to improve code quality, delivery speed, and productivity while validating outputs through peer review, automated testing, and secure coding standards.
• We apply knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.

Technologies:

• AI
• Airflow
• AWS
• Azure
• CI/CD
• Cloud
• Databricks
• ETL
• GCP
• Git
• Kafka
• Python
• PySpark
• SQL
• Scala
• Security
• Spark
• Unity
• GameDev

More:

We are partnering directly with JPMorganChase for this Software Engineer II role within our Corporate Investment Bank Payments Technology team. We are a global leader in financial services, providing strategic advice and products to corporations, governments, wealthy individuals, and institutional investors. Our Commercial & Investment Bank operates across banking, markets, securities services, and payments in more than 100 countries. We value trusted long-term partnerships, diversity and inclusion, and we provide reasonable accommodations for applicants and employees with religious practices and beliefs, as well as mental health or physical disability needs.

last updated 35 week of 2026