Dataiku Solution Architect

Salary: £53,000 - 93,000 per year

Requirements:

• Bachelors degree in Computer Science, Information Systems, Engineering, Data Science, or a related field.
• Significant experience in solution architecture, data architecture, application architecture, or enterprise integration architecture.
• Hands-on experience designing enterprise solutions using Dataiku DSS.
• Experience architecting or developing Dataiku applications and WebApps.
• Strong understanding of Dataiku Flows, recipes, datasets, managed folders, scenarios, plugins, APIs, automation, and deployment processes.
• Experience designing metadata-driven and configuration-driven data-processing platforms.
• Strong knowledge of Python and SQL-based data solutions.
• Experience integrating Dataiku with databases, APIs, file systems, enterprise applications, workflow platforms, and reporting tools.
• Experience with PostgreSQL, Amazon RDS, or comparable relational database platforms.
• Experience defining REST API, database, batch-file, XML, and event-based integration patterns.
• Understanding of cloud and on-premises integration architectures.
• Experience designing solutions with data lineage, traceability, audit logging, version control, role-based access, and evidence retention.
• Experience translating business and regulatory requirements into solution architecture and technical designs.
• Experience working in Agile delivery environments and leading architecture across multidisciplinary teams.
• Strong written and verbal communication skills, including the ability to present complex technical designs to business and executive stakeholders.
• Experience in energy trading, refined products, commodities, financial services, market risk, valuation, or regulatory reporting is preferred.
• Knowledge of Independent Price Verification processes, including consensus pricing, valuation marks, curves, variances, tolerances, exposures, and P&L impact, is preferred.
• Experience delivering applications or data platforms in a SOX-controlled environment is preferred.
• Experience replacing legacy Excel, VBA, Access, and email-based business processes is preferred.
• Experience with Power Automate and Power BI is preferred.
• Experience integrating with Murex or comparable trading and risk platforms is preferred.
• Familiarity with SOLARC, Position Manager, ONTAP, market-data providers, broker feeds, and exchange-pricing sources is preferred.
• Experience designing OCR or document-intelligence integrations for PDFs and image-based files is preferred.
• Familiarity with IAM, secrets management, single sign-on, Saviynt, Accsys, and enterprise cybersecurity controls is preferred.
• Experience supporting global platforms with standardized processes and controlled regional variations is preferred.
• Dataiku platform, architecture, or advanced developer certification is preferred.

Responsibilities:

• Define and maintain the end-to-end solution architecture for the IPV Control Hub across data acquisition, processing, workflow, storage, reporting, security, and governance layers.
• Translate business, risk, audit, security, and regulatory requirements into scalable technical architecture and solution designs.
• Establish architectural standards, integration principles, reusable patterns, and technical guardrails for the platform.
• Create solution architecture diagrams, data-flow diagrams, integration designs, interface specifications, security models, and deployment views.
• Provide technical leadership throughout design, development, testing, deployment, transition, and production stabilization.
• Evaluate design alternatives and recommend solutions based on scalability, auditability, resiliency, security, maintainability, and operational support requirements.
• Ensure clear separation of responsibilities across Dataiku, Power Automate, PostgreSQL or Amazon RDS, Power BI, and connected enterprise systems.
• Facilitate architecture reviews, technical design workshops, and solution walkthroughs with business and technology stakeholders.
• Identify architectural dependencies, design risks, integration constraints, and nonfunctional requirements early in the delivery lifecycle.
• Define the Dataiku architecture supporting ingestion, data preparation, transformation, business-rule processing, consensus pricing, variance calculation, threshold evaluation, historical replay, and exception management.
• Establish standards for Dataiku projects, Flows, datasets, managed folders, recipes, scenarios, plugins, APIs, applications, and reusable components.
• Design scalable Dataiku processing frameworks that support more than 70 internal and external data sources, including structured and unstructured data.
• Define metadata-driven frameworks for managing brokers, source priorities, mappings, thresholds, currencies, units of measure, regional variations, approval paths, and business rules.
• Design reusable Dataiku project templates and onboarding patterns for new brokers, market-data providers, commodities, regions, and business processes.
• Establish Dataiku development, testing, production, version-control, promotion, deployment, and rollback standards.
• Define monitoring, alerting, failure handling, restart, recovery, and operational support patterns for Dataiku processes.
• Ensure Dataiku processing is deterministic, traceable, version-controlled, and capable of reproducing historical IPV runs.
• Guide the migration of legacy Microsoft Access queries, Excel formulas, macros, and VBA logic into controlled Dataiku processing frameworks.
• Partner with Dataiku engineers and developers to review application designs, integration components, Python code, SQL logic, reusable recipes, and deployment configurations.
• Lead the architecture and design of Dataiku applications and WebApps used as the primary business interface for IPV operations.
• Design user experiences that support Last Deal Done capture, mandatory commentary, exception review, threshold-breach management, approvals, rejections, override requests, and rationale capture.
• Define application components that allow authorized business users to manage source mappings, broker configurations, thresholds, prioritization rules, validation rules, and regional settings.
• Design role-based application views for Commodity Risk, Trading, Finance, ARC, administrators, and operational support users.
• Ensure Dataiku applications provide clear processing status, validation results, exception queues, approval history, and audit evidence.
• Establish architecture patterns for communication between Dataiku WebApps, Dataiku processing pipelines, PostgreSQL or Amazon RDS, Power Automate, and Power BI.
• Ensure application activity is recorded in the governed evidence repository, including changes, approvals, rejections, overrides, comments, and resubmissions.
• Define application performance, availability, usability, security, logging, and support requirements.
• Promote configuration-driven application behavior to reduce hard-coded logic and minimize code changes when business rules are updated.
• Define secure, resilient, and supportable integrations between Dataiku and internal or external enterprise systems.
• Architect integrations with PostgreSQL, Amazon RDS, Power Automate, Power BI, Murex, SOLARC, Position Manager, ONTAP, shared mailboxes, exposure systems, broker feeds, exchange data, market reports, web services, and APIs.
• Establish interface standards for REST APIs, database connections, file transfers, event-driven processes, batch integrations, and workflow callbacks.
• Define common data-exchange standards using formats such as JSON, XML, XSD, CSV, and Parquet.
• Design governed ingestion patterns for broker marks, exchange prices, market reports, Last Deal Done values, position data, exposure data, and internal valuation curves.
• Define integration patterns for both structured and unstructured files, including PDFs, images, spreadsheets, text files, API feeds, and email attachments.
• Architect interfaces supporting OCR and text extraction for broker files received in PDF or image-based formats.
• Ensure integrations include validation, duplicate detection, reconciliation, error handling, retry logic, operational monitoring, and evidence capture.
• Define data contracts and interface specifications for source and downstream systems.
• Ensure integration designs reduce point-to-point dependencies and support future source onboarding through common, reusable patterns.
• Define the division of responsibilities between Power Automate, Dataiku, and the governed data repository.
• Architect Power Automate workflows that monitor shared mailboxes, retrieve attachments, route files through approved gateway connections, and deliver data to the ONTAP landing zone.
• Design workflow integrations for notifications, approvals, exception routing, escalations, resubmissions, and processing-status updates.
• Establish secure communication and status synchronization between Power Automate workflows and Dataiku processing pipelines.
• Ensure workflow decisions and approval outcomes are written to the governed evidence repository.
• Design workflows that support segregation of duties, controlled access, traceable decision-making, and auditable approval history.
• Define handling patterns for incomplete submissions, unsupported formats, failed validations, duplicate files, late-arriving data, and processing exceptions.
• Define the solution architecture for options data processing through Dataiku.
• Design the integration that generates XSD/XML outputs for delivery to the Murex ONTAP location.
• Define the controlled exchange of data between Dataiku, Murex, ONTAP, and PostgreSQL.
• Architect the return of Murex P&L calculation outputs to the governed data foundation.
• Ensure appropriate validation, reconciliation, lineage, and audit evidence exist across inbound and outbound Murex processes.
• Promote the use of shared data models, workflow components, audit controls, and reporting layers across Forwards and Options where appropriate.
• Coordinate with Murex, Trading, Commodity Risk, platform, and data teams to resolve interface dependencies and clarify system responsibilities.
• Define a canonical IPV data model that supports RPTA, RPTAP, and RPTE while accommodating governed regional variations.
• Architect the governed data foundation supporting raw source data, normalized data, consensus prices, variance calculations, exposure impacts, workflow decisions, approvals, evidence, and reporting outputs.
• Define data-layering, naming, ownership, quality, retention, archival, and purge standards.
• Ensure the architecture supports curve-level and deal-level variance calculations.
• Establish designs for currency conversion, unit-of-measure normalization, source prioritization, contributor counts, tolerance evaluation, and threshold-breach identification.
• Define data-quality controls, reconciliation points, exception rules, and completeness checks across the IPV lifecycle.
• Ensure data structures support historical analysis, operational reporting, audit inquiries, and replay of previous processing cycles.
• Lead the design of a centralized Metadata Control Plane governing source onboarding, validation, transformation, approval routing, audit evidence generation, reporting, and historical replay.
• Define metadata structures for sources, brokers, commodities, regions, mappings, thresholds, currencies, units of measure, business rules, approval paths, and retention policies.
• Establish governance processes for creating, reviewing, approving, versioning, activating, and retiring metadata configurations.
• Ensure metadata changes capture the requestor, approver, effective date, previous value, updated value, and business justification.
• Design effective-dating and versioning approaches so historical runs can be reproduced using the configurations that were active at the time.
• Enable business self-service capabilities while maintaining appropriate security, change control, segregation of duties, and approval requirements.
• Embed SOX control requirements into the architecture and design of all platform components.
• Define end-to-end lineage across source acquisition, transformation, consensus construction, variance analysis, approvals, reporting, and downstream outputs.
• Ensure every ingestion event, transformation, business-rule execution, threshold breach, override, approval, rejection, and output is traceable and auditable.
• Architect immutable audit logging and governed evidence storage.
• Ensure the platform supports three-year business-data retention and purge requirements, as specified in the proposal.
• Ensure audit and access logs support the proposed five-year retention requirement.
• Design the platform to reproduce a historical IPV run within the proposals two-hour requirement.
• Define segregation-of-duties controls across development, configuration, administration, business processing, approval, and production support activities.
• Ensure architecture alignment with applicable Information Security and Cyber Resilience policies, MITS requirements, IAM standards, and secrets-management controls.
• Support internal audit, external audit, compliance, and control-assurance reviews by providing architecture documentation and technical evidence.
• Define role-based access-control models for Dataiku applications, platform projects, datasets, workflows, reports, databases, and administrative capabilities.
• Design integration with enterprise identity and access-management services, including Saviynt, Accsys, and applicable single sign-on mechanisms.
• Ensure service identities, credentials, tokens, database connections, and API secrets are stored and managed through approved secrets-management solutions.
• Define security boundaries for user access, application access, service-to-service communication, and privileged administration.
• Ensure sensitive trading, valuation, position, and exposure data is protected throughout ingestion, processing, storage, reporting, and archival.
• Partner with cybersecurity and architecture-review teams to complete security assessments and resolve design findings.
• Define the reporting architecture supporting Power BI dashboards and analytical consumption.
• Ensure governed datasets are available for internal-versus-external mark comparison, consensus pricing, contributor analysis, percentage differences, curve-level variance, deal-level variance, threshold breaches, approval decisions, P&L impact, and senior-management reporting.
• Establish semantic-model and reporting-data requirements in collaboration with Power BI developers and business stakeholders.
• Ensure dashboards use controlled, reconciled, and traceable data from the governed platform.
• Define access, refresh, performance, lineage, and reconciliation standards for reporting solutions.
• Support curve observability and the monitoring of data quality, source availability, processing status, and workflow completion.
• Define nonfunctional requirements for performance, scalability, security, availability, resiliency, recoverability, observability, maintainability, and supportability.
• Design monitoring and alerting across Dataiku, Power Automate, PostgreSQL or Amazon RDS, integrations, WebApps, and reporting components.
• Establish recovery patterns for failed source deliveries, workflow errors, data-quality issues, integration failures, and interrupted processing.
• Define capacity and performance considerations for regional processing, historical replay, concurrent users, data growth, and peak IPV-cycle workloads.
• Ensure the architecture supports controlled deployment and minimal disruption to business operations.
• Define operational dashboards and service-health indicators for production-support teams.
• Serve as the primary architecture liaison across Commodity Risk, Trading, Finance, Technology, Security, Compliance, Audit, and delivery teams.
• Convert business processes and control requirements into detailed technical designs and implementation roadmaps.
• Work with product owners and business analysts to refine architecture epics, features, technical stories, and acceptance criteria.
• Partner with Dataiku engineers, integration developers, database engineers, Power BI developers, testers, DevOps teams, and platform administrators.
• Participate in Agile ceremonies, sprint planning, backlog refinement, solution demonstrations, and release-readiness reviews.
• Review technical deliverables to ensure alignment with approved architecture.
• Support testing strategy, including system integration testing, regression testing, performance testing, security testing, user acceptance testing, and control validation.
• Provide architecture support during deployment, production transition, stabilization, and operational handover.
• Develop technical documentation and knowledge-transfer materials for engineering and production-support teams.

Technologies:

• API
• Architect
• Cloud
• DevOps
• Excel
• Flow
• IAM
• Support
• JSON
• PostgreSQL
• Power BI
• Python
• REST
• SQL
• Security
• VBA
• Web
• XML
• CI/CD
• Git

More:

We are seeking an experienced Solution Architect with Dataiku application and integration expertise to lead the architecture and technical design of a modern Independent Price Verification (IPV) platform supporting our global Refined Product Trading organization. We are building a governed, metadata-driven IPV Control Hub that replaces fragmented Excel, Microsoft Access, VBA, and email-based processes with an automated, controlled, and SOX-compliant enterprise solution. Our platform uses Dataiku, Dataiku WebApps, Power Automate, PostgreSQL, Amazon RDS, and Power BI, and we work closely with Commodity Risk, Trading, Finance, Data Engineering, Cybersecurity, Platform, Architecture, Compliance, and business stakeholders to deliver a scalable, secure, auditable, supportable solution aligned with enterprise architecture standards.

last updated 37 week of 2026