Real-time data
I design scalable streaming pipelines for time-sensitive data, while processing less urgent workloads in batch to optimise performance and control infrastructure costs.
Data Platform Engineer · EU
Eight years of experience across banking, fintech, telecommunications, energy, and blockchain, designing scalable data platforms, real-time pipelines, cloud migrations, governed data systems, and analytics embedded directly into products.
6 areas · swipe to explore
I design scalable streaming pipelines for time-sensitive data, while processing less urgent workloads in batch to optimise performance and control infrastructure costs.
I design layered, scalable data platforms with infrastructure defined as code, giving teams a consistent foundation for deploying new pipelines, workloads, and services.
I build the data infrastructure behind customer-facing analytics, with centrally governed metrics and secure tenant-level access so each customer sees only the data relevant to them.
I engineer reliability into every stage of the data lifecycle through automated testing, observability, monitoring, and alerting, enabling issues to be identified before they affect downstream reporting or customers.
I work with business and domain owners to establish consistent metric definitions, then translate them into governed data models that stakeholders can understand, validate, and audit.
I modernise legacy data platforms incrementally, running existing and new pipelines in parallel until performance, data quality, and business logic have been fully validated.
Working through a similar data challenge? Book a 30-minute call →
Selected engagements across fintech, telecommunications, banking, SaaS, consulting, and data platforms. Client confidentiality is preserved; each case study sets out the outcome, context, my role, the challenge, and the technical approach.
Result. Critical fraud patterns were detected and surfaced in under 500ms, contributing to a 25% reduction in fraud-related losses. Platform infrastructure costs fell by 90% while throughput doubled.
Context. A fintech platform required near-real-time detection of suspicious transactions without materially increasing infrastructure expenditure.
My role. I owned the company’s data platform end to end and designed and implemented the transaction-monitoring pipeline.
Challenge. Detect high-risk transaction patterns within seconds while operating within a strict infrastructure budget.
Approach. I implemented a Kafka and Spark Structured Streaming architecture with tiered processing based on urgency. Critical patterns were evaluated and surfaced immediately, while lower-priority workloads were processed in batch. This preserved the required latency while keeping infrastructure costs sustainable.
Kafka · Spark Structured Streaming · Terraform · GitHub Actions
Result. Established an auditable source of truth for Finance and identified more than $1 million in annual excess payouts through reconciliation.
Context. A telecommunications business operated billing, usage, and settlement systems that applied different definitions of a completed call.
My role. I designed and built the reconciliation model and worked with Finance and Operations to establish the governing business definition.
Challenge. Reconcile tens of millions of daily call records across multiple systems when no individual source could be relied upon as the definitive record.
Approach. Rather than designating one system as authoritative, I staged each source independently and applied the agreed business definition through a conformed data model. Automated quality tests were embedded throughout the transformation process, with the reconciled dataset refreshed hourly.
dbt · Dimensional Modelling · Data Quality Testing
Result. Reduced platform setup time by 70% across more than five client environments.
Context. New data platforms were being established independently across multiple client engagements.
My role. As Lead Data Engineer, I designed and implemented the shared infrastructure and deployment foundation.
Challenge. Reduce the time and engineering effort required to provision each new client platform while maintaining consistency across environments.
Approach. I standardised infrastructure using Terraform and automated deployments through GitHub Actions, creating a reusable foundation from which each client platform could be provisioned and extended.
Terraform · GitHub Actions · CI/CD
Result. Enabled customer-specific reporting within the product using a single governed set of metric definitions.
Context. A SaaS product required analytics to be delivered directly within the application while maintaining strict separation between customer datasets.
My role. I contributed to the design and implementation of the analytics data layer supporting the product.
Challenge. Support high data volumes across multiple tenants while ensuring that each customer could access only its own information.
Approach. A semantic layer provided centrally governed metric definitions, while tenant-level query scoping ensured that every request returned data exclusively for the requesting customer.
Cube · Semantic Layer · Multi-tenant Access Control
Result. Migrated the data warehouse to Snowflake and Databricks and improved data retrieval performance by approximately 30% through query and partition optimisation.
Context. The bank relied on stored-procedure-based ETL with limited testing, unclear dependencies, and low operational visibility.
My role. As a Data Engineer on the programme, I led the migration of the data warehouse to Snowflake and Databricks.
Challenge. Modernise the platform without losing critical business logic embedded within the legacy ETL estate.
Approach. I established observability first using Grafana and Prometheus to understand pipeline behaviour and dependencies. Migration then proceeded incrementally, with legacy and cloud-native pipelines operating in parallel until outputs were reconciled and validated.
Grafana · Prometheus · Kafka · Spark · Snowflake · Databricks
Result. Enabled engineering teams to work through standardised SDKs and APIs rather than interacting directly with Spark and Delta Lake infrastructure.
Context. Engineering teams were working directly with Spark and Delta Lake when building large-scale data workloads.
My role. Senior Data Platform Engineer on the engagement.
Challenge. Provide engineers with a consistent, tested interface for reading and writing data without requiring each team to manage the underlying platform complexity.
Approach. I developed reusable SDKs and APIs that abstracted Spark and Delta Lake operations behind a simplified interface, standardising common data-access patterns across engineering teams.
Spark · Delta Lake · SDKs · APIs
I design data platforms around four core considerations: freshness, scale, access, and cost. The architecture below reflects the principles I typically apply, with implementation decisions adapted to the workload, operating environment, and business requirements.
Applications, transactional databases, connected devices, third-party platforms, and external APIs.
Streaming, change data capture, or scheduled ingestion selected according to latency, scale, and workload requirements.
Open table formats on object storage, enabling storage and compute to scale independently.
Version-controlled transformations with automated data-quality testing before information reaches downstream systems or users.
A governed semantic layer and high-performance analytical queries, with business metrics defined consistently and reused across applications.
Product features, embedded analytics, management reporting, operational decision-making, and AI-enabled applications.
Containerised workloads orchestrated to scale with demand.
Infrastructure defined and versioned as code for consistent, repeatable, and auditable deployments.
Metrics, logs, traces, and alerting across the platform to provide end-to-end operational visibility.
Centralised permissions, role-based access, and data masking applied consistently across the platform.
Governance is embedded by design, with data cataloguing, PII masking, and role-based access controls implemented across platforms serving more than five regulated industries.
Lead engineer across more than five client data platforms, with responsibility for architecture, technical delivery, and engineering quality. Line-managed up to two engineers and reviewed system design and code. Implemented data-governance controls across more than five regulated industries and developed natural-language querying over business data grounded in a governed semantic layer.
Built distributed Spark workflows processing more than 1TB of data daily for real-time recommendation systems, APIs serving AI safety scores to moderation tooling, and analytical workloads on BigQuery.
Modernised legacy stored-procedure ETL into cloud-native data pipelines and led the migration of warehouse workloads to Snowflake and Databricks.
Built pipelines for network and IoT telemetry and modernised legacy Hadoop batch workloads using Databricks and Spark.
Delivered payments analytics, stakeholder reporting, dashboards, and A/B experimentation to support product and commercial decision-making.
Designed and built the data infrastructure from the ground up, transforming building-energy data into carbon and climate-risk insights for the real-estate sector.
Built reusable SDKs and APIs that abstract Spark and Delta Lake complexity for engineering teams.
Built and maintained data pipelines powering AI-driven revenue intelligence.
Owned the data platform end to end, including real-time fraud detection with sub-500ms alerting, and provided line-management responsibility for engineering team members.
Python · SQL · Spark · Kafka · dbt · Airflow · Kubernetes · Terraform · BigQuery · Snowflake
Python · SQL · PySpark
Google Cloud Platform · Amazon Web Services · Microsoft Azure
Kafka · AWS MSK · Kinesis · Pub/Sub · Spark Structured Streaming · Debezium
Spark · Databricks · Dataproc · EMR · Apache Beam · dbt · DuckDB
Airflow · Dagster · AWS Step Functions · Cloud Scheduler
Airbyte · Fivetran · dltHub · AWS Glue · Dataflow
Apache Iceberg · Delta Lake · Snowflake · BigQuery · Redshift · ClickHouse · Cube
PostgreSQL · MySQL · MongoDB · DynamoDB · CockroachDB · Elasticsearch · Redis
Kubernetes · Helm · Terraform · Docker · Cloud Run · GitHub Actions · GitLab CI
Datadog · OpenTelemetry · Prometheus · Grafana · OpenMetadata
Certified Kubernetes Application Developer · Google Cloud Professional Data Engineer · View all certifications →