— Practice / 03 / Substrate

Data Engineering.

Models are downstream of data. We build the pipelines, lakehouses, and contracts that let your AI be trusted — in regulated industries, at the scale of the real business, with lineage you can defend.

§ 01
Why this practice exists

Every disappointing AI demo has the same root cause: the data substrate wasn't ready.

The model is the easy part. The hard part is contracts, lineage, freshness, governance, and the unglamorous discipline of making sure the right rows reach the right embedding at the right moment.

We treat data engineering as the load-bearing wall of an AI-native company — not as a cost centre to be outsourced.

§ 02
Capabilities

What we build.

From foundational lakehouses to production-grade retrieval — built for the workloads your models actually face.

— 01

Lakehouse & Warehouse Architecture

Databricks, Snowflake, BigQuery, Iceberg. Bronze→silver→gold isn't religion; we choose the shape that fits your access patterns and the questions your business actually asks.

IcebergDeltaSnowflake
— 02

Real-Time & Streaming Substrate

Kafka, Flink, Pulsar, change data capture. Decisions in milliseconds, not in tomorrow's batch window. Built with backpressure and replay assumed.

KafkaFlinkCDC
— 03

Vector & Retrieval Stack

Embeddings, hybrid search, evals, drift detection. The retrieval layer is the unsung hero of every serious AI system — and the reason most fail in production.

Vector DBHybrid searchEvals
— 04

Governance, Lineage & Privacy

Catalogues, contracts, PII isolation, residency. Defensible AI requires defensible data — and a lineage graph you can show the auditor.

LineagePIIResidency
— 05

Data Contracts & Quality

Schema-as-code, producer responsibility, contract tests in CI, freshness SLAs. The discipline that makes downstream teams stop firefighting.

ContractsQualitySLA
— 06

Feature Stores & Model Inputs

Feature pipelines, point-in-time correctness, online/offline parity. The plumbing that prevents the most expensive class of model bugs.

FeastTectonPIT
§ 03
Where we tend to live

A pragmatic stack.

Tool-agnostic, opinionated about patterns. The shape we reach for in most engagements.

— Lakehouse
Databricks, Snowflake, BigQuery, Iceberg, Delta, Hudi
— Streaming
Kafka, Confluent, Flink, Pulsar, Kinesis, Debezium
— Orchestration
Airflow, Dagster, Prefect, dbt, SQLMesh
— Retrieval & vectors
pgvector, Weaviate, Qdrant, Pinecone, Vespa, Elastic
— Governance
Unity Catalog, Polaris, OpenMetadata, Atlan, Collibra
— Quality & eval
Great Expectations, Soda, dbt tests, Ragas, custom evals
The model is the easy part.
The substrate is the work.
— Data Engineering at Revoleap
§ 04
Adjacent practices
— 01 / Practice in chief
AI Engineering
Read →
— 02 / Foundations
Cloud
Read →
— 04 / Surface
App Engineering
Read →
— 05 / Always-on
Platform Ops
Read →

Audit the substrate.

A working session with our data principals — pipeline reliability, contract maturity, retrieval-readiness benchmarks.

Begin a conversation
§ 04
Sample artifact · the layers of a modern lakehouse
— Figure · the data architecture, by layer

The number in the dashboard equals the number in the source.

— CONSUMPTION Dashboards · AI · Agents · APIs · Notebooks — GOLD · SEMANTIC LAYER Business-facing entities, contracts, embeddings — SILVER · CURATED Cleaned, conformed, joined, modelled — BRONZE · RAW Bytes-on-disk, immutable, audit-replayable Operational DBs CDC streams SaaS APIs CRM, ERP, finance Events Kafka, streams Files S3, object External market signals — SOURCES OF TRUTH

— Every team has a lakehouse. Few have the semantic layer on top of it that makes the lakehouse legible to a strategy team, a model, or an audit. That is the layer we obsess over.