Data lake, warehouse or lakehouse: read the client's data architecture before writing your first ML pipeline

Every kickoff has a moment like this. You ask where the data lives, and the client answers with complete confidence: “It’s all in the warehouse. Help yourself.”

That is usually only half true. The data exists, but someone may have shaped it for a different purpose and set it to refresh on a schedule you don’t yet know. There is a good chance it has lost exactly the signal your model needs.

Misreading a data architecture is a hard mistake to catch, because nothing breaks straight away. The pipeline runs, the model produces numbers, and it can be weeks before you realise you built on the wrong foundation. What follows shows how to read that foundation correctly in your first week.

The difference is when the data gets “shaped”

Microsoft Azure defines a data lake as a centralised repository that ingests and stores large volumes of data in its original form. The mechanism that makes this possible is schema-on-read: any kind of data is written in raw form, and structure is applied only when someone reads it.

A data warehouse works the other way round. Azure describes a warehouse as holding data that has been processed and transformed for a specific purpose. That purpose is usually reporting, and the purpose of a report rarely matches the purpose of a model.

The lakehouse is an attempt to combine the two. Databricks defines the lakehouse as an open data management architecture. It combines the flexibility, cost-efficiency and scale of a data lake with the features of a warehouse, so that BI and ML can run on the same data.

The technical core is a metadata layer that tracks which files belong to which version of a table. This layer is what gives a lakehouse ACID transactions, along with features such as schema enforcement, schema evolution and data validation.

Aspect	Data lake	Data warehouse	Lakehouse
Data form	Original, raw	Transformed for one purpose	Raw files managed by a metadata layer
Schema applied on	Read	Write	Can be enforced on write, with evolution allowed
Main risk for ML	Schema drifts and nobody notices	Signal aggregated away	Assuming data is clean when the features are not switched on
First question	Who checks data on write?	Which report does this table serve?	Are table versions kept?

A worked example: a churn prediction model

Imagine you are asked to build a model that predicts which customers are about to churn for a retail chain, with a requirement to flag them 7 days in advance. The client has a warehouse holding a monthly revenue summary table for a dashboard, and a lake holding app event logs in JSON. Every night, a job loads data from the lake into the warehouse.

This is exactly the two-tier setup that Databricks says requires constant maintenance and often leaves data stale. That is the view of a vendor selling lakehouses, but it matches fairly closely what FDEs tend to find on client sites.

The first problem appears as soon as you open the warehouse table. It has been aggregated by month, so behaviour such as “hasn’t opened the app once this week” has been compressed away. You need to warn 7 days ahead, but the data is only as granular as a month, and no amount of model tuning will recover the lost signal.

So you go back to the lake, and immediately hit the second problem: the schema may have drifted without anyone noticing. Schema-on-read means nothing stops bad data at write time, so the checking is now your job. A simple PySpark query is enough for a first pass:

from pyspark.sql import functions as F

df = spark.read.json("s3://lake-cua-khach/app_events/")
(df.groupBy(F.date_trunc("month", "event_time").alias("thang"))
   .agg(F.count("*").alias("so_dong"),
        F.sum(F.col("customer_id").isNull().cast("int")).alias("thieu_customer_id"))
   .orderBy("thang")
   .show(36))

(The aliases are Vietnamese: thang is month, so_dong is row count, thieu_customer_id is rows missing customer_id.)

If the number of rows missing customer_id sits near zero for months and then suddenly jumps, the app team very likely renamed the field in a release. The lake keeps accepting data as normal, and the dashboard keeps working because someone quietly patched the nightly job. Your feature, meanwhile, silently turns into nulls.

The last problem is freshness. Processing in large batches or in real time (streaming) is one of the core distinctions in data engineering. If the client wants to score risk the moment a customer opens the app, a nightly job cannot deliver that, and you need to know before you promise any milestone.

ETL or ELT decides what you get to keep

IBM defines a data pipeline as a system that ingests raw data from multiple sources and then transforms it. Whether you transform first or load first decides whether the raw data survives.

With ETL, data is extracted and transformed before it is loaded into the store. This suits warehouse-centric architectures, but the raw copy may be gone after transformation. With ELT, data is loaded first and transformed inside a cloud warehouse, so the raw copy remains and you can rebuild features the way your model needs them.

In the retail example, the sensible choice is to pull training data from the lake in ELT fashion, write the feature transformations yourself and manage them as your own code. If the client already has a lakehouse, use table versions to rebuild the exact training set as of a given day. Turn on schema enforcement to stop field renames at write time.

Five questions for the first week

Start with lineage. For every table the client hands you, ask where it comes from and which report it serves. A table built for the finance team’s dashboard is aggregated and filtered the way finance needs, not necessarily the way your model needs.

Next, ask whether the schema is checked on write or only on read, and who gets told when a field is renamed. Then ask about the refresh schedule: nightly batch, hourly or streaming, and the actual latency compared with what the use case needs.

The last two questions are about reproducibility and ownership. Does the client keep table versions so past data can be reconstructed? Who owns the transformation jobs? If you know who that is, you will hear about changes to a job before they happen rather than discovering them afterwards.

The most common mistakes

The most common mistake is trusting the table name. A table called “customer_features” can still be a reporting summary that has aggregated away exactly the time dimension you need.

The second is training on transformed warehouse tables but serving the model from raw lake data, or the reverse. The two paths go through different transformation logic, so what the model sees in production will differ from what it learned.

The third is hearing “lakehouse” and assuming the data is clean. Schema enforcement and data validation are available features; that does not mean every table has them switched on. Check the configuration rather than trusting the slides.

How to show this skill when job hunting

When reading a job description for an FDE role, look for terms such as lakehouse, ELT, batch and streaming: they tell you which layer of the data you will be working at. On a CV, a specific line such as “found the source table was aggregated by month, switched to raw data and rebuilt features at daily granularity” is far more convincing than a list of tools.

You can always look up the definitions of the three architectures. What is worth practising is the reflex: when you hear “it’s all in the warehouse”, you know immediately what to ask next.