§ 1.2section 2 of 25·~4 min
The data engineering lifecycle
Tools change every few years. The jobs those tools do don't. Joe Reis and Matt Housley's book Fundamentals of Data Engineering (O'Reilly, 2022) organises the whole field into a data engineering lifecycle with five stages:
GENERATION ──▶ INGESTION ──▶ TRANSFORMATION ──▶ SERVING ──▶ analytics, ML, other apps
└───────────────── STORAGE ─────────────────┘
Storage sits underneath because data is stored at every step, not just at the end.
| Stage | The question it answers | Example |
|---|---|---|
| Generation | Where is the data created? | The app writes each order to its PostgreSQL database |
| Storage | Where do we keep it, and how? | Raw files in cloud object storage, tables in a warehouse |
| Ingestion | How do we get it out of the source? | A nightly job copies new orders into the warehouse |
| Transformation | How do we make it useful? | Clean types, remove duplicates, join with customers, total by day |
| Serving | Who uses it, and how? | A sales dashboard, a churn model, a CSV for the finance team |
The undercurrents
The book also names six undercurrents: concerns that run under every stage rather than happening once.
- Security: who can see what. Mask personal data, give people the least access they need.
- Data management: knowing what data you have, what it means, and where it came from (catalogs, lineage, quality).
- DataOps: treating data like software: tests, monitoring, alerts, fast fixes.
- Data architecture: the big-picture design and trade-offs (cost, speed, flexibility).
- Orchestration: running the steps in the right order, on schedule, with retries (Airflow is the classic tool).
- Software engineering: clean code, version control, code review, CI.
The rest of this chapter walks through the stages one by one, plus the modeling ideas you need to design the tables in the middle.