Upstream and downstream

Data flows like a river. Everything that feeds your data is upstream. Everything that uses your output is downstream. The terms are relative: your pipeline is downstream of the app database and upstream of the dashboard.

text

 app database ──▶ raw.orders ──▶ clean.orders ──▶ fct_daily_sales ──▶ sales dashboard

 (upstream)                                                       └──▶ demand-forecast model

                                                                         (downstream)

Producers (upstream) create the data: app teams, vendors, devices.
Consumers (downstream) depend on it: analysts, data scientists, finance, other pipelines.

Why it matters: changes flow downstream

Say the app team renames amount to amount_paise in the orders table. Nobody tells you. Tonight one of two things happens:

Your pipeline fails loudly. Annoying, but at least you know.
Your pipeline keeps running and loads NULL amounts. Tomorrow morning the finance dashboard shows zero revenue, and nobody trusts the data team for a month.

The second is worse. Good pipelines fail loudly on surprises instead of quietly loading bad data.

Lineage

Data lineage is the map of which tables feed which. Before you change or drop a column, you check the lineage to find everything downstream that might break. Tools such as dbt draw lineage graphs automatically from your SQL.

Data contracts

A data contract is an agreement between a producer and its consumers: which fields exist, their types and meaning, how fresh the data is, and how changes are announced.

yaml

dataset: orders

owner: checkout-team

freshness: new rows within 1 hour

fields:

  order_id:     {type: bigint, unique: true, nullable: false}

  amount_paise: {type: bigint, nullable: false, meaning: "total incl. GST, in paise"}

  status:       {type: string, allowed: [placed, paid, shipped, cancelled]}

changes: announced 14 days ahead in #data-contracts

on the job "I didn't know anyone used that column" is one of the most common ways dashboards break. Check what's downstream before every change, and tell the consumers before you ship it.