Ingestion: moving data
Ingestion gets data out of source systems and into your platform. Three choices shape every ingestion job.
Batch or streaming
| Batch | Streaming | |
|---|---|---|
| How | Move a chunk on a schedule | Move each event as it happens |
| Delay | Minutes to a day | Seconds or less |
| Example | Load yesterday's orders at 2 a.m. | Flag a suspicious card payment right now |
| Tools | SQL, Python, Spark, Airflow | Kafka, Flink, Spark Structured Streaming |
| Complexity | Lower | Higher |
Most company data is still moved in batches. Use streaming when the business truly needs an answer in seconds, not because it sounds modern.
Full or incremental loads
- Full load: copy the whole table every run. Simple, and fine for small tables like a list of 30 warehouses.
- Incremental load: copy only what changed since the last run. Needed once a table has millions of rows.
An incremental load usually remembers a watermark: the latest timestamp it has already loaded.
Change data capture (CDC)
Databases keep a log of every insert, update and delete (PostgreSQL's write-ahead log, MySQL's binlog). Change data capture reads that log and turns each change into an event, without querying the tables themselves. Debezium is a popular open-source CDC tool.
Push or pull
- Pull: your job asks the source for data (query a database, call an API).
- Push: the source sends data to you (webhooks, events published to a stream).
watch out An incremental load based on updated_at never sees rows that were deleted at the source: they simply stop appearing. If deletes matter (cancelled accounts, GDPR or DPDP erasure requests), use CDC or a periodic full comparison.