§ 1.3section 3 of 25·~5 min
Data generation: where data comes from
Data is generated in source systems: the apps, devices and partners that create it. You usually don't own them. Another team or another company does, and your job is to get data out of them without breaking anything.
| Source | Example | How you usually get it |
|---|---|---|
| Application database | The orders table behind a shopping app | Query a read replica, or capture changes (CDC) |
| Events | "user tapped Add to cart" sent by the mobile app | A stream such as Kafka, or files of events |
| Logs | Web server and application logs | Log shippers into storage |
| Third-party APIs | Payment gateway settlements, ad platform spend | Call the API on a schedule |
| Files | A vendor's daily CSV over SFTP or email | Pick up and load the file |
| Devices (IoT) | Delivery bike GPS pings, smart meter readings | A stream of small messages |
Three shapes of data
- Structured: fixed columns and types, like a database table. Easy to query.
- Semi-structured: has structure, but it can vary from record to record: JSON, XML, event payloads.
- Unstructured: no fixed schema: images, PDFs, audio, free text.
A click event usually arrives as semi-structured JSON:
{"event": "add_to_cart", "user_id": 81231, "sku": "TSHIRT-M-BLK",
"price_paise": 49900, "ts": "2026-10-18T09:14:03+05:30",
"device": {"os": "android", "app_version": "5.2.1"}}
Questions to ask about any source
- Who owns it, and who do I call when it changes?
- How is data created: only inserted, or also updated and deleted?
- How much arrives, and how fast: a file a day, or thousands of events a second?
- What's the schema, and how often does it change?
- Can I read it without slowing down the app that depends on it?
- Does it contain personal data (names, phone numbers, Aadhaar or PAN) that needs protecting?
These are often summed up as the 3 Vs: volume (how much), velocity (how fast) and variety (how many shapes).