What a data engineer does

Every app you use produces data: orders placed, payments settled, buttons tapped, deliveries tracked. Before an analyst can chart it or a model can learn from it, someone has to collect it, move it, clean it and serve it reliably, every single day. That person is the data engineer.

A data engineer builds and runs the systems that turn raw data from many places into trustworthy data that other people can use. You're not the one asking the business question. You make sure the answer is built on data that is complete, correct and on time.

Where the role came from

In his 2017 essay The Rise of the Data Engineer, Maxime Beauchemin (who later created Apache Airflow and Apache Superset) describes joining Facebook in 2011 as a business intelligence engineer and leaving in 2013 as a data engineer. His point: the job had changed. Drag-and-drop ETL tools were giving way to pipelines written as code, and the role had become closer to software engineering than to reporting. Data engineers build tools, infrastructure, frameworks and services.

What the work looks like

A realistic week might include:

• Building a pipeline that copies yesterday's orders from the app database into the warehouse every night.
• Fixing a job that broke because another team renamed a column.
• Designing tables so analysts' queries are fast and give the same answer every time.
• Adding checks that alert you when today's row count drops by half.
• Making a slow, expensive query cheaper.
• Preparing a clean, well-documented dataset that data scientists can train a model on.

Who does what

The lines blur in small companies, where one person may do three of these jobs.