Why Python, and your first script

Almost every data engineering JD lists Python right next to SQL. You'll use it to move data between systems, clean it on the way, and glue tools like Airflow and Spark together. This chapter covers only what you need for that, and skips the rest.

Running a script

Python reads a file top to bottom and runs each line in order. Save this as pipeline.py and run python pipeline.py in a terminal.

python

# a comment: Python skips this line

print("pipeline started")

rows = 3

print("rows loaded:", rows)

OUTPUT

pipeline started rows loaded: 3

on the job Every pipeline reports what it's doing with lines like these. print is the simplest version. In production you'll swap it for the logging module so messages get timestamps and levels.