What you’ll learn
- Model a scoped relational dataset and ingestion contract.
- Implement repeatable transformations with data-quality checks.
- Inspect lineage, missing data and incremental-processing failures.
Proposed course content
6 sections · 13 illustrative lesson records
Data contracts and models3 lessons
- Introduction to Data Engineering with Python & SQL
Build repeatable Python and SQL pipelines with schema checks, incremental processing and transparent data-quality reporting.
Public introduction · Video not supplied - Define source schemas and relational keys
Define source schemas and relational keys. Module: Data contracts and models.
Enroll to access · Video not supplied - Record provenance and permitted data uses
Record provenance and permitted data uses. Module: Data contracts and models.
Enroll to access · Video not supplied
Ingestion and validation2 lessons
- Read files or authorized test endpoints
Read files or authorized test endpoints. Module: Ingestion and validation.
Enroll to access · Video not supplied - Validate types, dates and required fields
Validate types, dates and required fields. Module: Ingestion and validation.
Enroll to access · Video not supplied
SQL transformations2 lessons
- Join and aggregate with explicit assumptions
Join and aggregate with explicit assumptions. Module: SQL transformations.
Enroll to access · Video not supplied - Check duplicates, missing values and key integrity
Check duplicates, missing values and key integrity. Module: SQL transformations.
Enroll to access · Video not supplied
Python pipelines2 lessons
- Separate extraction, transformation and loading steps
Separate extraction, transformation and loading steps. Module: Python pipelines.
Enroll to access · Video not supplied - Use repeatable configuration and bounded error handling
Use repeatable configuration and bounded error handling. Module: Python pipelines.
Enroll to access · Video not supplied
Incremental processing2 lessons
- Track checkpoints and late-arriving data
Track checkpoints and late-arriving data. Module: Incremental processing.
Enroll to access · Video not supplied - Design idempotent loads and recovery tests
Design idempotent loads and recovery tests. Module: Incremental processing.
Enroll to access · Video not supplied
Quality and lineage2 lessons
- Produce validation reports and source-to-output lineage
Produce validation reports and source-to-output lineage. Module: Quality and lineage.
Enroll to access · Video not supplied - Document freshness, retention and unresolved data gaps
Document freshness, retention and unresolved data gaps. Module: Quality and lineage.
Enroll to access · Video not supplied
Requirements
- Basic Python programming and introductory SQL queries.
- Synthetic, owned or licensed datasets permitted for local analysis.
Tools and resources
- A development computer with resources appropriate to the proposed local exercises.
- Python, a local SQL database and synthetic structured data.
- A test runner, version control and optional local orchestration tools.
Description
Build repeatable Python and SQL pipelines with schema checks, incremental processing and transparent data-quality reporting.
Who this course is for
Programmers designing reliable data ingestion and transformation workflows.
Example practical project
Proposed capstone: a local pipeline that ingests synthetic order files into a relational database and produces a quality report with reproducible aggregates.