Building Data Pipelines
Apache Airflow 1.10
13 Videos | 1h 14m 40s - Includes Assessment
- Earns a Badge
Explore data pipelines and methods of processing them with and without ETL (extract, transform, load). In this course, you will learn to create data pipelines by using the Apache Airflow workflow management program. Key concepts covered here include data pipelines as an application that sits between raw data and a transformed data set, between a data source and a data target; how to build a traditional ETL pipeline with batch processing; and how to build an ETL pipeline with stream processing. Next, learn how to set up and install Apache Airflow; the key concepts of Apache Airflow; how to instantiate a directed acyclic graph in Airflow. Learners are shown how to use tasks and include arguments in Airflow; how to use dependencies in Airflow; how to build an ETL pipeline with Airflow; and how to build an automated pipeline without using ETL. Finally, learn how to test Airflow tasks by using the airflow command line utility, and how to use Apache Airflow to create a data pipeline.
WHAT YOU WILL LEARN
-
describe data pipelines and automation
build a traditional ETL pipeline with batch processing
build a ETL pipeline with stream processing
setup and install Apache Airflow
describe the key concepts of Apache Airflow
create and instantiate a directed acyclic graph in Airflow -
use tasks and include arguments in Airflow
use dependencies in Airflow
build an ETL pipeline with Airflow
build an automated pipeline without using ETL
test Airflow tasks using the airflow command line utility
use Apache Airflow to create a data pipeline
IN THIS COURSE
-
1.Course Overview1m 38sUP NEXT
-
2.Data Pipelines Overview4m 17s
-
3.Traditional ETL Pipeline with Batch Processing3m 45s
-
4.Data Pipeline Tools3m 54s
-
5.Setup and Install Airflow3m 29s
-
6.Apache Airflow6m 47s
-
7.Airflow Workflows7m 53s
-
8.Airflow Tasks8m 24s
-
9.Airflow Dependencies4m
-
10.ETL Pipeline with Airflow8m 20s
-
11.Automated Pipeline without ETL6m 35s
-
12.Airflow Command Line Testing5m 34s
-
13.Exercise: Using Apache Airflow4m 34s
EARN A DIGITAL BADGE WHEN YOU COMPLETE THIS COURSE
Skillsoft is providing you the opportunity to earn a digital badge upon successful completion of this course, which can be shared on any social network or business platform
Digital badges are yours to keep, forever.
