Processing Data: Integrating Kafka with Apache Spark

Apache Kafka    |    Intermediate
  • 12 videos | 1h 45m 38s
  • Includes Assessment
  • Earns a Badge
Rating 4.7 of 29 users Rating 4.7 of 29 users (29)
Flexible and Intuitive, DataFrames are a popular data structure in data analytics. In this course, build Spark applications that process data streamed to Kafka topics using DataFrames. Begin by setting up a simple Spark app that streams in messages from a Kafka topic, processes and transforms them, and publishes them to an output sink. Next, leverage the Spark DataFrame application programming interface by performing selections, projections, and aggregations on data streamed in from Kafka, while also exploring the use of SQL queries for those transformations. Finally, you will perform windowing operations - both tumbling windows, where the windows do not overlap, and sliding windows, where there is some overlapping of data.

WHAT YOU WILL LEARN

  • Discover the key concepts covered in this course
    Build a spark application that reads from a kafka topic
    Manipulate streaming data and publish the output to the console
    Subscribe to multiple kafka topics from a spark application
    Write an app that generates data to periodically send to a kafka topic
    Develop a spark application that publishes transformed data to a kafka topic
  • Transform streaming data with spark sql
    Perform aggregations on spark dataframes and order their contents
    Perform group by, aggregations, and ordering
    Describe what windows are in the context of spark streaming and define them using dataframes
    Define operations on tumbling and sliding windows
    Summarize the key concepts covered in this course

IN THIS COURSE

  • 1m 5s
  • 10m 12s
  • Locked
    3.  Transforming Kafka Messages with PySpark
    10m
  • Locked
    4.  Reading from Multiple Kafka Topics
    11m 35s
  • Locked
    5.  Setting up a Producer and Consumer with Kafka
    11m
  • Locked
    6.  Publishing to Kafka from PySpark
    10m 25s
  • Locked
    7.  Transforming Data with Spark SQL
    7m 28s
  • Locked
    8.  Aggregations on Streaming Data
    10m 49s
  • Locked
    9.  Exploring Grouping and Ordering
    10m
  • Locked
    10.  Defining Window Operations
    12m 6s
  • Locked
    11.  Creating Tumbling and Sliding Windows
    8m 41s
  • Locked
    12.  Course Summary
    2m 17s

EARN A DIGITAL BADGE WHEN YOU COMPLETE THIS COURSE

Skillsoft is providing you the opportunity to earn a digital badge upon successful completion on some of our courses, which can be shared on any social network or business platform.

Digital badges are yours to keep, forever.

YOU MIGHT ALSO LIKE

Audiobook Kafka in Action
Book Practical Machine Learning with Spark: Uncover Apache Spark's Scalable Performance with High-Quality Algorithms Across NLP, Computer Vision and ML
Course Working with Apache Cassandra: Indexes, Aggregations, & Programmatic Access
Rating 4.5 of 8 users Rating 4.5 of 8 users (8)

PEOPLE WHO VIEWED THIS ALSO VIEWED THESE

Course Building Data Pipelines
Rating 4.3 of 43 users Rating 4.3 of 43 users (43)
Course Graph Modeling on Apache Spark: Working with Apache Spark GraphFrames
Rating 4.7 of 24 users Rating 4.7 of 24 users (24)
Course Spark for High-speed Big Data Analytics
Rating 4.5 of 62 users Rating 4.5 of 62 users (62)