Continuous Data: Automating Data Ingestion from Cloud Storage into Snowflake

Snowflake    |    Intermediate
  • 11 videos | 1h 27m 27s
  • Includes Assessment
  • Earns a Badge
Rating 4.6 of 28 users Rating 4.6 of 28 users (28)
The Snowpipe feature allows Snowflake to input micro-batches of data as it becomes available, generally within minutes of the data being added to a stage and submitted for ingestion. In this course, you will implement the auto-ingestion of CSV files from external Snowflake stages located on the AWS and Azure cloud platforms. You will begin by setting up a continuous data ingestion pipeline where the data source is located in an Azure Storage Container. This pipeline will include several components, such as queues, enterprise applications, and storage integrations as well as the permissions required to get these pieces to talk to one another. You will then implement something similar with an Amazon S3 bucket as the source of data. This set-up will involve AWS services such as IAM roles, SNS topics, as well as Snowflake objects such as notification integrations and pipes.

WHAT YOU WILL LEARN

  • Discover the key concepts covered in this course
    Set up an azure storage account to serve as an external snowflake stage
    Configure a snowflake integration to be notified of any file additions to an azure cloud storage container
    Link the azure and snowflake resources with a snowpipe for continuous data ingestion
    Test out the auto-ingesting of data from azure storage into a snowflake table
    Set up an amazon web services (aws) identity and access management (iam) role for the ingestion of data into snowflake
  • Create a snowflake storage integration to set up a simple storage service (s3) bucket as an external stage
    Create a simple notification service (sns) topic and configure the permissions needed for an s3 bucket and snowflake account
    Create a pipe to auto-load data from amazon s3 into a snowflake table
    Ingest data continuously from s3 into snowflake
    Summarize the key concepts covered in this course

IN THIS COURSE

  • 2m 1s
  • 10m 33s
  • Locked
    3.  Creating an Azure Notification Integration
    12m 31s
  • Locked
    4.  Integrating Snowflake with Azure
    9m
  • Locked
    5.  Streaming Data from Azure
    9m 13s
  • Locked
    6.  Creating an AWS IAM Role
    9m 19s
  • Locked
    7.  Setting Up an S3 Integration in Snowflake
    10m 23s
  • Locked
    8.  Configuring an SNS Topic
    10m 30s
  • Locked
    9.  Integrating AWS with Snowflake
    4m 24s
  • Locked
    10.  Testing the Integration with AWS
    7m 8s
  • Locked
    11.  Course Summary
    2m 25s

EARN A DIGITAL BADGE WHEN YOU COMPLETE THIS COURSE

Skillsoft is providing you the opportunity to earn a digital badge upon successful completion on some of our courses, which can be shared on any social network or business platform.

Digital badges are yours to keep, forever.

YOU MIGHT ALSO LIKE

Course Developing Solutions for Microsoft Azure: Azure API Management
Rating 4.5 of 13 users Rating 4.5 of 13 users (13)
Course Data Loading in Snowflake: Unloading Data
Rating 4.8 of 32 users Rating 4.8 of 32 users (32)
Course Microsoft Azure Administrator: Storage Account Blob Management
Rating 4.7 of 36 users Rating 4.7 of 36 users (36)

PEOPLE WHO VIEWED THIS ALSO VIEWED THESE

Course Queries in Snowflake: Search Optimization, External Table Partitions, & Views
Rating 4.6 of 43 users Rating 4.6 of 43 users (43)
Course Continuous Data: Ingesting Continuous Data in Snowflake
Rating 4.8 of 33 users Rating 4.8 of 33 users (33)
Course Data Loading in Snowflake: Using External Stages
Rating 4.6 of 32 users Rating 4.6 of 32 users (32)