Data Engineering With Snowflake And Aws

Data Engineering With Snowflake And Aws
Published 1/2024
MP4 | Video: h264, 1920x1080 | Audio: AAC, 44.1 KHz
Language: English | Size: 2.58 GB | Duration: 4h 24m

Learn the main tasks a Data Engineer perform on Snowflake with AWS

What you'll learn

Data Engineering

Snowflake

SQL

Extraction, Transformation and Data Loading

AWS

ETL

Requirements

Familiarity with SQL is recommended but not mandatory

Familiarity with AWS is recommended but not mandatory

Description

Snowflake course for data engineersThis comprehensive Snowflake course is designed for data engineers who want to improve their ability to efficiently and scalably manage data in the cloud. With a hands-on focus, participants will be guided from the basics to advanced concepts of the Snowflake platform, which provides a modern and fully managed data warehouse architecture.Benefits of using Snowflake for data engineering: Elastic scalability: one of the key benefits of Snowflake is its cloud data storage architecture, which allows for elastic scalability. This means that data engineers can easily scale resources on demand to efficiently handle variable workloads and ensure consistent performance regardless of data volume.Simplified data sharing: Snowflake offers a unique approach to sharing data across departments and teams. Using the concept of secure and controlled data sharing, data engineers can create a single data source that promotes efficient collaboration and consistent data analysis across the organisation.Seamless integration with analytics tools: Snowflake is designed to integrate seamlessly with a variety of data analytics tools, allowing data engineers to create complete ecosystems for advanced data analysis. Compatibility with standard SQL makes it easy to migrate to the platform, while interoperability with popular tools such as Tableau and Power BI expands options for data visualisation and exploration.In this course we deal with:Snowflake basicsPlatform architectureVirtual warehouses - the clustersWorking with semi-structured dataIntegrating Snowflake with AWSUsing Stages, Storage Integration, and SnowpipesUsing AWS S3, SQS, IAMAutomatic ingestion of data in near real time

Overview

Section 1: 1. Introduction

Lecture 1 1.1 Introduction - Check this before starting

Lecture 2 1.2 What is a Modern Data Platform?

Lecture 3 1.3 Introduction to Snowflake

Lecture 4 1.4 Snowflake Architecture

Lecture 5 1.5 Costs

Lecture 6 1.6 Instance Types

Section 2: 2. Understanding the Snowflake Environment

Lecture 7 2.1 Creating a free Account

Lecture 8 2.2 Exploring the Menus

Lecture 9 2.3 Exploring Worksheets

Lecture 10 2.4 Snowflake Clusters: Virtual Warehouses

Lecture 11 2.5 Default Roles

Section 3: 3. Data Ingestion 1: COPY command

Lecture 12 3.0 Snowflake Ingestion Methods

Lecture 13 3.1 Open Data for Ingestion

Lecture 14 3.2 Stage Types

Lecture 15 3.3 Configuring an External Stage

Lecture 16 3.4 Ingesting Data to Snowflake

Lecture 17 3.5 File Formats

Lecture 18 3.6 Using FLATTEN command

Section 4: 4. Integrate AWS and Snowflake

Lecture 19 4.1 Creating the Storage Integration

Lecture 20 4.2 Loading JSON to Snowflake

Lecture 21 4.3 Semi-Structured Data 1

Lecture 22 4.4 Semi-Structured Data 2

Lecture 23 4.5 Code Versioning

Lecture 24 4.6 Semi-Structured Data 3

Lecture 25 4.7 Inserting Data into a Table

Section 5: 5. Data Ingestion 2: SNOWPIPE - Microbatches

Lecture 26 5.1 Ingesting with SNOWPIPE

Lecture 27 5.2 Creating a PIPE

Lecture 28 5.3 Configuring SQS and Testing the PIPE

Lecture 29 5.4 How to check Erros in PIPES

Data Engineers,Data Analysts,Database Administrators,Analytics Engineers,Cloud Engineers,Software Engineers,Database Developers,Python Developers