Azure Data Engineering-Master 6 Real-World Projects + Fabric

Last updated 8/2023
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz
Language: English | Size: 7.46 GB | Duration: 9h 3m

Advance Your Azure Data Engineering Skills Using Microsoft Fabric And Other Azure Data Engineering Services 6 Projects

What you'll learn
The basics of Azure data engineering and the services available in Azure for data engineers
How to design, implement and manage data pipelines using Azure Data Factory, Azure SQL, Azure Storage Account, and Data Lake Storage
How to create dynamic and reusable mapping data flows in Azure Data Factory
How to use metadata-driven frameworks in real-time projects in Azure Data Factory
How to perform incremental data loading using Azure Data Factory and watermarking techniques
Core Concepts of Microsoft Fabric
Techniques for validating source schema using Azure Functions and Azure SQL
Six Real-world use cases and scenarios for data engineering in Azure
Master Azure Data Factory Advance Configurations
How to design A real-world azure data engineering solution using multiple azure services
How to log and audit data pipeline details using Azure Data Factory and Azure SQL
How to mount a storage account in Azure Databricks ?
Real time use cases of Azure Data Factory And Other Azure data engineering services
Common Azure Data Engineering Interview questions and answers
How to apply Azure services to real-world data engineering projects and use cases.
Best practices for logging and auditing data pipelines in Azure using Azure SQL
You will learn To Design Data Engineering Solution Using Azure Databricks, Azure SQL Server
How to implement incremental loading using Azure Data Factory and watermarking
How To Design Data Engineering Solution Using Azure Data lake storage Gen 2
Azure Data Factory Metadata Driven Frameworks Concepts
You will learn how to secure your credentials using Azure Key vault
You will learn how to create and store secret token
Azure Data Engineering Concepts
Azure Data Factory Dynamic Pipelines
Tracking Azure Data Factory Pipelines Runs
Azure Data Factory Metadata Driven Framework
Logging Azure Data Factory Pipeline Audit Data Using Stored Procedure
Build an end to end Azure Data Engineering project using Azure Services
Create And Use Azure Synapse Analytics For Big Data Processing
Creating and Enabling a Microsoft Fabric Account

Requirements
Basics of Azure Cloud computing
Internet Connections
Mobile Phone / Laptop /Desktop
You need azure subscriptions (Only If you want to try these demos)
If you are using Azure pay as you go subscriptions, you will be charged according to your azure resource usages

Description
Hello,"Learn to tackle real-world data engineering challenges with Azure by building hands-on projects in this comprehensive course. Dive into Azure's data engineering services such as Data Factory, Azure SQL, Azure Storage Account, and Data Lake Storage to design, implement, and manage data pipelines. This course is tailored for data engineers, data scientists, and developers looking to enhance their skills and apply them in real-world scenarios.No previous experience with Azure is required, but some background in data engineering and a general understanding of Azure will be beneficial. The course includes five practical projects that cover a range of use cases and scenarios for data engineering in Azure. By the end of this course, you will have the ability to design, construct, and manage data pipelines using Azure services.This course, Azure for Data Engineering: Real-world Projects, focuses on five practical projects that address everyday data engineering issues using Azure technologies. With an emphasis on real-world scenarios, this course aims to provide you with the skills and knowledge to apply Azure to your own data engineering projects. Whether you are new to Azure or have some experience, this course is designed to help you take your data engineering skills to the next level."Is Azure good for data engineers?Azure is a great choice for data engineers because it offers a comprehensive set of tools and services that make it easy to design, implement, and manage data pipelines. The Azure Data Factory, Azure SQL, Azure Storage Account, and Data Lake Storage are just a few of the services available to data engineers, making it easy to work with data no matter where it is stored.One of the biggest advantages of using Azure for data engineering is the ability to easily integrate with other Azure services such as Azure Databricks, Azure Cosmos DB, and Power BI. This allows data engineers to build end-to-end solutions for data processing and analytics. Additionally, Azure provides options for data governance and security, which is a critical concern for data engineers.In addition, Azure offers advanced features such as Azure Machine Learning and Azure Stream Analytics that can be used to optimize and scale data pipelines, allowing data engineers to quickly and easily process and analyze large amounts of data.Overall, Azure provides a powerful and flexible platform for data engineers to work with, making it a great option for data engineering projects and real-world scenarios.Project One: Simplifying Data Processing in Azure Cloud with Data Factory, Functions, and SQLThis course is designed for professionals and data enthusiasts who want to learn how to effectively use Azure cloud services to simplify data processing. The course covers the use of Azure Data Factory, Azure Functions, and Azure SQL to create a powerful and efficient data pipeline.You will learn how to use Azure Data Factory to extract data from various online storage systems and then use Azure Functions to validate the data. Once the data is validated, you will learn how to use Azure SQL to store and process the data. Along the way, you will also learn best practices and case studies to help you build your own real-world projects.This project is designed for professionals who want to learn how to use Azure Data Factory for efficient data processing in the cloud. The project covers the use of Azure functions and Azure SQL database for validation of source schema in Azure Data Factory.The course starts with an introduction to Azure Data Factory and its features.You will learn how to create and configure an Azure Data Factory pipeline and how to use Azure functions to validate source schema.You will also learn how to use the Azure SQL database to store and retrieve the schema validation details.Throughout the project, you will work on hands-on exercises and real-world scenarios to gain hands-on experience in implementing Azure Data Factory for data processing. You will learn how to use Azure functions to validate the source schema and how to use the Azure SQL database to store and retrieve the schema validation details.By the end of this course, you will have a solid understanding of Azure Data Factory and its capabilities, and you will be able to use it to validate source schema using Azure functions and Azure SQL database. This will enable you to design and implement efficient data processing solutions in the cloud using Azure Data Factory, Azure functions, and Azure SQL database."This project is suitable for anyone with a basic understanding of data processing, who wants to learn how to use Azure cloud services to simplify data processing.Project Two: Create dynamic mapping data flow in Azure data factoryIn this project, you will learn how to use the powerful data flow feature in Azure Data Factory to create dynamic, flexible data pipelines. We will start by learning the basics of mapping data flows and how they differ from traditional data flows. From there, we will delve into the various components that make up a mapping data flow, including source, transformations, and sink. We will then explore how to use expressions and variables to create dynamic mappings and how to troubleshoot common issues. By the end of this course, you will have the knowledge and skills to create dynamic mapping data flows in Azure Data Factory to meet the specific needs of your organization. This course is ideal for data engineers and developers who are new to Azure Data Factory and want to learn how to build dynamic data pipelines."The project will cover the following topics:Introduction to dynamic mapping data flow and its benefitsUnderstanding the concepts of mapping data flow and how it differs from traditional data flowHands-on exercises to create and configure dynamic mapping data flow in Azure Data FactoryBest practices for designing and implementing dynamic mapping data flowCase studies and real-world examples of dynamic mapping data flow in actionTechniques for troubleshooting and optimizing dynamic mapping data flowHow to process multiple files with different schema.These projects cover how you could reuse your mapping data flow, to process multiple files with different schema. It is very easy to design your mapping data flow and process files with the same schema. In this course, we will learn how you could create dynamic mapping data flow so that you could reuse your entire complicated transformations to transform your files and tables with different schema.Project three: Real-time Project using Metadata Driven Framework in Azure Data FactoryImplement a Metadata driven framework to load multiple source tables from your source system to your Azure Storage account. In this project, we will take our azure data processing approach one step further by making ADF data pipelines metadata-driven. In a metadata-driven approach, you can process multiple tables and apply different transformations and processing tasks without redesigning your entire data flows.This Project is designed to provide hands-on experience to the participants in implementing a real-time project using a metadata-driven framework in Azure Data Factory. The course will cover the concepts of a metadata-driven framework and its implementation in ADF. after this project, you will learn how to design and implement a metadata-driven ETL pipeline using ADF and how to use ADF's built-in features to optimize and troubleshoot the pipeline.By the end of the project, you will have a strong understanding of the Metadata Driven Framework in Azure Data Factory and how to use it in real-time projects. You will be able to design and implement data pipelines using the framework and will have the skills to optimize and troubleshoot them.This project is perfect for data engineers, data architects, and anyone interested in learning more about the Metadata Driven Framework in Azure Data Factory.Project Outline:Introduction to Metadata Driven Framework in ADFSetting up the Metadata RepositoryDesigning the Metadata-Driven PipelineImplementing the Metadata-Driven PipelineOptimizing and Troubleshooting the PipelineReal-time Project Implementation using Metadata Driven FrameworkCase Studies and Best PracticesPrerequisites:Basic knowledge of Azure Data FactoryBasic understanding of ETL conceptsFamiliarity with SQL scripting.Target Audience:Data EngineersETL DevelopersData ArchitectsProject four: Incremental Data Loading in the Cloud: A Hands-on Approach with Azure Data Factory and WatermarkingIn this project, you will learn how to implement incremental load using Azure Data Factory and a watermark table. This is a powerful technique that allows you to only load new or updated data into your destination, rather than loading the entire dataset every time. This can save a significant amount of time and resources.You will learn how to set up a watermark table to track the last time a load was run and how to use this information in your ADF pipeline to filter out only new or updated data. You will also learn about the different types of incremental loads and when to use them. Additionally, you will learn about the benefits and best practices for using this technique in real-world scenarios. By the end of this course, you will have the knowledge and skills to implement incremental load in your own projectsThis course will guide you through the process of how to efficiently load and process large amounts of data in a cost-effective and timely manner, while maintaining data integrity and consistency. The course will cover the theory and best practices of incremental loading, as well as provide hands-on experience through practical exercises and real-world scenarios. By the end of the course, you will have a solid understanding of how to implement incremental loading for multiple tables using Azure Data Factory and watermarking, and be able to apply this knowledge to your own projectsProject Five: Auditing and Logging Data Pipelines in Azure: A Hands-on ApproachIn this project, you will learn how to implement a robust auditing and logging system for your Azure Data Factory pipelines using Azure SQL and stored procedures. You will gain a deep understanding of how to capture and store pipeline execution details, including start and end times, status, and error messages.You will also learn how to use stored procedures to query and analyze your pipeline logs to identify patterns and trends. Throughout the project, you will work on real-world examples and use cases to solidify your knowledge and skills. By the end of this project, you will have the knowledge and skills needed to implement an efficient and effective auditing and logging system for your Azure Data Factory pipelines.In this project, we will learn how to log audit details.Using system variables.Using the output of exciting activities.Using the current item from your for each loop.Using dynamic expressions.By the end of the project, participants will have a thorough understanding of how to implement an advanced monitoring and auditing system for their Azure Data Factory pipelines and be able to analyze and troubleshoot pipeline performance issues more effectively."Project Six: Introductions To Azure Synapse Analytics And Azure Data EngineeringWe are excited to announce the release of a new module in our Azure Data Engineering course, dedicated to explaining Azure Synapse Analytics. This module covers everything you need to know about Azure Synapse Analytics, from what it is and how to create it, to the different components and how to access data from an Azure Data Lake Gen2.If you are not familiar with Azure Synapse Analytics, it is a limitless analytics service that brings together big data and data warehousing. It provides a unified experience for data ingestion, big data processing, and data warehousing, and allows you to query both structured and unstructured data using the same familiar tools and languages.In our new module, we cover all the essential topics related to Azure Synapse Analytics, including:What is Azure Synapse Analytics, and why should you use it?How to create an Azure Synapse Analytics workspaceThe different components of Azure Synapse Analytics, such as SQL pools, Spark pools, and PipelinesHow to access data from an Azure Data Lake Gen2 using Azure Synapse AnalyticsWe have designed this module to be easy to follow, with step-by-step instructions and real-world examples to help you understand how Azure Synapse Analytics works and how it can be used in your own projects.By the end of this module, you will have gained a deep understanding of Azure Synapse Analytics and how it can be used to solve big data and data warehousing challenges. You will also have the skills necessary to create an Azure Synapse Analytics workspace and access data from an Azure Data Lake Gen2.So, whether you are a data engineer, data scientist, or data analyst, our new module on Azure Synapse Analytics is the perfect way to deepen your knowledge of this powerful analytics service. Enroll in our Azure Data Engineering course today to access this new module and start learning!Project Seven (New): Introduction to Microsoft Azure Fabric (Added On Aug-2023)Welcome to the "Introduction to Microsoft Fabric" module! This comprehensive course is designed to equip you with essential knowledge about Microsoft Fabric, an all-in-one analytics solution for enterprises that covers data movement, data science, real-time analytics, and business intelligence. Whether you're a beginner or an experienced professional, you'll delve into the core concepts of Microsoft Fabric, learn to create and enable a Microsoft Fabric account, and explore the process of creating a new workspace within the Microsoft Fabric environment. Stay tuned for upcoming lectures. Join us on this exciting journey into the world of Microsoft Fabric, where simplicity, integration, and end-to-end solutions await!Please Note: This course covers advanced topics in Azure Data Factory, and while prior knowledge of the platform is beneficial, it is not required as we will be covering all necessary details from the ground up. So, whether you're new to Azure Data Factory or looking to expand your existing knowledge, this course has something to offer everyonePlease Note: This course comes with a 30-day money-back guarantee. If you are not satisfied with the course within 30 days of purchase, Udemy will refund your money, (Note: Udemy refund conditions are applied)

Overview
Section 1: Azure Data Engineering Real World Project 1: First Part

Lecture 1 Introduction To Project 1

Lecture 2 Introductions To Part 1 Of This Project

Lecture 3 Save Raw Data In GitHub

Lecture 4 Create Azure Data Lake Storage Gen 2 Account (ADLS)

Lecture 5 How To Create Azure Data Factory Account

Lecture 6 How To Create Containers in ADLS?

Lecture 7 How To Create Linked Services ADF?

Lecture 8 How To Create Data Set In ADF?

Lecture 9 How To Create A Pipeline In ADF and Configure Copy Activity

Lecture 10 Create New Data Set and Copy Second files

Lecture 11 How To Reuse Data Set With The Help Of Parameter

Lecture 12 Copy 16 Files Using Single Copy Activity

Section 2: Azure Data Engineering Real World Project 1:-Second Part

Lecture 13 Azure Functions: Intro

Lecture 14 How to Test & Validate Blob Trigger Functions In Azure Functions App

Lecture 15 How To Add Logical Testing Code In Azure Functions, For Validations

Lecture 16 How To Add Output Binding in Azure Functions

Lecture 17 End To End Testing HTTP to Azure Storage Using ADF And Validate Functions App

Lecture 18 Azure Function App: Fix File Name Issues

Section 3: Azure Data Engineering Real World Project 1: Final Part

Lecture 19 Final Part Of This Project

Lecture 20 How To Create Azure SQL DB ?

Lecture 21 How to Connect To Azure SQL Using SSMS & From Azure Portal

Lecture 22 How To Create Linked Service To Access Azure SQL

Lecture 23 How To Create Data Set To Access Azure SQL DB?

Lecture 24 How To Copy Data Into Azure SQL

Lecture 25 How To Copy Full Data Into Azure SQL

Lecture 26 How To Fix Common Issues

Section 4: Project 2- Part 1: Mastering ADF Dynamic Pipelines

Lecture 27 Introductions To Project Requirements

Lecture 28 Understand Data and Data Transformations Requirements

Lecture 29 Design Target Table For First Data Set

Lecture 30 Create Data Set: Azure Data Lake and Azure SQL Data set

Lecture 31 Create Data Flow And Add Multiple Source ( ADLS File & Azure SQL Table)

Lecture 32 Make Our Data Flow Using Parameters

Lecture 33 How To Derive New Columns From Existing Columns And Parameters.

Lecture 34 How To Use Exist To Validate Source And Target Data

Lecture 35 Calculate New Surrogate Key And Max Surrogate Key

Lecture 36 Join Max Surrogate Key With New (Or Updated) Data Set

Lecture 37 Derive Additional Columns: Active Status ,and Current Dates

Lecture 38 Select Relevant Column Using Select Activiti-Role Based Mapping

Lecture 39 Process Updated Data Using New Branch Activity

Lecture 40 Select Proper Columns Using Role Based Mappin(Different Expression)

Lecture 41 Define Insert Set Data And Update Set Data

Lecture 42 Merge Insert And Update Data Sets

Lecture 43 Add Sink And Execute Our Pipeline

Lecture 44 Unit Testing: Validate Pipeline Execution Step 1

Lecture 45 Unit Testing : Validate Pipeline Executions Step 2

Section 5: Project 2-Part 2: Mastering ADF Dynamic Pipelines

Lecture 46 Introduction To New Data Set

Lecture 47 Make Our Data Set Dynamic Using Parameters

Lecture 48 Make Our Pipeline Dynamic

Lecture 49 Execute Our Pipeline With New Data Set

Section 6: Project 2-Part 3: Mastering ADF Dynamic Pipelines

Lecture 50 Introductions To Final Requirements

Lecture 51 Defining Table To Store Structure Of The Table

Lecture 52 How Define A Dynamic Stored Procedure To Read File Structure

Lecture 53 How To Validate File Structure of Two files

Lecture 54 How To Store Structure Details In SQL Table

Lecture 55 Validate Structure Using Azure SQL table and Stored Procedures

Lecture 56 Execute New Pipeline After validations

Lecture 57 Test All the Scenarios (Same Structure , different Structures ) End To END Unit

Section 7: Project 3: Introductions To Azure Databricks & Mount Azure Data Lake

Lecture 58 Create Azure Data Lake Gen 2 And Azure Databricks

Lecture 59 Register an application with Azure AD and create a service principal

Lecture 60 Assign Roles To The Application To Provide The Service Principal Permissions

Lecture 61 Add application secret to the Azure Key Vault

Lecture 62 Create a Secret Scope in Azure Databricks

Lecture 63 Create Containers ( bronze/ Raw, silver / Processed , and gold/Final)

Lecture 64 Create Your First Cluster in Databricks

Lecture 65 Create A Notebook

Lecture 66 Mount Azure Data Lake without Key Vault

Lecture 67 Read CSV file from Data Lake

Lecture 68 Mount Data lake using Azure Key Vault

Section 8: Project 4: Introductions To Azure Data Factory Metadata Driven Framework

Lecture 69 Introductions To Azure Data Factory Metadata Driven Framework

Lecture 70 Create Azure SQL Data Base (Source System)

Lecture 71 Create New Schema For Metadata Tables

Lecture 72 Create Linked Services To Access Source and Sink

Lecture 73 Create Source Data Set Using Linked Services

Lecture 74 Read Our Metadata From ConfigDB Using Lookup Activity

Lecture 75 Configure For Each Activity To Process Each Metadata Entries

Lecture 76 Add Copy Activity And Configure Sink Settings

Lecture 77 Configure Source Settings As Dynamic SQL Query

Lecture 78 Configure Sink To Save into Correct Folder

Lecture 79 Configure Metadata Table Load Multiple Source Tables Into Sink

Lecture 80 Configure Metadata To Load Tables With Relevant Columns and Proper Name (Rename)

Section 9: Project 5: Log Azure Data Factory Pipeline Executions Azure SQL&Stored Procedure

Lecture 81 Logging Audit Data Requirements

Lecture 82 Create New Metadata Table For Saving Log Details

Lecture 83 Create Stored Procedure For Saving Log Details Into A Azure SQL Table

Lecture 84 Configure Stored Procedure Activity To Log Error Details From System Variables

Lecture 85 Configure Stored Procedure Activity To Log Error Details From Activity Output

Lecture 86 Configure Stored Procedure Activity To Log Error Details Using Expressions

Lecture 87 Execute Our Pipeline And Verify Our Log Details

Section 10: Incremental Load: Delta Data Loading From Database By Using A Watermark Table

Lecture 88 Create New Tables And Insert Data

Lecture 89 Create Watermark Table And Insert Data

Lecture 90 Create Stored Procedure To Update Watermark Table

Lecture 91 Create Pipeline And Configure Lookup Activity: Existing WatermarkTable,Parameter

Lecture 92 Configure Lookup Activity To Get New Watermark Value

Lecture 93 Configure Copy Activity

Lecture 94 Configure Stored Procedure Activity

Lecture 95 Add Parent Pipeline And Execute Incremental Load For Multiple Tables

Lecture 96 Re-execute After Inserting New Values (End To End Testing)

Section 11: Incremental And Full Load Using Meta Data Framework

Lecture 97 Understand Our New Requirement And Design New Metadata Framework

Lecture 98 Insert Metadata Entries

Lecture 99 Update/Create Our Stored Procedure

Lecture 100 Create And Configure Full Load Pipeline

Lecture 101 Configure Parent Pipeline

Lecture 102 Execute Full Load Pipeline

Lecture 103 Configure Incremental Pipeline

Lecture 104 Configure Incremental Parent Pipeline

Lecture 105 Execute Full Load And Incremental Load Using Metadata Entries

Section 12: Azure Synapse Analytics - Environment Setup

Lecture 106 Introduction To Azure Synapse Analytics

Lecture 107 Create Azure Synapse Analytics

Lecture 108 Azure Synapse Analytics Workspace Overview

Lecture 109 Azure Synapse Studio Overview

Lecture 110 Azure Synapse Studio Home Tab

Lecture 111 Azure Synapse Studio Data Tab

Lecture 112 Azure Synapse Develop Tab

Lecture 113 Azure Synapse Integrate Tab

Lecture 114 Azure Synapse Moniter Tab

Lecture 115 Azure Synapse Manage Tab

Lecture 116 Create the Apache Spark pool in Synapse

Lecture 117 Introductions To Azure Synapse Spark Pool & Notebook

Lecture 118 Access Azure Data Lake G2 (primary) Using Azure Synapse Notebook (Spark Pool)

Lecture 119 Role Assignment and Creating New Linked Service-Azure Synapse Analytics

Lecture 120 Read CSV files from Other ADLS g2 Using Our Azure Synapse Notebook

Section 13: Introduction to Microsoft Azure Fabric (Added On Aug-2023)

Lecture 121 An Introduction to Microsoft Fabric: What is Microsoft Fabric ?

Lecture 122 How to Create and Enable a Microsoft Fabric Account ?

Lecture 123 How to Create a New Workspace in Microsoft Fabric ?

Lecture 124 Upcoming Modules

Section 14: Azure Data Factory Interview Questions

Lecture 125 Question 1

Lecture 126 Question 2

Lecture 127 Question 3

Lecture 128 Question 4

Lecture 129 Question 5

Lecture 130 Question 6

Lecture 131 Questions 7

Lecture 132 Questions 8

Lecture 133 Questions 9

Lecture 134 Question 10

Lecture 135 Question 11

Lecture 136 Question 12

Lecture 137 Question 13

Lecture 138 Question 14

Lecture 139 Question 15 & 16

Section 15: Bonus

Lecture 140 Bonus

To Gain Hands-on Real-Time Project Experience in Azure Data Engineering Services as a Data Engineer,This course is designed for data engineers, data scientists, and developers who want to take their skills to the next level.,This course is also suitable for those who are new to Azure and want to learn how to use Azure services for data engineering.,This course is also a good fit for those who are experienced in data engineering but want to learn how to apply Azure to their data engineering projects.,This course also good for professionals who are looking to take their data engineering skills to the next level by learning how to build and manage data pipelines using Azure services.,If You Are Looking For A Real World Data engineering Uses Cases, then this course is for you,Any student who is planning to learn azure data factory , azure data bricks or Azure Data Engineering,For all the database developer who wants to learn azure data engineering,For business analyst and data analyst who wants to learn azure data engineering,Beginners: Those new to Microsoft Fabric and cloud computing can join the course to build a strong foundation and grasp the core concepts, enabling them to start their analytics journey with confidence,Cloud Computing Enthusiasts: Individuals interested in cloud computing and its applications will find this course as an opportunity to explore Microsoft Fabric, an integrated analytics solution, and its role in simplifying data-driven workflows

Buy Premium Account From My Download Links & Get Fastest Speed.