The Philosophy of Azure Data Engineering - From Fundamentals to Practice
NextGen Data Aspirants Community (NDAC) Microsoft Azure Data Connect - An Interactive Knowledge Sharing Event The NextGen Data Aspirants Community (๐๐๐๐) is back with another knowledge-boosting and career-shaping session โ designed to help you master the fundamentals of Data Engineering on Azure and build a strong foundation for modern data solutions. Date: 30th Aug 2025 Time: 6:00 PM IST Duration: 1 Hour Mode: Online Session Abstract: In todayโs data-driven world, organizations need scalable and flexible platforms to transform raw data into actionable insights. This session will walk you through the core building blocks of Azure Data Engineering, from foundational concepts to practical pipeline development. Session Takeaways: By the end of this session, participants will: โข Gain a solid introduction to Data Engineering concepts. โข Understand why Azure is a game changer for modern data workloads. โข Learn the role of key Azure Data Services (ADLS, ADF, Databricks, Synapse, Power BI). โข Differentiate between Data Warehouse, Data Lake, and Lakehouse architectures. โข Explore the Medallion Architecture (Bronze, Silver, Gold) and its practical use cases. โข Watch a live demo of a data pipeline on Azure Databricks. Donโt miss this opportunity to unlock the Azure Data Blueprint and elevate your data engineering journey with NDAC! ๐น๐๐๐๐๐๐๐ ๐๐ ๐ด๐๐๐๐๐: https://www.meetup.com/nextgen-data-aspirants-community/events/310628057/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link&utm_version=v2 ๐ฑ๐๐๐ ๐๐ ๐๐ ๐ด๐๐๐๐๐: https://www.meetup.com/nextgen-data-aspirants-community/ ๐ฑ๐๐๐ ๐๐๐ ๐พ๐๐๐๐๐จ๐๐ ๐ฎ๐๐๐๐ ๐๐๐ ๐๐๐๐๐ ๐๐๐ ๐๐๐๐ & ๐๐๐๐: https://chat.whatsapp.com/GHBzZ7rKqylLyXTGJqyTxz ๐จ๐๐๐ ๐๐๐๐๐๐๐๐๐ ๐๐ ๐ด๐๐๐๐๐๐๐๐ ๐ป๐๐๐๐: https://teams.live.com/l/community/FEAVIPJvr_9CQiYFgw ๐ฑ๐๐๐ ๐๐๐ ๐พ๐๐๐๐๐จ๐๐ ๐ช๐๐๐๐๐๐ ๐ณ๐๐๐: https://whatsapp.com/channel/0029VbB2EH33AzNVJEyA6L0r Letโs Explore, Learn, and InnovateโTogether!Parquet, ADLS Gen2, ETL, and Incremental Refresh in one Power BI Dataset
A year ago, I was developing a solution for collecting and analyzing usage data of a Power BI premium capacity. There were not only some simple log files, but also data that I had to convert into a slowly changing dimension type 2. Therefore, I decided for the following architecture - Azure Data Factory pipelines collect data on daily basis, the raw data is stored in a data lake forever, and the cleansed data is then moved to a SQL Server database. Because the data is stored on a SQL Server, I can use incremental refresh in Power BI service. It works perfectly. But the times are changing, new requirements are coming, and I have found a new straight way, how to load logs directly from the data lake into a Power BI dataset. And all that incrementally! Where I need a SCD, a SQL Server database will stay in the middle, whereas for all other data I can use a new mindset. In this article I want to show you how you can load parquet files stored in an Azure data lake direct into your Power BI dataset. It involves an incremental refresh and an ETL process, too!
