Resume your capacity ➜ Run pipelines ➜ Refresh Power BI datasets ➜ Suspend capacity — all automated thanks to PowerShell and Azure Automation ! 

Microsoft Fabric revolutionizes data architecture by offering a unified platform that integrates Power BI, data science, real-time analytics, and more. At the heart of this ecosystem is the Lakehouse, a powerful, flexible, and scalable storage layer tailored for modern data engineering workflows.

In this article, we explore how Lakehouses work in Microsoft Fabric, how to set one up, and how they serve as the foundation for managing both files and structured data—all without the traditional complexity of data platforms.

💡Ever wondered how to bring AI into your data engineering workflows in Microsoft Fabric?

In my latest hands-on project, I show how to automate Sentiment Analysis on customer feedback using:

• Microsoft Fabric Lakehouse

• PySpark notebooks

• Azure OpenAI (GPT-4)

• Fabric Data Pipelines

• Power BI for real-time insights

This solution is fully integrated, no external services required, and takes just 10 minutes to set up.

📖Check out the full blog and video tutorial to see it in action: 

#MicrosoftFabric #AzureOpenAI #DataEngineering #SentimentAnalysis #PowerBI #PySpark

Explore a more native and streamlined alternative to detect and anonymize PII data. With just a few lines of code, Fabric’s built-in AI functions like ai.extract and ai.generate_response allow you to identify and redact PII directly within your data pipelines - no external libraries required.

Discover the Variable Library in Microsoft Fabric, designed to empower users to define and manage variables at the workspace level. Seamlessly integrate across various workspace items, including data pipelines, notebooks, and Lakehouse shortcuts. This feature addresses several pain points and enhances the overall user experience within Fabric.

In the ever-evolving landscape of data management, optimizing storage and access times is paramount. This article delves into the innovative V-Order feature in Fabric, a game-changer for data read times and storage efficiency. Discover how V-Order's write-time optimization technique enhances performance, reduces costs, and transforms data operations. Join us as we explore the technical intricacies, benefits, and real-world applications of V-Order, and learn how it can revolutionize your data management strategies.

Learn how to connect cross Tenant Azure Data Factory (and other services) to Fabric.

Simplify analytics with Spark Connector for Microsoft Fabric Data Warehouse: seamlessly access Fabric Warehouse via a secure Spark API.

OneLake is a unified storage system in Microsoft Fabric that eliminates data silos by storing all data in a single location. Now, we’re going to discuss Direct Lake, a new way Power BI interacts with this storage for faster performance and efficiency.

Source: https://learn.microsoft.com/en-us/fabric/fundamentals/direct-lake-overview

A Guide to Working with Lakehouses in Microsoft Fabric

This guide explores the data engineering experience in Microsoft Fabric, focusing specifically on Lakehouses. This guide takes a hands-on approach, demonstrating how to work with Lakehouses and manage data effectively.

Downloadable Files 

In my last newsletter on LinkedIn, I explained how to export AdventureWorks2022 tables to csv files. If you don’t want to generate them, you can get them, as stated in the last post (after my edit in which I realized I made a mistake). My actual blog kayondata.com is currently being moved to another place, hence not updated at the moment.

CSV files are still very common, so I am using this approach to showcase some tricks and to exercise some data engineering stuff. 

In this post I share my personal opinion as to why the Microsoft Fabric Data Engineer Associate certification is relevant right now.

Which is the new Data Engineering certification that was announced during the keynote at the European Microsoft Fabric Community Conference.