Fabric Metadata & Data Quality Assessment Framework
Overview A reusable Microsoft Fabric notebook for metadata discovery, single-pass PySpark data profiling, rule-based data quality assessment, and Data Quality Index (DQI) scoring. Key Capabilities Dual Execution Modes: Run on synthetic e-commerce data or target existing Delta tables in OneLake. High-Performance Profiling: Single-pass PySpark aggregations compute distinct counts, null ratios, and boundaries in a single Spark action call. Multi-Dimension Quality Auditing: Evaluates Completeness, Uniqueness, and Validity using custom weighted thresholds. Executive Reporting: Generates an aggregate Data Quality Index (DQI) score and automated remediation guidance. Use Cases Lakehouse data quality monitoring and health checks Data product readiness for Gold layer consumption Pre- and post-migration validation Enterprise data platform modernization Full Notebook & Code: Download the complete .ipynb notebook file from the Github RepositoryFabric Spark Pool Optimiser
Fabric Spark Pool Optimiser — right-size your Spark pools in 3 minutes Every workspace in Microsoft Fabric gets the same default Spark pool. Medium node, up to 10 nodes. Nobody changes it — even in production, even when the actual workload is a 5,000-row dimension table or a monitoring notebook reading 0.002 GB. This notebook analyses 7 days of real Spark session history across all your workspaces and tells you exactly which pools are oversized, undersized, or correctly sized — with a step-by-step configuration guide for each one. What it does: - Auto-discovers all workspaces you have access to - Detects orchestrator workspaces automatically (runMultiple / Data Factory) - Separates automated pipeline sessions from interactive dev sessions — dev sessions skew duration data and are excluded from the CU calculation - Analyses GB read/written/shuffled via the Spark History stages API - Estimates monthly CU savings based on real usage - Renders an interactive dashboard directly in the notebook output No lakehouse needed. No configuration. Just import and Run All. Tested across two organisations. In one run: 50 workspaces analysed, 8 pools to change, 1,441 CU estimated monthly saving. Feedback welcome — especially if you find API behaviour that differs in your environment. https%3A%2F%2Fgithub.com%2Fenekoegiguren%2Ffabricsparkpooloptimiser
Semantic Link – Dataflow Gen1 to Gen2 Migration Assistant
Semantic Link – Dataflow Gen1 to Gen2 Migration Assistant is a reusable Fabric Notebook that helps developers safely modernize Power BI semantic models by detecting Dataflow Gen1 sources, mapping them to Dataflow Gen2 equivalents, updating the model BIM, creating a backup, rebinding the correct Fabric connection, refreshing the model, and validating the result. The tool follows a dry-run-first workflow to reduce manual edits, credential mistakes, and migration risk. Link Repooooooo!!: https://github.com/vicente2121/ChallengeSemantiklabs.git https%3A%2F%2Fgithub.com%2Fvicente2121%2FChallengeSemantiklabs%2Fblob%2Fac67250f4044e0db6d63b87a2fd84b36913e5089%2FSemantic%2520Link%2520%25E2%2580%2593%2520Dataflow%2520Gen1%2520to%2520Gen2%2520Migration%2520Assistant.ipynb
Semantic Link - Power BI Fixer
🔧 Power BI Fixer — The All-in-One Development Environment for Power BI in Fabric Author: Alexander Korn · Solution Engineer Data Platform @ Microsoft Repository: github.com/KornAlexander/semantic-link-labs (branch: feature/pbi-fixer-ui) Website: actionablereporting.com/pbi-fixer Built on: Semantic Link Labs by Michael Kovalsky 📋 Abstract The Power BI Fixer is an interactive development environment that runs natively inside Microsoft Fabric Notebooks. It combines Semantic Link, ipywidgets, and TOM (Tabular Object Model) to provide a comprehensive tool for scanning, exploring, fixing, and documenting Power BI reports and semantic models — all from a single notebook cell. The Problem It Solves Power BI developers face a fragmented workflow: Report issues (pie charts, missing data labels, page sizes) require manual visual-by-visual fixes in Power BI Desktop Model best practices (missing descriptions, floating point types, foreign key visibility) need Tabular Editor or manual scripting No single tool lets you scan a report AND its underlying model, see the results side-by-side, and fix everything in one workflow PBIR format adoption is slow because upgrade tooling is scattered The PBI Fixer addresses all of these by providing 12+ interactive tabs, 17 report fixers, 25+ semantic model fixers, and a unified scan-and-fix workflow — running entirely in a Fabric Notebook. ~17,000 lines of new Python code across 5 UI & tab modules (9,439 lines — incl. Fix All, Model Explorer, Report Explorer, Prototype, Translations, Model Diagram, About), 15 report fixers (3,038 lines), 4 report helpers (593 lines), and 33 semantic model fixers (3,797 lines) — all written from scratch for this project. 🎬 Demo Video Watch the full walkthrough: Power BI Fixer v2 Demo on YouTube https%3A%2F%2Fgithub.com%2FKornAlexander%2Fpbi_fixer%2Fblob%2Fmain%2F2026_SemanticLink_AlexanderKorn_PowerBIFixer.ipynbAutomated Static RLS Role Management for Power BI Semantic Models
Dynamic RLS is the go-to solution when Power BI security requirements are complex — multiple dimensions, multiple filters, users mapped to many combinations. But on large fact tables it evaluates USERPRINCIPALNAME() on every query for every user, and at scale that cost is significant. This notebook takes a different approach: it reads an RLS mapping table from your Lakehouse, generates static roles automatically, applies DAX filters on dimension tables (letting relationships propagate to fact tables), and assigns members by UPN email — all programmatically via the Tabular Object Model (TOM). ──────────────────────────────────────── WHAT IT DOES ──────────────────────────────────────── → Reads distinct security values from a Spark DataFrame (e.g. Country, Brand, Company) and generates one role per value → Applies role-specific DAX filters on dimension tables (DT_*), not fact tables — filters propagate through relationships automatically → Supports global filters applied to every role on their own fixed table (e.g. a consolidation flag always on DT_Customer, an active flag on DT_Product) → Handles create-or-replace — safe to re-run at any time, existing roles are removed and recreated cleanly → Adds members from a Username (UPN) column, saving one member at a time to isolate invalid UPNs without blocking valid ones → Exports a JSON failure report to Lakehouse Files for any members that could not be saved → Supports partial runs via config_keys — process only the dimensions you need ──────────────────────────────────────── REQUIREMENTS ──────────────────────────────────────── - Microsoft Fabric workspace with Lakehouse attached to the notebook - Power BI Semantic Model published to the workspace - XMLA Read/Write enabled on the Fabric capacity - semantic-link-labs (installed automatically via %pip install) - RLS source table with: Username (UPN email) + one column per secured dimension ──────────────────────────────────────── STATIC VS DYNAMIC RLS — WHY IT MATTERS ──────────────────────────────────────── Static roles are evaluated once at connection time. Dynamic RLS evaluates on every query. On a 10M row fact table with hundreds of security combinations across Country, Brand, and Company — the difference is felt immediately by end users. This notebook makes generating and maintaining hundreds of static roles as straightforward as running two cells. https%3A%2F%2Fgithub.com%2Fenekoegiguren%2Frls_role_management_tmdl
Cities of Tomorrow: Fabric-Ready Urban Sustainability Playbook
🏙️ Findings Summary: Cities of Tomorrow – Urban Growth & Sustainability This project leveraged Python and standard data science libraries (pandas, numpy, matplotlib, seaborn) to perform an exploratory data analysis (EDA) on an urban planning dataset sourced from Kaggle. The mission was to uncover insights into how cities evolve, sustain, and innovate for the future, ultimately aiming to inform city planners and policymakers. 📊Methodology & Data Quality The analytical workflow involved loading the dataset, conducting extensive Exploratory Data Analysis (EDA), and visualizing key patterns. The data quality was found to be high, with no missing values or duplicates detected across the features. A preliminary statistical review showed the Urban Sustainability Score had a mean of 0.48 and a standard deviation ($\sigma$) of 0.17, indicating moderate variability in current sustainability performance across the studied urban areas. Feature correlation analysis identified several strong relationships between various urban metrics. ✅Key Sustainability Drivers & Insights The analysis yielded several crucial insights regarding the core drivers of urban sustainability: Balancing Density and Green Space: A strong positive correlation was found between the Population Density (when balanced) and Green Space Ratio with higher sustainability scores. This suggests that cities managing density effectively while preserving ample green areas achieve better outcomes. The Power of Transit: Public Transport Accessibility was identified as a top driver of urban sustainability. Efficient, widespread transit systems are crucial for reducing carbon footprints and improving urban quality of life. Environmental Initiatives: Both Renewable Energy Usage and the Waste Recycling Rate showed significant positive impacts on the sustainability score, confirming the importance of direct environmental policy initiatives. Crucial Social Infrastructure: Cities with a higher Education Index and broader Healthcare Coverage consistently demonstrated better sustainability outcomes. This highlights the essential, supportive role of robust social infrastructure in achieving long-term urban sustainability goals. 🎯Conclusion The findings underscore the multifaceted nature of urban sustainability, which is driven by a complex interplay of environmental, infrastructural, and social factors. The results provide city planners with actionable areas for strategic focus, specifically in enhancing public transit, prioritizing green infrastructure, and investing in core social services to build truly sustainable "Cities of Tomorrow." https%3A%2F%2Fgithub.com%2FSkarthikak%2FPython-Docs%2Fblob%2Fbb8118f8483e3d34ad99425118d6ed50bf9e198c%2FNov2025_%255BSaikarthikAK%255D_CitiesOfTomorrow.ipynb
Cities of Tomorrow – Urban Growth & Sustainability
Project Summary: Cities of Tomorrow – Urban Growth & Sustainability Overview As rapid urbanization reshapes the globe, cities face the dual challenge of supporting economic growth while maintaining environmental sustainability and livability. This project explores these dynamics by applying Data Science techniques—data cleaning, exploratory data analysis, and predictive modeling—to understand how modern cities evolve and what factors drive their sustainability. Using the provided Urban Planning Dataset along with optional global indicators, the project uncovers patterns in population density, green space availability, renewable energy usage, and sustainability performance across cities. The goal is to reveal insights that can help policymakers design more resilient, livable, and eco-friendly cities of the future. 1. Data Cleaning & Preparation Steps Performed Loaded the urban planning dataset and inspected shape, types, and missing values. Detected key columns using heuristic matching: Population Density Green Cover Percentage Renewable Energy Usage Urban Sustainability Score (used as the prediction target) Cleaned missing values using: Median imputation for numerical features Mode imputation for categorical features Removed identifier-like columns (city_name, id) from modeling. Prepared a clean, machine-learning-ready DataFrame with consistent types and no missing values. Outcome A fully preprocessed dataset suitable for EDA and modeling. 2. Exploratory Data Analysis (EDA) Correlation Analysis A numeric correlation matrix was generated to understand linear relationships. Key observations typically include: Higher green cover tends to correlate positively with sustainability Greater renewable energy usage also shows a positive relationship with sustainability High population density may correlate negatively with green cover These patterns highlight how environmental and demographic factors shape urban resilience. Scatter Plots Clear visual relationships were plotted: Population Density vs Green Cover Green Cover vs Sustainability Score Renewable Energy Usage vs Sustainability Score CO₂ Emissions vs Sustainability Score (if available) These helped verify whether relationships are linear, clustered, or affected by outliers. 3. Predictive Modeling Model Used Random Forest Regressor (n=200 trees) Chosen because: Handles non-linear relationships Robust to outliers Automatically captures interaction effects Works well when feature importance is required Target Variable urban_sustainability_score Model Training Split into 75% training / 25% testing Fit random forest on selected features (green cover, renewable usage, etc.) Model Performance Metrics R² Score: Measures how much variance the model explains MAE (Mean Absolute Error): Measures average prediction error in sustainability score units This gives a quantitative measure of how well city sustainability can be predicted from environmental and infrastructure-related indicators. Feature Importance Results The model ranks which features contribute most to sustainability. Typical top features include: Green Cover Percentage Renewable Energy Usage Infrastructure Score (if present) Population Density This helps identify what factors drive sustainable cities. 4. Data Storytelling & Insights Key Insights Cities with higher green cover consistently score better on sustainability metrics. Adoption of renewable energy sources is a strong indicator of long-term ecological resilience. Population density impacts green availability and carbon indicators, showing how urban design must balance growth and environmental health. Sustainability is multi-factorial—green space, clean energy, and efficient infrastructure together shape a city’s livability. Narrative Summary Urban sustainability is not the result of one single factor but a system of interconnected elements. The data shows that cities that invest early in green infrastructure and renewable energy build stronger foundations for future livability. Meanwhile, rapidly growing cities must mitigate density-driven stress through smart zoning and urban greening programs. 5. Project Deliverables Notebook: Nov2025_TirthBhanushali_CitiesOfTomorrow.ipynb Contains: Data loading & cleaning EDA visualizations Predictive modeling Markdown storytelling https%3A%2F%2Fgithub.com%2Fbhanushalitirth26-cell%2FCities-of-Tomorrow%2Fblob%2Fmain%2FNov2025_TirthBhanushali_CitiesOfTomorrow.ipynbOptimize Power BI Report with Model Best Practice Analyzer and Vertipaq Analyzer using Semantic Link
Every Power BI Developer pays attention to the underlying performance of their dashboard beyond just churning out reports. This is why adhering to best practices rules should no longer feel like a task, irrespective of your level of experience with Power BI. Semantic Link Labs is here to get the job done. If you are reading about Semantic Link Labs for the first time, you can read my demo introductory article here. This is a Python library developed by Michael Kovalsky that makes this process easier with just a few lines of code in Microsoft Fabric Notebook. Our Use Case: David, A Power BI developer at the AMA enterprise, got several complaints from colleagues that the Sales and Return dashboards take too long to load and sometimes time out. David thought of many reasons why this could have happened, but he wants to start by using the Model Best Practice Analyser Method and Vertipaq Analyzer to investigate the issue. Before this day, David knew he could use an external tool like Tabular Editor for this task. But he recently found out about a new Python library called Semantic Link Lab using Microsoft Fabric Notebook that can help him figure out this issue with the report with just a few lines of code. The sample Sales & Returns Sample v201912 dashboard use for this demo is available for download here You can also download this Microsoft Fabric Notebook used for this Demo here. David’s Task: Perform a Model Best Practice Analyser (BPA) on the Semantic Model of your Power BI report. Perform a Vertipaq Analyzer on your semantic model. Let’s get started: Open your Notebook In Microsoft Fabric, let's investigate what went wrong. Before we get started, we must make sure we have the Semantic Link Labs library installed as well as the other necessary libraries. #Install the Semantic link Labs library in your Fabric notebook %pip install semantic-link-labs After that, we will also be installing the necessary libraries in your notebook. ### Once installed, run this code to import the library into your notebook import sempy_labs as labs from sempy_labs import report as rep from sempy_labs.report import ReportWrapper 1. Performance Model Best Practice Analyzer (BPA) on a Semantic Model report. Model Best Practice Analyzer uses best practice rules to scan your semantic models for bad DAX code, improper formatting and usage of data types, inappropriate relationships, and potentially risky calculation patterns that should be fixed in your report. #Model Best Practice Analyzer import sempy_labs as labs # Enter the name or ID of your semantic model dataset = 'Sales & Returns Sample v201912' # Enter the name or ID of the workspace in which the semantic model # resides workspace = 'Fabricday' labs.run_model_bpa(dataset=dataset, workspace=workspace) When you toggle on the results below, we can see that we have: 16 DAX expression with high rule violation severity, 10 Medium severity, report pages with other details in the Sales & Return Sample v201912 Dashboard. 64 Formatting Medium rule violation severity and 41 on low-level/hints severity. 146 Maintenance low-level/hints rule violation severity and 10 Medium rule violation severity. 46 Performance Medium rule violation severity. & Returns Sample v201912 2. Perform a Vertipaq Analyzer on your Semantic Model. import sempy_labs as labs # Enter the name or ID of your semantic model dataset = 'Sales & Returns Sample v201912' # Enter the name or ID of the workspace in which the semantic model # resides workspace = 'Fabricday' #x = labs.vertipaq_analyzer(dataset=dataset, workspace=workspace) # Setting export='table' will export the results to delta tables in # the lakehouse attached to the notebook x = labs.vertipaq_analyzer(dataset=dataset, workspace=workspace, export='table') # Setting export='zip' will export the results to a .zip file in the lakehouse attached to the notebook. # x = labs.vertipaq_analyzer(dataset=dataset, workspace=workspace, export='zip') The result below shows that the Vertipaq Analyzer Analysis of the Sales & Returns Sample v201912 has been exported to a delta table in my Lakehouse. A snapshot of the Vertipaq Analyzer in my Lakehouse. This shows the Vertipaq Analyzer columns, hierarchies, tables, model, partitions, relationships and tables which can be visualize to understand the semantic model performances. Conclusion If you ever have a slow Power BI Dashboard that takes time to load, and you want to understand the reason for the slow performance. You can download the Microsoft Fabric Notebook used for this optimisation and enhancement Power BI report with Semantic Link Labs here. References Semantic Link Labs Git Hub Repository by Michael Kovalsky: https://github.com/microsoft/semantic-link-labs Feel free to connect with me via my socials below if you want to discuss further. LinkedIn: Musili Adebayo Twitter: Musili_Adebayo https%3A%2F%2Fgithub.com%2FMusili-Adebayo%2F-Musili-Adebayo-fabric_day_project_dashboard%2Fblob%2Fmain%2Fpowerbi_report_notebook.ipynbDelta table statistics, maintenance and properties configuration
This notebook serves as an example of the possible implementation and subsequent automation of audit and maintenance operations on the tables in our lakehouse. It extends functionality and allows you to streamline maintenance operations available from the lakehouse explorer and schedule them as needed, as well as check the evolution of the tables residing in it and the configuration of their properties. In this way, we achieve a more efficient, robust, and complete ecosystem. You will need to: Download and import the notebook to a Fabric capacity backed workspace Attach your own lakehouse to the notebook in order to test and try the samples provided Configure the specified variables as you need them before running, specially the properties setting section The notebook is divided into different cells for each section, including comments and type hinting. The code does not modify your own data. However it will delete unused Delta table files if the VACUUM operation is executed, read everything first before executing anything. Feel free to adapt the samples for your own needs, hope you find it useful. Credits & acknowledgments Sandeep Pawar for its base analysis for Delta table stats Miles Cole for its brilliant article about table compaction Notebook is available here. https%3A%2F%2Fgithub.com%2Fl2aFa%2Fpbi-notebook-gallery%2Fblob%2Fmain%2Fdelta_one_for_all.ipynbComposite Model: Direct Lake + Import
Composite Model Deployment: Direct Lake + Import This notebook deploys a Composite Semantic Model based on Direct Lake and Import storage modes. Import tables can come from any supported data source and relationships between Direct Lake on OneLake and Import tables are regular relationships. Small dimension or lookup tables already in Direct Lake storage mode can instead use import storage mode, giving you the option to extend the table with calculated columns and structuring the table with hierarchies for use in Power BI reports and Excel pivot tables. The feature was announced in Power BI May 2025 release. Requirements: Direct Lake on One Lake semantic model: This model is based on the AzureStorage.DataLake protocol. The model must be created in the Power BI Desktop, otherwise it will use the Sql.Database protocol. Import semantic model: This model will be used as a reference for replacement and deployment. Only the desired tables to be in Import mode are required. XMLA read/write: This will enable the deployment through XMLA endpoint. The permission is enabled at the Capacity level and permissions are managed at the Admin Portal. Libraries The only mandatory library is Semantic Link Labs, a Python library designed for use in Microsoft Fabric notebooks. This library extends the capabilities of Semantic Link offering additional functionalities to seamlessly integrate and work alongside it. In this notebook, Semantic Link Labs is used to capture the reference models definition files and to create/update the Composite Model. Remaining libraries are optional, and are used to export the Composite Model definition file to a Lakehouse. # Install Semantic Link Labs %pip install semantic-link-labs # Required Library import sempy_labs as labs # Optional Libraries import json from notebookutils import fs Parameters All the parameters expect the name or the id of the object, except the import_tables parameter. This one requires a list of names of the tables that will be in Import mode. # Required parameters workspace = 'ws_demo' # Reference workspace dataset_import = 'sm_import' # Source: Import model dataset_directlake ='sm_directlake' # Source: Direct Lake model dataset_composite = 'sm_composite' # Sink: Composite model import_tables = ['table1', 'table2', 'table3'] # List of tables that will be set to Import mode # Optional parameters storage = 'lh_demo' # Lakehouse to ouput the Composite Model definition file Model Definition Process workflow: Capture the original model definition (BIM) files Remove from the Direct Lake file the tables that will be kept in Import mode Add in the Direct Lake file the Import tables based on the Import file definition Deploy the model Existing relationships on the Direct Lake model are persisted. It will fail if the column names are different (between Direct Lake and Import models). # Get Model Definition file (BIM) from source models bim_import = labs.get_semantic_model_bim( dataset = dataset_import, workspace = workspace ) bim_directlake = labs.get_semantic_model_bim( dataset = dataset_directlake, workspace = workspace ) # Delete the tables that will change storage mode from the Direct Lake file bim_directlake['model']['tables'] = [ table for table in bim_directlake['model']['tables'] if table['name'] not in import_tables ] # Include the tables with changed storage mode to Import in the Direct Lake file for table in bim_import['model']['tables']: if table['name'] in import_tables: bim_directlake['model']['tables'].append(table) Deployment The deployment is done via XMLA endpoint. It will create or update the model if it already exists. After the deployment, a new connection will be displayed in the Gateway and Cloud Connections section of the Semantic Model Properties. A rebind of the Import tables connection is required, since it cannot rely on Single Sign On as Direct Lake does (this is only required after the first deployment). If a connection already exists for the Import table, just check it on the Map to. Otherwise, create a new connection and then Map to it. After the rebind the semantic model can be refreshed. # Create/Update the Composite Model try: labs.create_semantic_model_from_bim( dataset = dataset_composite, bim_file = bim_directlake, workspace = workspace ) except: labs.update_semantic_model_from_bim( dataset = dataset_composite, bim_file = bim_directlake, workspace = workspace ) Optional Steps The steps below are optional. The first one validates if the Import connection has been binded (this step must be done manually after deployment). The last one outptus the Composite Model definition file to a Lakehouse. Analyze Connection Bindings # Retrieves the list of connections visible in the Fabric environment connections = labs.list_connections() # Retrieves the list of connection dependencies in the Semantic Model model_connections = labs.list_item_connections( item_name = dataset_composite, item_type = 'SemanticModel', workspace = workspace ) # Connections that do not rely on AzureDataLakeStorage (Direct Lake on Onelake) connection_ids = model_connections[ model_connections['Connection Type'] != 'AzureDataLakeStorage' ]['Connection Id'] for con in connection_ids: connection_path = model_connections[ model_connections['Connection Id'] == con ]['Connection Path'].iloc[0] if con: connection_name = connections[ connections['Connection Id'] == con ]['Connection Name'].iloc[0] print(f'The path {connection_path} has been mapped to the {connection_name} connection.') else: print(f'No connection found for path {connection_path}. Bind the connection before refreshing the semantic model.') Output Model Definition File # Save model definition to Lakehouse path = f'abfss://{workspace}@onelake.dfs.fabric.microsoft.com/{storage}.Lakehouse/Files/{dataset_composite}.json' file = json.dumps( bim_directlake, indent = 2 ) fs.put( path, file, overwrite = True ) https%3A%2F%2Fgithub.com%2Fdiego-dsanalytics%2Ffabric-notebooks%2Fblob%2Fmain%2FFiles%2Fcomposite_model.ipynb
