data science
91 TopicsAdd AI Smart Data Validation in Copy Job
I suggest adding an AI Smart Data Validation feature to Data Factory Copy Job. It should automatically detect missing values, duplicate records, data type errors, and mapping issues before copying. This will reduce errors, save time, improve data quality, and make Copy Job easier for all Microsoft Fabric users.20Views0likes0CommentsAI-Powered Data Insights in Power BI
I would like to suggest adding AI-powered data insights directly within Power BI to help users uncover trends, anomalies, and key takeaways faster. Key benefits: • Auto-generate insights and summaries using AI. • Detect anomalies and outliers in visuals. • Natural language explanations for charts and reports. • Save time and improve data-driven decision making. This feature will make Power BI even more powerful and user-friendly for all users.14Views0likes0CommentsAI-Powered ICU Mortality Prediction using MIMIC-IV
This project proposes an AI-powered ICU mortality prediction system using the MIMIC-IV dataset. The system analyzes patient demographics, vital signs, laboratory results, and ICU admission data to identify high-risk patients. Machine learning models such as XGBoost and Random Forest will be used to predict mortality risk. Explainable AI techniques will help clinicians understand the factors influencing predictions. An interactive Power BI dashboard will provide insights into patient risk levels, mortality trends, ICU stay duration, and clinical indicators. The goal is to support early intervention, improve resource allocation, and enhance patient outcomes through data-driven decision making. Technologies: Python, Pandas, Scikit-learn, XGBoost, Power BI, MIMIC-IV Dataset. Expected Outcome: Accurate mortality prediction, interpretable risk analysis, and a healthcare analytics dashboard for ICU patient monitoring.25Views0likes0CommentsDax functions that mimic List.Generate
Functions that could essentially iterate and expression to a suggested limit simliar to list.generate in power query. Iterate(SeedTable, NextState) that would mimic hard coding something like this State0 = SeedTable State1 = NextStep(State0) State2 = NextStep(State1) State3 = NextStep(State2) ... Could use this in dynamic scheduling, BOM explosions, heirarchy traversing, Fibinocci sequence etc. All can be done but require explicit calls to variables. Not true recursion but iterators. Please.41Views1like0CommentsSupport for Fabric Data Agents in Deployment Pipelines
Currently, Microsoft Fabric deployment pipelines do not support Fabric Data Agents, which limits our ability to fully automate deployments of the Fabric Data Agents We have also reached out to Microsoft support regarding this issue. Currently its not in the plan, It would be helpful is this feature is enabled.170Views1like1CommentAI-Powered Resume Analytics Starter Kit for Fabric Lakehouse
As a participant in #DataDays2026 SQL + AI Promptathon, I built ResumeCraft - an AI resume analyzer. Problem: Every student/company building resume/job analytics tools has to start from scratch in Fabric. Data cleaning, table design, SQL queries for skills/location trends - sab khud likhna padta hai. Solution Requested: Microsoft should provide an official "Resume Analytics Starter Kit" template in Fabric with: 1. Pre-built Lakehouse schema for Resume JSON/CSV data 2. 5-10 Sample Copilot prompts for SQL: "Find top 10 skills", "Location wise applications" 3. Ready Power BI dashboard template for HR insights 4. Python notebook for skill extraction using AI Impact: This will help 1000s of students, recruiters and beginners learn Fabric + SQL + AI in 1 hour instead of 1 week. Perfect for Data Days challenges. Please upvote if you also faced this issue!78Views0likes0CommentsIQR Clustering Analysis in Power BI & Excel Example
Below is an explanation of my IQR Clustering Analysis in Power BI in Excel hotspot model analysis applied to over 190,000 records of 311 Toronto Customer Initiated Service Request Data from January 1 to April 30, 2026. Model Input: The input data for this model analysis was an Excel worksheet (or it could be queried using PowerQuery M language from or located in some other place like a database) with 26 columns and 533 rows of data. Column A of the Excel worksheet lists each of the 533 reasons (Service Request Types) why a resident of Toronto has contacted 311 Toronto to complain or make a request over the four month period studied. Columns B-Z column headings are the names of each one of the 25 City of Toronto wards. Each cell contained a raw count of the number of 311 Toronto Customer Initiated Service Requests made by residents of a City of Toronto ward about a specific Service Request Type. Model Output: The output data for this model analysis is shown in an Excel worksheet that you can email me to obtain: it still has 26 columns and 533 rows of data (buy I do also have output without the raw counts overwritten). Instead of raw counts for the cell values, one of the following 5 text labels are used: 'High Outlier', 'High', 'Normal', 'Low', 'Low Outlier'. The 'High Outlier' values in cells indicate that, based on distribution of raw counts of service requests (across Service Request Type) received by 311 Toronto within a given ward, complaints for a specific Service Request Type in a ward have been found to be significantly, statistically high based on the IQR Rule theory of the box & whisker plot. Model Conclusion: Based on the attached Excel output sample, my analysis revealed that not all City of Toronto Wards are the same in terms of the time, energy and resources that the City must spend fixing the various service requests logged by residents. Animal Noise complaints are significant issue in only 3 of the 25 City of Toronto Wards for the first four months of 2026: Parkdale-High Park (04), Toronto Centre (13) and Toronto-Danforth (14). It is for this reason, that their cell values are shaded yellow and have the words 'High Outlier' written in red. Residential / Yard Waste / Missed proved a significant issue for only 2 Wards and their cell values are shaded yellow with the words 'High Outlier' written in red. The Hotspot Analysis Spreadsheet Template used to automate this analysis works on over 8,000 columns and as many rows of data as your version of Microsoft Excel accepts. The process will run as fast as your computer chip can handle. And, this IQR analysis process is an easier to apply and interpret segmentation process than either K-means Clustering or Hierarchical Clustering.63Views0likes0CommentsAllow us to rename fabric data agents published to m365
If we use deployment pipelines to promote fabric data agents between dev, test/UAT, and prod fabric workspaces, we need to keep the name of the fabric agent the same in each workspace. If we want to publish the data agent in both UAT and prod to m365, they're would be two agents with the same name. It would be helpful to be able to rename the published agent name in M365 to <agent>UAT when published so users can easily distinguish between the UAT and prod agents.169Views0likes0CommentsGraph Based RAG for strong reasoning and accuracy across layer
Current RAG implementations in Microsoft Fabric (via AI Skills, Azure AI Search integration, and OneLake-backed vector stores) rely predominantly on vector similarity search over chunked documents. While this works well for surface-level semantic retrieval, it has significant limitations for enterprise scenarios that require multi-hop reasoning across structured and unstructured data: Loss of relational context: When documents are chunked and embedded, the relationships between entities (customers, products, transactions, regulations, contracts) are lost. Vector search retrieves semantically similar chunks but cannot traverse "Customer A → owns → Account B → linked to → Transaction C → flagged by → Regulation D." What Would Help A first-class Graph-Based RAG capability natively integrated into Fabric, including: OneLake Graph artifact — a new item type (alongside Lakehouse, Warehouse, Eventhouse) that stores property graphs in an open format (e.g., extending Delta with graph metadata, or adopting an open standard like Apache AGE or GraphAr), with full Purview lineage and OneLake shortcut support. Automatic graph construction pipelines — Data Factory / Fabric Notebook templates that extract entities and relationships from Silver-layer tables and unstructured documents using Azure OpenAI / Phi models, with human-in-the-loop validation. Why It Matters Accuracy where it counts: Enterprise and regulated customers Unlocks Fabric's strategic differentiator: Fabric already unifies data estates in OneLake. Graph-RAG is the reasoning layer that turns that unified estate into a reasoning estate Reduces architectural sprawl: Customers currently stitch together Azure AI Search + Cosmos DB Gremlin + custom orchestration110Views0likes0Comments