Fabric IQ: Do Ontology entity synonyms work with Data Agent?
Hi, everyone! Short intro Currently I'm using the "super-duper-mega-nano-ultra" product - Microsoft Fabric to build natural language processing flow on-top of Microsoft Fabric Warehouse data. As for now It's rather a POC than production solution. I found a lot of the official Microsoft' documentation related to my task, but I have a little problem... The solution architecture (high-level) I made some investigations and as the result is the following architecture, which I want to implement (picture below): The idea is the next: the Microsoft Fabric Warehouse schema is connected to Microsoft Fabric Lakehouse, using shortcut; the Microsoft Fabric Ontology consumes the Microsoft Fabric Lakehouse as a data source for data binding; the Microsoft Fabric Data Agent uses a Microsoft Fabric Ontology (enriched with business context) to process natural language questions. The Microsoft Fabric Warehouse contains the following (dummy) objects (picture below): Tenant settings Microsoft documentation says, that specific tenant configurations should be applied to use Microsoft Fabric Ontology with Microsoft Fabric Data Agent (https://learn.microsoft.com/en-us/fabric/data-science/data-agent-tenant-settings) - everything is configured properly. Ontology configuration My Microsoft Fabric Ontology is configured as below (the configuration is influenced by Microsoft Fabric Ontology tutorial, which can be found here - https://learn.microsoft.com/en-us/fabric/iq/ontology/overview). Main view - two entities with a single relationship: ETLEntity entity configuration - the entity has the description, one synonym, metadata: ETLEntityRun entity configuration - the entity has the description, one synonym, metadata: As for now the Microsoft documentation says, that Microsoft Fabric Ontology descriptions, synonyms, metadata help Microsoft Fabric Data Agent to better understand the context (https://learn.microsoft.com/en-us/fabric/iq/ontology/how-to-add-semantic-enrichment). The problem My Microsoft Fabric Data Agent is connected to my Microsoft Fabric Ontology, which is described above, but the agent can't answer the simple questions about entities and the questions examples are provided below (the Microsoft Fabric Ontology Graph model was refreshed successfully before questions were asked): ETLEntity successful question without synonym usage: ETLEntityRun successful question without synonym usage: ETLEntity failed question with synonym usage: ETLEntityRun failed question with synonym usage: Looks like Microsoft Fabric Data Agent can't figure out, which entities are unicorn/wizard, even if they have appropriate synonyms. It's not my first iteration - I tried a lot, but result still the same every time. I feel like I missed something obvious in my configuration, but what... What are your thoughts? P.S.: the provided configuration is simple; objects and their metadata has no business context - It's just a sample, which I built to test some scenario; I think It's enough to check such use-case.
Paginated Report Matrix with Dynamic Columns Causing PDF Export Issues
Hi everyone, I'm working on a Paginated Report (RDL) that contains a matrix. The matrix columns are grouped by Product Type, so the number of columns grows dynamically depending on the data returned. Scenario Rows contain summary data. Columns are grouped by Product Type. The number of Product Types varies and can become quite large. As more Product Types are returned, the matrix expands horizontally. Issue 1: Managing Page Width in PDF Export When exporting the paginated report to PDF, the dynamically generated columns can exceed the page width. Questions What is the recommended approach for handling matrices with dynamically growing columns when exporting to PDF? Is there a way to automatically scale the matrix to fit the PDF page width? Are there any best practices for: Switching between portrait and landscape layouts? Splitting columns across multiple pages? Using page breaks for column groups? Handling a variable number of dynamic columns? Issue 2: Extra Blank Page Appearing in PDF Export In addition to the width issue, each report page is followed by an extra blank page when exported to PDF. Current Settings Report has a fixed page size. Margins appear to be within normal limits. Matrix width changes dynamically based on Product Type values. Questions Can dynamic matrix width cause additional blank pages during PDF export? What are the common causes of a blank page being generated after every report page? What should I check to identify whether the issue is caused by: Report body width Matrix width Rectangles/containers Page size and margins Hidden objects extending beyond the printable area Expected Outcome I would like the report to: Export to PDF without creating extra blank pages. Handle a varying number of Product Type columns gracefully. Maintain readability even when the matrix expands horizontally. Any guidance, best practices, or examples would be greatly appreciated. Thanks!Spark Runtime 2.0 - Practical Overview and Stress Testing (with a built framework)
Link to the notebooks to reproduce: https://github.com/iurii-iurchenko-1/source-materials/tree/main/2026Q3_spark_runtime_2_0 Outline 1. What is Runtime 2.0 and why does it matter? 2. The five features explained 2.1 Vectorized CSV parsing in the Native Execution Engine 2.2 ANSI SQL mode inside the native engine 2.3 Incremental liquid clustering 2.4 Arrow-native Python UDFs 2.5 The VARIANT type in Delta tables 3. What are the main improvements as shown in stress testing 4. How did I test that? – Building the framework – The three notebooks – The tables and what each one measures 5. Main conclusions 5.1 If you are an architect 5.2 If you are an engineer 6. Resources used 7. Notebook results 7.1 Runtime 1.3 metrics 7.2 Runtime 2.0 metrics 8. How to reproduce 9. Links to documentation What is Runtime 2.0 and why does it matter? What is Runtime in Spark notebooks? It is an environment that includes the version of Spark, the default version of Delta tables, and other components that provide the engine to run Spark Notebooks in Fabric. The previous version of the runtime was Runtime 1.3, which included Spark 3.5. The latest Runtime 2.0 includes Spark 4.x and an upgraded version of Delta Tables. I chose 5 particular features introduced in Runtime 2.0 and built a framework to measure the difference in performance between versions 1.3 and 2.0. These features are: Vectorized CSV parsing in the Native Execution Engine ANSI SQL mode inside the native engine Incremental liquid clustering Arrow-native Python UDFs The VARIANT type in the delta tables Let me explain these features in depth. 1 Vectorized CSV parsing in the Native Execution Engine This feature means the engine-level optimization of reading CSV files. It may help if your organization has heavy jobs to ingest CSV files at scale. 2 ANSI SQL mode inside the native engine This feature is powerful, and not a lot of people know about it. If you run Spark queries and a formula fails to calculate some values, Spark quietly returns NULL. If ANSI is enabled, the failing will be loud. It is needed in critical pipelines. Now that feature is supported by the native engine. 3 Incremental liquid clustering This new feature influences the behavior of the OPTIMIZE command on top of a Delta table. In the old version, the behavior was to reprocess all the files. In the new version, only new unclustered files will be reprocessed. Isn't that good? 4 Arrow-native Python UDFs This feature introduces the new optimized algorithm to process Pandas UDF functions. When are they needed? During a migration of the Python/Pandas notebooks. So, instead of rewriting Pandas functions and introducing new bugs, implementing this mechanism may help. IN the new engine, it is optimized to avoid converting to Pandas internally, which saves time and resources. Also, if you know PyArrow functions to proceed with strings, they may also be utilized instead of plain Python. 5 The VARIANT type in the delta tables Variant type promises performance optimization on top of JSON-like data. So, it looks like you decoded the data source as JSON, wrote it once (as a VARIANT column), and it may be processed faster later, when you need to extract some information from it. What are the Main Improvements as Shown in Stress Testing Final Results CSV Scan: I observed ~50% performance improvement on reading CSV files in Runtime 2.0 (9.2 sec vs 6.5 sec) ANSI SQL mode - significant improvement in Runtime 2.0 (see the code and logs at the end of the notebook). 5.3 sec ANSI off →12 sec ANSI ON on Runtime 1.3 vs 3.6→3.7 on Runtime 2.0. Incremental liquid clustering - the new algorithm introduces incremental clustering. So, now, we may have two strategies: incremental clustering or full rebuilding of Z-cubes. So, the introduces incremental clustering gives a significant boost in performance, but still loses the battle compared with full optimization/rebuilding of Z-cubes. Arrow-native Python UDFs give almost a 3x boost on the new runtime due to avoiding converting it to Pandas and back Variant type. Here, I observed that Variant may not give a significant boost in reading performance. Sometimes, it may have even slower performance. But it uses disk space better. How did I test that? To test that, I vibe-coded a framework in Claude to cover all the features mentioned above. It was a back-and-forth process. I carefully selected 5 features recently introduced to Runtime 2.0, built tech requirements for the system to test it, generated the code for three notebooks, ran it, got the first issues and output, improved it, and repeated it a few times till the process was smooth. I tested it in my personal MS Fabric environment with F2 capacity. The solution includes three notebooks: NB_00_GENERATE_DATA NB_10_BENCH_RUNTIME_1_3 NB_20_BENCH_RUNTIME_2_0 The tables are: fact_sales — 20M-row Delta fact table. Feeds the ANSI × NEE arithmetic query and all four UDF variants. Files/bench/raw_csv — the same 20M rows written as uncompressed CSV, 2.1 GB. The only asset that tests vectorized CSV parsing in the native engine. logs_json — 3M nested JSON documents held in a STRING column. The baseline for the VARIANT comparison, and the source for the VARIANT migration. events_lc_r13 — liquid-clustered Delta table, left with unoptimized new data on top. Measures the Delta 3.2 full-rewrite OPTIMIZE on Runtime 1.3. events_lc_r20_inc — identical twin of the above. Measures Delta 4.2's incremental OPTIMIZE on Runtime 2.0. events_lc_r20_full — identical triplet. Measures OPTIMIZE ... FULL on Runtime 2.0, giving a same-session comparison against the incremental arm. logs_variant — not from the generator; created by the Runtime 2.0 notebook itself via parse_json. Its creation time isthe VARIANT migration cost, and it is what the colon-operator query reads. After that, I ran the notebooks NB_10_BENCH_RUNTIME_1_3 and NB_20_BENCH_RUNTIME_2_0 to get the main metrics. You may find these metrics below. They show how new features in Runtime 2.0 differ in performance for different use cases. Main Conclusions If you are an architect The main point with Runtime 2.0 is that it becomes GA. That means the older versions will be deprecated step-by-step, and it is time to plan the modernizational upgrade of the current infrastructure to use Runtime 2.0. If you are an engineer Invest your time in learning about NEE (Native Execution Engine) features, and test how they impact your real data flow. They are efficient, especially on complex logic query Arrow native Python UDFs may be 3x faster compared to ordinary UDFs if you use only the library's functionality and don't use Python variables there. This library is rich. So, it is possible, especially for text operations. If you didn't know what ANSI SQL mode is, now you know. If you want to Spark fails loudly if some formula didn't take place, that's it. Now, it is much more supported by the engine. Resources Used I tested all that on a personal tenant with Fabric Capacity F2. Notebooks Results Runtime 1.3 Metrics Runtime 2.0 Metrics How to Reproduce? Import the notebooks mentioned at the beginning of the article to your Fabric tenant Create a Lakehouse "LH_RUNTIME2_0_TEST Run NB_00_GENERATE_DATA Sequentially, run NB_10 & NB_20, creating before runtimes 1.3 and 2.0 for them. Links to Documentation https://community.fabric.microsoft.com/blog/fbc_fabricupdatesblogs/fabric-august-2026-feature-summary/5325824 https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Fabric-July-2026-Feature-Summary/ba-p/5325823 https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-2-0 https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview?tabs=sparksql
Plan module (used to be powertable), single select columns becomes blank when sorted.
We have used powertable before, and have migrated into the plan module to have our master data, including our Employee table When sorting by ReportingTo the EmployeeName column becomes blank. I have added the configuration of the EmployeeName but its similar to Initials that is not affected Does anyone have any idea of what is going on? Had it affected all single select column the error had been consistant, but EmployeeName and Initials are both using the same jira.user table in its look up.
Fabric Trial Activation Screen Appears but Trial Is Not Available
Hello Microsoft Fabric Community, I am a BCA student from India and I am currently learning Microsoft Fabric for educational purposes. I need access to Fabric features such as Lakehouse, Notebooks, and Data Engineering for hands-on practice. I am facing a confusing issue while trying to activate the Microsoft Fabric trial. My Power BI individual trial is already active and shows 59 days remaining. However, when I click "Start Fabric trial", I get the "Activate your 60-day free Fabric trial capacity" screen. The activation screen shows: - A 60-day free Fabric trial capacity - Trial capacity region: Default – Central India - An "Activate" button However, after proceeding, I receive this message: "Your Power BI trial is active Your free Power BI trial gives you 60-day access to Power BI paid features. A Fabric trial isn't available for your account. See the Microsoft documentation for eligibility requirements." This is confusing because the previous screen allows me to activate a 60-day Fabric trial capacity, but the next message says that a Fabric trial is not available for my account. I have also already confirmed that: - Users can create Fabric items is enabled. - Users can try Microsoft Fabric paid features is enabled. - My Power BI trial is active. - I am using a newly created tenant. - I am trying to use Fabric for learning and educational purposes. I do not have a college-issued student email/Student ID that I can use for verification. Could someone please clarify: 1. Why is the Fabric trial activation screen appearing if my account is not eligible? 2. Is my newly created tenant too new for a Fabric trial? 3. Is there any legitimate way to enable a Fabric trial capacity for this account? 4. If I am not eligible, what is the recommended way for a student to practice Fabric features such as Lakehouse and Notebooks? I would really appreciate any guidance, as I need Fabric hands-on practice for my learning. Thank you.
Datagateway stuck at "Failed to sign in. Please Verify your credentials and try again."
Im droping this here as i didnt see anywhere where this had been addressed. When trying to signin to the data gateway the app freezes and says "Failed to sign in. Please Verify your credentials and try again." The credentials are good and everything was fine prior to update. I got this after updateing the datagateway. The app freezes up and you have to force close. The work around is that you click signin options prior to entering email select these options and it will redirect you to your browser to login and wont lock up hope this someone
