Fabric Lakehouse SQL Analytics Endpoint - SQL Projects extract.
When extracting a SQL database project from a Lakehouse SQL analytics endpoint using Visual Studio, I only receive the CREATE VIEW definitions. The CREATE TABLE definitions for the underlying Delta tables are missing, even though the tables are visible and queryable through the endpoint. The behavior is inconsistent because table definitions are generated for some SQL analytics endpoints but not for others. Has anyone encountered this issue or found a reason for the difference?
How do I just write a CSV file to a lakehouse with a notebook?
Greetings, all. I'm exploring Fabric notebooks and have a question. I am trying to write a simple CSV file to a lakehouse using the following code: spark.createDataFrame(dataframe).write.mode("overwrite").csv(LakehousePath + FilePath + FileName_NoType + '_altered') When I write this, however, I get a folder with files in it rather than a single .csv file: How do I just save a normal csv file with the name I've chosen?
Lakehouse Lookup Activity No Longer Shows T-SQL Query Option
Hi, I have a Microsoft Fabric pipeline that uses a Lookup activity to execute a SQL query against a Warehouse and return a single value. When I use a Lakehouse connection in the same Lookup activity, I can only select a table. The T-SQL Query (Preview) option is currently disabled. I was able to use this option approximately a month ago. Is this functionality no longer supported for Lakehouse connections? What is the recommended approach for executing a T-SQL query against a Lakehouse from a Fabric pipeline and returning a single value, such as: SELECT MAX(rec_mod_dtz) AS last_update FROM fact_survey_detail;
Advanced NL2SQL and advanced DAX generation in Fabric data agents (Generally Available)
Data agents in Microsoft Fabric let users ask questions about data in plain language. As a creator, you connect an agent to supported sources—including SQL databases, lakehouses, Power BI semantic models, and eventhouses—add the business context it needs, and share it with users across Fabric, Teams, Microsoft 365 Copilot, Copilot Studio, and Microsoft Foundry. The data agent can then respond like a domain expert that understands what your tables mean, how your metrics are defined, and which source can answer a question. The usefulness of a data agent ultimately depends on a simple question: can it translate what a user asks into the right query? That is harder than it sounds. Enterprise schemas are rarely intuitive. Several tables, columns, measures, or join paths may appear to answer the same question, so a query can be technically valid but still return the wrong result. Advanced NL2SQL and advanced DAX generation help address this problem by allowing the agent to explore the schema and data, validate assumptions, and refine its approach before generating a query. Both capabilities are available today, and we'll begin rolling the core improvements out to the standard runtime on November 2, with the rollout expected to complete by November 10. In this post, we'll look at how they work, what our internal benchmarks show, the new configurations available for shaping how the agent behaves, and what to consider as you re-test and evaluate your data agents. Why conversational analytics gets harder with enterprise data Turning a plain-language question into a correct query is a complicated problem, both for SQL and DAX, requiring the agent to resolve several challenges at once: Schemas are large. Enterprise sources can contain hundreds of tables, columns, and measures, and the names alone may not explain when or how they should be used. Business context exists outside the data. Definitions such as “revenue” or “active customer” often live in documentation, previous queries, or analysts’ knowledge rather than in the source schema. Filter values must match the way data is stored. Users describe concepts naturally, but those concepts must map to the values in the data. For example, a user may ask about “California” when the relevant column stores the value “CA.” Questions are ambiguous. More than one column, calculation, or join may appear to satisfy a request. Individually, each of these can be managed with enough hand-written guidance. The difficulty is that they compound: a single-pass generator has to resolve all of them at once — your schema, your definitions, your filter values — committing to an interpretation before it has any evidence that interpretation is right. When it guesses wrong, the query still runs, and the answer looks no different from a correct one. Advanced NL2SQL and advanced DAX generation are built around that problem. Rather than generating a single pass, the agent gathers evidence first. Advanced NL2SQL: the agent explores before it writes The shift is architectural. Instead of one-shot generation, the agent has sub-tools to explore schema, test joins, and look up real filter values before it writes SQL. Under the hood, you can inspect each step the agent took, and the assumptions and reasoning behind it. It operates in three modes, each backed by its own sub-tools: Mode What it does Example query following Finds a relevant example query and answers by making approved modifications, such as literal filter or parameter substitutions. Clarifying questions Determines it can’t answer reliably without more information and asks targeted clarifying questions before generating SQL. Data exploration When no example directly applies, learns from existing example queries to infer business logic, explores the data to validate assumptions, and iteratively refines its approach. What you get as a creator: Advanced NL2SQL explores the SQL source schema, tests joins, and looks up real filter values before generating SQL, helping it choose the appropriate tables, columns, relationships, and filters. Schema descriptions and topics that teach the agent what your data means, so you get better answers with fewer instructions. Better example-following and filter-value substitution for novel questions. Handles large schemas far better and runs faster end to end, with more reasoning detail for debugging. New configurations: schema descriptions and topics Two new configurations give the agent more of the context it needs. Both are available for SQL data sources on the preview runtime. Schema descriptions Schema descriptions attach business context to the objects in your data source — what a table holds, what a column represents, and how each should be used. The agent uses them to interpret large or ambiguous schemas and generate a more accurate query. They matter most where object names don’t carry meaning on their own, or where several tables and columns look interchangeable. Figure: Schema descriptions add business context to schema objects, helping the data agent interpret your data. Topics Topics help you give the agent detailed guidance without sending every instruction with every question. The agent searches your topic content and uses only the sections relevant to the current request. Use data source instructions for rules that must always apply, and use topics for deeper guidance that applies only to a specific subject. You can write topics in Markdown, import multiple files, and review the run steps to see which topics the agent selected. Figure: Topics organize subject-specific instructions in Markdown to guide the data agent’s query generation. Internal benchmark testing: Advanced NL2SQL on Preview Runtime vs NL2SQL on Standard Runtime Better reasoning matters only if it produces better answers without making users wait substantially longer. In an internal benchmark set evaluating answer quality and end-to-end latency, the preview runtime delivered the stronger outcome, answering more questions correctly than the standard runtime while reducing both median and tail latency. Smart-answer accuracy increased from 78.8% to 90.9%. Latency improved at the same time: P50: 28.4 seconds to 26.0 seconds P95: 46.8 seconds to 32.9 seconds Figure: Smart answer accuracy and end-to-end latency, standard runtime versus the Advanced NL2SQL preview, on an internal benchmark set. Accuracy rises 12.1 points to 90.9%, while P50 latency falls 8.3% and P95 falls 29.8%. *Performance results are based on Microsoft internal testing conducted in 2026 using an internal evaluation set comparing standard NL2SQL to the advanced NL2SQL preview in Microsoft Fabric. Actual results may vary depending on workload, data model, and configuration. The P95 result shows the largest latency improvement in this benchmark, with latency decreasing from 46.8 seconds to 32.9 seconds. This improvement is most relevant to the slower, more complex queries represented at the tail of the distribution. Advanced DAX generation: agentic reasoning over semantic models The agent calls tools, inspects results, and refines its approach to answer harder questions. Instance value indexing searches real column values, so filters match your data. And you get consistent results across Data Agent, Power BI, and M365 Copilot. For users, this means the agent is more likely to resolve filter terms to the values stored in the semantic model and return consistent answers across supported Microsoft experiences. Internal benchmark testing: Advanced DAX generation improves accuracy while cutting latency Our internal Advanced NL2DAX evaluation showed improvements in both answer accuracy and latency. Smart-answer accuracy increased from 89.15% to 91.86%. At the same time: P50 latency: 39.1 seconds to 20.4 seconds P95 latency: 97.5 seconds to 34.7 seconds This represents an approximately 48% reduction in P50 latency and a 64% reduction in P95 latency, along with a 2.7 percentage-point increase in smart-answer accuracy. *Performance results are based on Microsoft internal testing conducted in 2026 using an internal DAX generation evaluation set comparing the baseline to the advanced DAX generation preview in Microsoft Fabric. Actual results may vary depending on workload, semantic model, and configuration. Rolling out the runtime improvements We'll begin rolling the core Advanced NL2SQL and advanced DAX generation improvements out to the standard runtime on November 14, with the rollout expected to complete by November 10. As part of this rollout, the standard runtime will also move to GPT-5.6 Terra. As the rollout reaches your environment, data agents using the standard runtime will receive these improvements and the model upgrade by default, with no opt-in required. What is moving to the standard runtime: Advanced NL2SQL: improved example-query following, filter-value substitution, and data exploration. Advanced DAX generation: the agentic tool-calling loop and instance value indexing. Model upgrade: The standard runtime will move to GPT-5.6 Terra for orchestration and supported data agent capabilities. What stays in preview: Schema descriptions and topics for SQL sources. We believe these changes deliver significantly better results, based on improvements observed in our internal evaluations. However, results can vary by data source, model, instructions, examples, and workload. Creators can re-test and evaluate their agents on the standard runtime between November 14 and November 20 as the rollout reaches their environment, or try the changes now on the preview runtime. Moving from one-shot generation to an agent that explores data can change how existing instructions and examples are interpreted. We recommend re-testing: Run your existing question set against the standard runtime as the rollout reaches your environment between November 14 and November 20, or switch to the preview runtime to try the changes now. Rerun your evaluations with the Fabric data agent SDK and a ground-truth set. Compare the new results with your previous baseline, and use the visible reasoning steps to identify where the agent’s assumptions differ. Tune your example queries and instructions based on what the evaluation shows. The agent now explores your data and adapts examples through substitution, so guidance written to work around the previous behavior may no longer be needed. Re-testing and evaluating your agents will help you confirm the expected quality improvements for your own data and scenarios. Next steps If you build data agents in Microsoft Fabric, re-test your agents on the standard runtime as the rollout reaches your environment between November 14 and November 20, or try the changes now on the preview runtime. For more details: Fabric data agent runtime: Learn what the standard and preview runtimes each contain, how to switch between them, and what’s in the preview runtime today. SQL sources in Fabric data agent: Learn how NL2SQL and Advanced NL2SQL generate queries over Lakehouse, Data Warehouse, SQL Database and Mirrored Database sources, and which configurations each one can use. Semantic model best practices for data agent: Semantic model best practices for data agent: Learn about advanced DAX generation and how to prepare a semantic model so the agent can answer questions accurately.
Where Data Meets AI Extending Azure Databricks Investments with Microsoft Fabric
When organizations discuss enterprise AI, the conversation almost always begins with foundation models, prompts, and agents. Yet in practice, every production enterprise AI initiative succeeds or fails based on how well it answers three equally critical questions: Where does the trusted enterprise data come from? How does business intelligence and context get created from that data? Where will business users ultimately experience and consume that intelligence? This is the modern Azure journey: Data → Intelligence → Experience. And at the very start of this journey sits a foundational architectural principle that guides everything in this post: Modernization does not mean forced migration. Don't change a customer's existing Azure Databricks architecture just to introduce Microsoft Fabric or AI. Instead, demonstrate what additional Microsoft capabilities can be layered around their existing investment with zero data duplication. Why "Why Would We Change Our Architecture?" Is the Right Question If an enterprise has spent years building curated Medallion architectures (Bronze, Silver, Gold), Lakeflow / Delta Live Tables (DLT) pipelines, and fine-grained governance in Azure Databricks Unity Catalog, asking them to rip-and-replace or duplicate pipelines just to light up Power BI Direct Lake or Microsoft Foundry agents is a non-starter. A modern lakehouse strategy shouldn't say: "Here is how to leave Databricks." Instead, it should prove: "Here is how an organization already invested in Azure Databricks can extend that architecture with Microsoft Fabric, OneLake, and Microsoft Foundry—while minimizing architectural disruption and avoiding unnecessary data movement." Data Is 1/3 of the Story: The 3-Act Framework As we designed our end-to-end reference implementation, we structured the architecture around three distinct acts. Your Databricks and Fabric connectivity foundation is Act 1—the essential 1/3 that unlocks the entire downstream AI and user experience: https://raw.githubusercontent.com/anujpandey2/Databricks-Blogs/2fda4c01be139dd54742bb9d9af90593a7c106c4/images/3-act-architecture-diagram.svg Act Pillar Guiding Principle Key Question Answered Core Technologies Act 1 (1/3) DATA Meet customers where their data already lives. Where is my enterprise data, and how can the broader Microsoft ecosystem work with it without copying it? Azure Databricks, Unity Catalog, Mirrored Catalog, Microsoft Fabric, OneLake Act 2 (2/3) INTELLIGENCE Turn trusted business data into intelligence. How can AI models and agents reason over governed enterprise ontologies and metrics rather than raw schemas? Fabric IQ (Ontology, Graph & Semantic Models), Fabric Data Agents, Microsoft Foundry, Azure OpenAI Act 3 (3/3) EXPERIENCE Bring intelligence to where work happens. Where and how will business users actually consume this intelligence without knowing the underlying plumbing? Microsoft 365 Copilot, Microsoft Teams, Power BI, Custom APIs/Apps State of the Union (As of November 2026): Azure Databricks & Microsoft Fabric Connectivity As of late 2026, Azure Databricks and Microsoft Fabric interoperability has matured into a true bidirectional, zero-copy ecosystem built on open standards (Delta Lake, Apache Iceberg, and Unity Catalog Open APIs). Here is the complete landscape of how the two platforms connect today: 1. Inbound to Fabric: Mirroring Azure Databricks Unity Catalog (Generally Available) How it works: Fabric connects directly to your Azure Databricks Unity Catalog metastore using the Unity Catalog Open APIs. It creates a Mirrored Azure Databricks catalog item in Fabric and automatically generates OneLake shortcuts (Databricks type shortcuts) pointing to the underlying ADLS Gen2 Delta tables. Zero-Copy & Auto-Sync: No data is moved or duplicated. When schemas, tables, materialized views, or streaming tables are added or dropped in Databricks Unity Catalog, Fabric automatically synchronizes the metadata (typically propagating changes within seconds to a few minutes). Streaming Tables & Materialized Views Support: By enabling TBLPROPERTIES ('pipelines.externalMetadata.enabled' = 'true') on your DLT / Lakeflow pipelines or standalone objects, even Databricks Materialized Views and Streaming Tables can be mirrored directly into Fabric. Built-in Consumption: Every Mirrored Azure Databricks catalog item automatically provisions a SQL Analytics Endpoint (for T-SQL queries, views, and warehousing), supports OneLake table shortcuts into Fabric Lakehouses, and powers Power BI Direct Lake semantic models and Fabric Data Agents. 2. Outbound from Fabric: OneLake Catalog Federation in Azure Databricks (Preview / Beta) How it works: Going the other direction, Azure Databricks Lakehouse Federation now natively supports OneLake Catalog Federation (CREATE CONNECTION ... TYPE onelake and CREATE FOREIGN CATALOG). Zero-Copy Access from Databricks Compute: Data engineers and data scientists running on Databricks Runtime 18.0+ (or DBR 19.0+ using workspace-specific endpoints <workspace-id>.z<xy>.dfs.fabric.microsoft.com) and SQL Warehouses (2025.40+) can query Fabric Lakehouse and Warehouse tables directly using standard three-part Unity Catalog names (fabric_catalog.schema.table) with zero data copying. Unstructured Data Volume Support: Setting create_volume_for_lakehouse_files 'true' automatically mounts the Fabric Lakehouse /Files directory as a read-only Unity Catalog Volume (/Volumes/<catalog-name>/onelake-folders/files/) for Spark binary/Parquet/PDF processing in Databricks. 3. Shared Storage: OneLake & ADLS Gen2 Shortcuts Both platforms can continue to read/write to existing ADLS Gen2 storage accounts, or Databricks can mount/read/write directly to OneLake endpoints (abfss://<workspace>@onelake.dfs.fabric.microsoft.com/) using Entra ID Service Principals, Managed Identities (via Access Connectors for Azure Databricks), or credential passthrough. What Our E2E Interoperability Implementation Proves (The 5 Architectural Proofs) To make this concrete—following the philosophy of Build Once → Prove It → Document It → Enable Others—our end-to-end walkthrough is designed to prove five non-negotiable enterprise requirements: Existing Databricks investment remains meaningful: We start with a realistic Azure Databricks environment where data engineers curate Bronze, Silver, and Gold tables via Delta Live Tables (DLT) / Lakeflow pipelines registered in Unity Catalog. Interoperability is real (not slideware): We configure live Unity Catalog Mirroring into Fabric and OneLake Catalog Federation back into Databricks. Unnecessary data movement is eliminated: We explicitly show what stays in Azure Databricks and ADLS Gen2 (100% of the physical Parquet/Delta files) versus what gets federated into Fabric (metadata and zero-copy pointers). Fabric layers immediate business value: Rather than stopping at "I can see my Databricks table in Fabric," we layer a Power BI Direct Lake semantic model, Copilot in Power BI, and a Fabric Data Agent on top of the mirrored Databricks tables. Governance and security boundaries are preserved: We document exact authentication, network firewall rules (Workspace Identity + Trusted Workspace Access), and where Unity Catalog vs. OneLake security policies are enforced. End-to-End Walkthrough: One Business Scenario, Multiple Personas Let's walk through a unified enterprise scenario across four personas—Data Engineer, Analytics Engineer / BI Analyst, AI Developer, and Business User—answering a single executive business question: "Why are sales declining and customer churn rising in our target segments, and which products and retention actions should we prioritize?" Act 1 — DATA: Extending the Existing Azure Databricks Estate into Microsoft Fabric Step 1: Curate Enterprise Data in Azure Databricks (Data Engineer Persona) Our data engineering team already runs its core ingestion and Medallion transformations in Azure Databricks. We do not rewrite these pipelines in Fabric. From the Azure Portal, we launch our existing Unity Catalog-enabled Azure Databricks workspace and sign in with Microsoft Entra ID: Please refer to this Github link : anujpandey2/Databricks-Blogs: The Modern Azure Journey: Where Data Meets AI — Azure Databricks & Microsoft Fabric Interoperability Blog and Reference ArchitectureOneLake table APIs: Now with Iceberg writes, plus richer table metadata (Preview)
OneLake table APIs already let Iceberg-compatible engines discover and read tables in Microsoft Fabric. Today we're taking the next step with two additions: Iceberg REST Catalog (IRC) writes through the OneLake table endpoint, now available, and a new additional table metadata API, in preview, for adding descriptions and tags to tables and columns. Together, they make OneLake a place where any engine can not only read data, but also publish it, with the business context people and AI agents need to use it well. Write native Iceberg tables from the engines you already use Until now, the OneLake Iceberg endpoint supported read-only metadata operations. Now, IRC-compatible clients can create native Iceberg tables, commit new snapshots, and request short-lived storage credentials through the same endpoint they already use for reads. That means engines such as PyIceberg, Snowflake, and DuckDB can land results directly in OneLake using standard Iceberg tooling, with no custom connectors or copy jobs. Some of the supported Iceberg REST Catalog operations at the OneLake table endpoint include: Operation What it does Status List/get/check namespaces Discovers the schemas in a data item Available List/get/check tables Discovers tables and returns their Iceberg metadata Available Create table Creates a native Iceberg table in a schema New! Commit table updates Commits new snapshots and metadata changes to a native Iceberg table New! Drop table Removes a table and its data from OneLake New! Load table credentials Returns short-lived storage credentials for a table (credential vending) New! Writes are built for multi-engine environments. Commits use optimistic concurrency checks, so a conflicting update fails instead of silently overwriting a newer table state. Credential vending lets clients write data files without managing separate storage access. And because the tables live in OneLake, they're immediately discoverable by every tool connected to the catalog. Many industry partners are already building on this feature. Snowflake supports the OneLake table APIs through the catalog-linked database feature, which automatically discovers schemas and tables in a Fabric data item, so Snowflake users can read, create, and write Iceberg tables in OneLake. ClickHouse also announced that it now writes Iceberg tables directly to OneLake using these APIs. Add business context to tables and columns Data is only as useful as people's understanding of it. The new additional table metadata API (preview) lets you read and write descriptions and key-value tags for a table and its columns, for example to explain a table's business purpose, record its owning team, or describe how to interpret a column. It uses one common REST route, so you don't need a format-specific catalog API to do it. It also works through internal OneLake shortcuts: updates apply to the target table, so everyone sees the same context whether they read through the shortcut or the source. Explore more in the OneLake documentation Check out our documentation for more information. This includes end-to-end samples, from configuring PyIceberg, Snowflake, and DuckDB for writes and credential vending to a Python sample that updates table metadata safely. Start with the following resources: OneLake table APIs overview OneLake Iceberg metadata API Iceberg table API samples Read and write additional table metadata (preview) We want your feedback! Start writing Iceberg tables to OneLake and try the additional table metadata preview today and share your experience with us. Have feedback? Share your ideas and vote at the Fabric Ideas site and join the conversation in the Microsoft Fabric Community.Fabric Community Engineering Call - November 2026
🚀 We're invited: Fabric Community Engineering Call Our user group has been invited to join the Fabric community engineering call, and we'd love to have you there with us. It's a chance to hear what's happening, ask questions, and connect with the wider Fabric community. 📅 When: Thursday, November 19, 2026 - 9:00 AM to 10:00 AM PDT 📍 Where: Online 🎤 Speakers: Yoav Dobrin & Sandeep Pawar ✨Topic: Deployment Plans: Control What Happens and When, including sequencing deployment and actions consistently across Git, APIs, and deployment pipelines, plus configuring Fabric data agents for conversational analytics 🔗 RSVP on Meetup: https://www.meetup.com/microsoft-fabric-credit-unions-group/ Not on Meetup yet? We're using it to share the details and keep track of who's joining. Click the link, create a free account, join our group, and RSVP. It takes about a minute, and you'll get invites to our future sessions too. Hope to (virtually) see you there! 👋
