User Profile
preshah
Microsoft Employee
Joined 1 year ago
User Widgets
Contributions
Access Amazon S3 Shortcuts Securely and Seamlessly with Microsoft Entra Service Principals (Preview)
Microsoft Fabric now offers a preview of support for Microsoft Entra service principals when using Amazon S3 Shortcuts. This feature allows the use of Entra service principals to securely access S3 buckets without the need for long-term AWS access keys. Previously, S3 shortcuts required access keys. With this update, organizations can authorize access using Microsoft Entra credentials, simplify identity management, and improve security with short-lived, standards-based tokens. Why This Integration Matters Many organizations use both Microsoft and AWS platforms to support their data and analytics workflows, but managing identities across clouds can be complex, error-prone, and time-consuming. This integration makes that much easier. By using OpenID Connect (OIDC), Microsoft Entra service principals can securely assume AWS roles without needing separate IAM users or long-lived access keys. Instead of static credentials, Entra issues short-lived tokens, which AWS trusts at the time of access. This reduces credential sprawl, simplifies security, and gives you centralized control through Entra’s identity policies. All activity is logged in AWS CloudTrail, providing full visibility into cross-cloud activity. Key Benefits Cross-Cloud Identity Management - Use Microsoft Entra service principals to manage access to AWS S3, allowing consistent identity policies across both cloud platforms. Secure by Design - Leverage OIDC for modern, token-based authorization with short-lived credentials. Simplified Operations - Eliminate the need to create or manage IAM users, reducing complexity and credential sprawl. Auditable Access - Audit all role assumptions in AWS CloudTrail, giving you full visibility and traceability. Minimal Disruption - Works with your existing AWS setup, with only a few configuration steps required to get started. Getting Started To enable this integration, use the following steps, a detailed setup guide is available in the documentation. Register a Service Principal in Microsoft Entra Create an app registration, generate a client secret, and capture the tenant ID, client ID, and object ID. 2. Configure AWS IAM Set up an OIDC identity provider using your Entra tenant ID and create IAM roles with trust policies that reference your service principal. 3. Connect via Microsoft Fabric Use the Fabric interface to create a connection to S3 using the role ARN and Entra credentials. Then, create S3 Shortcuts using OneLake’s shortcut interface. Security Best Practices Use a unique service principal per AWS IAM role for strong isolation and auditability. Rotate service principal secrets regularly and store them securely. Monitor AWS CloudTrail logs for STS activity and role assumptions. Current Limitations This integration currently supports only the service-principal based approach. OAuth and workspace identity support are not yet available. Access to S3 buckets behind firewalls, such as through On-premises Data Gateway connections, is not supported with this service principal–based integration. Try it Today If your organization is already using Microsoft Entra and S3, we encourage you to try it out and see how it can simplify your data access and governance. Setup is straightforward, and you’ll be able to take advantage of secure, efficient access to your data from day one. Get started now!Extending interoperability: Azure Databricks can now store Unity Catalog managed tables directly in OneLake
As organizations scale their data and AI investments, they increasingly adopt a multi-platform approach, enabling teams to use the tools that best fit their needs. The challenge has been enabling both platforms to work from a single copy of data without duplicating storage or building complex pipelines. With today's updates, Azure Databricks and Microsoft Fabric take a major step toward true bi-directional interoperability through OneLake. Reading data across platforms is already available: Onelake catalog federation (now generally available) lets Azure Databricks query OneLake data directly, and Mirrored Azure Databricks Catalog makes Unity Catalog tables stored in ADLS Gen2 accessible in Fabric. Now, we are taking this a step further — Azure Databricks can store Unity Catalog managed tables directly in OneLake: OneLake external location support for Unity Catalog (Beta): Azure Databricks can now store Unity Catalog managed tables directly in OneLake. Expanding your Azure storage options alongside ADLS Gen2, your workloads can seamlessly target OneLake as an additional foundational storage layer for Unity Catalog. Publish to Fabric (Preview): A streamlined workflow to create Mirrored Azure Databricks Catalog items directly from Azure Databricks Catalog Explorer. This works for Unity Catalog tables regardless of where they're stored, whether in OneLake or in ADLS Gen2. Once published, tables are immediately queryable across all Fabric workloads. Why this matters With support for both reading and storing directly in OneLake, Azure Databricks customers can now use OneLake as a native storage layer for their Delta tables without managing separate storage systems. This provides flexibility to store data in OneLake while using preferred tools in either Microsoft Fabric or Azure Databricks for each project. Because these remain fully managed Unity Catalog tables, all existing governance and optimization capabilities extend to data stored in OneLake. Because there is a single copy of data, changes made in one environment are immediately reflected in the other. This eliminates the need for data movement, reduces duplication, and simplifies how organizations manage and govern their data estate. And with Publish to Fabric, you don't have to choose between storage locations to get your tables into Fabric. Whether your Unity Catalog tables are backed by OneLake or ADLS Gen2, Publish to Fabric creates mirrored items in any Fabric workspace, initiated directly from the Azure Databricks experience. The result is the same data accessible in both Azure Databricks and Fabric with no copies, no movement, and no duplication. This represents a shift toward a shared data foundation across platforms. Organizations no longer need to choose between tools or duplicate data to support different workloads. Key capabilities Store Unity Catalog managed tables directly in OneLake (Beta) This is the core of the announcement. OneLake external location support enables you to create Unity Catalog managed tables stored in Microsoft OneLake. You create a UC external location mapped to a OneLake path, and from that point forward, all UC asset types (managed tables, views, materialized views, and streaming tables) are stored directly into OneLake. Unlike the existing pattern where UC tables are stored in ADLS Gen2 and surfaced in Fabric through mirroring, this approach makes OneLake the primary storage destination. Data written by Azure Databricks lives directly in OneLake from the start. Publish to Fabric: Surface OneLake backed tables in Fabric (Preview) Once your Unity Catalog tables are stored in OneLake, you need a way to make them visible and queryable in Fabric. That's where Publish to Fabric comes in. While Mirrored Azure Databricks Catalog already enables reading UC tables in Fabric, Publish to Fabric provides a streamlined, Azure Databricks native workflow to create mirrored catalog items directly from Catalog Explorer. Once published, Fabric polls Unity Catalog metadata to keep the mirrored catalog in sync. DDL changes made in Azure Databricks (new tables, dropped tables, and schema modifications) are reflected in Fabric at the next sync interval. You can also Refresh on the mirrored item to force an immediate sync. Tables are read-only in Fabric since the source of truth remains in Azure Databricks, and you can publish the same catalog to multiple Fabric workspaces independently. Getting started Part 1: Set up OneLake as UC storage Configure identity and access — Create or identify a Managed Identity or Service Principal in Azure, and assign it a workspace role (Admin, Member, or Contributor) in your target Fabric workspace. Provision storage in Fabric — Create an Azure Databricks Storage item in your Fabric workspace. Create a storage credential and external location — In Azure Databricks, create a Unity Catalog storage credential and an external location pointing to OneLake. Create catalogs and tables — Create Unity Catalog assets on the OneLake external location. Data written to these tables lands directly in OneLake with no copy or pipeline. Part 2: Publish the catalog to Fabric Grant permissions — Assign the required Unity Catalog permissions to your Service Principal or user identity so Fabric can access the catalog metadata. Create a Fabric connection — In Fabric, set up a cloud connection to your Azure Databricks workspace. Publish from Catalog Explorer — In Azure Databricks, select your catalog and use the Publish to Fabric workflow to push it to your chosen Fabric workspace. Query in Fabric — Once published, tables are immediately queryable across all Fabric workloads including SQL analytics, notebooks, and Power BI. For detailed step-by-step instructions, review Configuring OneLake external locations and Publish a Unity Catalog catalog to Microsoft Fabric. Resources Mirroring Azure Databricks Unity Catalog Configuring OneLake external locations Publish a Unity Catalog catalog to Microsoft Fabric Microsoft OneLake documentation We'd love to hear your feedback. How are you using Azure Databricks and Fabric together? What additional capabilities would make this integration more powerful for your team? Share your thoughts in the Fabric Ideas forum!Nested folders support in shortcut transformations (Generally Available)
Nested folders, shortcut transformations, subfolder processing, OneLake, shortcuts, Lakehouse Data lakes rarely come in flat structures. In practice, data is organized across multi-level folder hierarchies, partitioned by date, region, source system, or business unit. Previously, shortcut transformations processed files at a single folder level, requiring users to set up separate transforms for each subdirectory. Moving forward, nested folders support in shortcut transformations will enable recursive discovery and transformation of files across your entire directory hierarchy, automatically. This builds on the recently announced shortcut transformations (Generally Available) for structured file formats (CSV, Parquet, JSON), extending the zero-ETL model to work with hierarchical folder structures out of the box. Why this matters Consider a typical data lake layout: sales data partitioned by year, then quarter, then month, with new folders appearing as time progresses. Previously, transforming this data required targeting each leaf folder individually, or flattening your storage structure to work within the single-level constraint. Neither approach scales well, nor both add manual overhead that shortcut transformations were designed to eliminate. Nested folders support removes this limitation. You point a shortcut transformation at a top-level folder, and the system takes care of the rest, discovering files across all subdirectories, detecting changes incrementally, and applying transformations consistently across the entire hierarchy. Key capabilities Nested folders support extends shortcut transformations with capabilities designed for hierarchical data layouts. Recursive change detection The system automatically discovers files across all subdirectories within the target folder. When new files land in any subfolder, they are detected and processed incrementally. No manual intervention, no missed files buried three levels deep. Automatic partition pickup This is particularly powerful for data lakes with continuous ingestion patterns. When new partitions appear over time (for example, a new month or region folder), they are picked up automatically without requiring reconfiguration. Directory structure preservation The transformed output maintains the relative folder hierarchy of the source. If your source data lives in sales-data/2026/Q1/jan/transactions.csv, the transformed Delta table preserves that path structure. This ensures traceability, downstream compatibility, and logical organization of transformed data. Safety and cycle prevention Shortcuts nested inside the target folder are not traversed, only physical folders and files are processed. This prevents infinite recursion, cyclic dependencies, and unexpected data duplication. Consistent transformation across all levels The same transformation logic, delimiter settings, header configuration, and schema inference applies uniformly to every file discovered across the folder tree. One configuration covers the entire hierarchy. Getting started Select a top-level folder as your shortcut target In your Lakehouse, create a new table shortcut and browse to the folder that contains your hierarchical data. Select the top-level folder rather than individual subfolders. Figure: Selecting a source folder in the OneLake shortcut creation wizard in Microsoft Fabric. Enable subfolder processing The "Include subfolders" option is enabled by default; the system is designed to work recursively out of the box. If you need single-level behavior for a specific shortcut, you can opt out by unchecking this option. Figure: Subfolder option during shortcut creation. Configure and create Set your transformation options (delimiter, headers, table name) and create the shortcut. Fabric will discover all files across the folder hierarchy and begin transforming them into a Delta table. Managing nested folder shortcuts Shortcut properties clearly indicate whether subfolder transformation is enabled, providing clear visibility into which shortcuts are processing recursively: Existing shortcuts created before this feature will show subfolder transformation as "No". New shortcuts with subfolders enabled will show "Yes". Figure: Shortcut properties showing subfolder inclusion status. Resources To learn more about nested folders support, configuration options, and how shortcut transformations work with hierarchical data, refer to transform structured files into Delta tables. We would love to hear your feedback. What folder structures are you working with? What additional controls would be most valuable? Your input directly shapes what we build next. Let us know in the Fabric Ideas forum!Shortcut transformations: Turn files into Delta tables without pipelines (Generally Available)
Organizations today manage data across multiple storage systems, often in formats like CSV, Parquet, and JSON. While this data is readily available, turning it into analytics-ready tables typically requires building and maintaining complex ETL pipelines. Shortcut transformations remove that complexity. With Shortcut transformations, you can convert structured files referenced through OneLake shortcuts into Delta tables without building pipelines or writing code. This release applies to structured file formats such as CSV, Parquet, and JSON. AI powered transformations are currently available in public preview as we continue to expand and refine these capabilities. Why this matters Preparing data for analytics has traditionally required building ingestion pipelines, managing compute jobs, and orchestrating refresh schedules. This introduces complexity, increases operational overhead, and slows down time to insight. Shortcut transformations change this model. Instead of moving and transforming data through pipelines, you can reference data where it lives using OneLake shortcuts and let Fabric handle ingestion, transformation, and synchronization. Fabric automatically converts files into Delta tables and keeps them continuously synchronized with source data. This ensures your data is always current and ready for analysis, without requiring pipeline orchestration. The result is simpler architecture, reduced operational overhead, and a faster path from raw data to insights. Key capabilities Shortcut Transformations bring together a set of capabilities designed for modern data workflows. No pipelines or code required Convert files into analytics‑ready Delta tables with a fully managed ingestion and sync experience. Always in sync with source data Shortcut transformations continuously detect changes and apply them incrementally, keeping tables current without requiring scheduled jobs. Support for nested folder structures Automatically detect and process files across hierarchical folders, ensuring changes are captured regardless of how data is organized. Automatic schema handling Fabric automatically infers schema and safely evolves tables as new columns appear, with built in support for semi structured data such as nested JSON. Native Delta Lake output All transformations produce Delta tables that are immediately available across Microsoft Fabric, including SQL, Spark, and Power BI. These tables support analytics, reporting, and AI workloads on a unified data foundation. Improved cost efficiency Eliminates always on pipelines and unnecessary compute. Transformations run only when changes are detected, minimizing compute and storage overhead. Getting started Create a new table shortcut In your Lakehouse, select New Shortcut under the Tables section. For Lakehouses with schema, select New table Shortcut for a particular schema. Connect to your data source Choose from supported sources such as Azure Data Lake Storage, Azure Blob Storage, Amazon S3, Google Cloud Storage, Dataverse, SharePoint, OneDrive, and more. Select files and configure the transformation Browse to your data and configure how the data should be interpreted directly in the Fabric user experience, without writing code Define the delimiter (comma, semicolon, pipe, tab, etc.) Indicate whether the first row contains headers. Provide a friendly name for the table. Figure: Auto‑transform applied during shortcut creation, converting CSV data to Delta. Create the table Fabric automatically transforms the selected files into a Delta table in your Lakehouse /Tables folder. Resources To learn more about supported file formats, configuration options, and how shortcut transformations work in practice, refer to the shortcut transformations documentation.Mirroring Azure Databricks catalogs from Azure Databricks workspaces behind private endpoints (Generally Available)
Enterprises increasingly deploy Azure Databricks workspaces in locked-down network environments where public network access is disabled, and all access is routed through private endpoints. While this security posture is essential for meeting enterprise and regulatory requirements, it has historically limited integration scenarios for the Mirrored Azure Databricks catalog item in Microsoft Fabric. Earlier, the Mirrored Azure Databricks Catalog item required Azure Databricks workspaces to be reachable through public network paths, which made it incompatible with private only deployments. Microsoft Fabric now supports mirroring Azure Databricks catalogs from Azure Databricks workspaces that are behind private endpoints, enabling a more seamless, combined experience for customers using both Azure Databricks and Fabric for data analytics use cases. This capability is now generally available, and it uses the Virtual Network (VNet) data gateway to establish secure private connectivity. Customers can mirror governed Unity Catalog metadata and data into Fabric from Azure Databricks workspaces that are accessible only through private endpoints. Why this matters Numerous customers are standardizing on Azure Databricks and Microsoft Fabric together for their end‑to‑end analytics platforms, often running Azure Databricks in highly regulated virtual network environments with private endpoints and no public network access. Until now, customers with Azure Databricks workspaces secured behind private endpoints were unable to use the Mirrored Azure Databricks catalog item. With this release, customers can maintain strong network isolation for Azure Databricks while still delivering a unified analytics experience in Fabric, so they no longer need to choose between security and integration when using the two platforms side by side. Microsoft Fabric now seamlessly integrates with Azure Databricks environments that have public network access disabled. Key benefits Secure connectivity for private Azure Databricks workspaces Mirrored Azure Databricks Catalog items can connect to Azure Databricks workspaces that are accessible only through private endpoints, including workspaces with public network access disabled. Enterprise grade compliance and data protection All communication between Microsoft Fabric and Azure Databricks occurs through private IP space on the Azure backbone. Sensitive metadata and data remain isolated from the public internet. Simplified networking for IT and data teams The solution relies on standard Azure building blocks such as private endpoints, managed identities, and Virtual Network data gateways. This avoids the need for custom networking or proxy solutions. Consistent experience across Fabric This capability aligns the Mirrored Azure Databricks Catalog item with other Fabric connectivity models, delivering consistent governance and analytics experience across Fabric workloads. How it works Mirrored Azure Databricks Catalog connecting to a VNET-injected Databricks workspace using a private endpoint Microsoft Fabric uses the Virtual Network data gateway to establish secure private connectivity to Azure Databricks workspaces behind private endpoints. The gateway is deployed into a customer managed virtual network and routes requests from Fabric to Azure Databricks through a private endpoint. All communication remains on the Azure backbone and does not traverse public networks. This approach preserves the managed Software as a Service experience of Fabric while aligning with enterprise network security requirements. Try it today If your organization is already using Azure Databricks workspaces secured behind private endpoints, you can start using this capability today. To get started, refer to mirroring Azure Databricks catalogs in Microsoft Fabric from workspaces accessible only through private endpoints. Learn more For an overview of mirroring Azure Databricks catalogs in Fabric, see Mirroring Azure Databricks Unity Catalog. For details on private connectivity in Azure Databricks, see Configure private connectivity to Azure Databricks. For more information on how private connectivity is established from Fabric, see Create virtual network data gateways.Securely Access VPC-Protected Amazon S3 Buckets in Microsoft Fabric with Entra Integration (Preview)
When we first introduced Amazon S3 shortcut integration with Microsoft Entra ID, customers gained a powerful new way to connect S3 data to Microsoft Fabric — without storing or rotating AWS access keys. Using OpenID Connect (OIDC), Fabric authenticates directly with AWS Identity and Access Management (IAM), enabling secure, identity-based access to cloud storage. However, many enterprises keep their S3 buckets locked down inside Virtual Private Clouds (VPCs) or behind corporate firewalls. In these environments, Entra OIDC can authenticate identities, but it cannot provide network access — so Fabric still cannot reach the S3 endpoint. That changes today. With support for the on-premises data gateway, Microsoft Fabric can now securely access S3 buckets that reside behind a VPC or firewall, bridging identity-based Entra OIDC authentication with the private network reach your environment requires. Why This Integration Matters Enterprises in regulated sectors often face a trade-off between security and agility. They need to keep data inside private clouds or corporate networks, but they also want to unlock the insights that Fabric’s unified analytics platform delivers. This new capability removes that trade-off. By combining Microsoft Entra ID based OIDC authentication with the on-premises data gateway, Fabric can securely reach S3 buckets that live inside VPCs or behind corporate firewalls — without ever exposing data to the public internet. You maintain strict network boundaries while still enabling governed, analytics-driven experiences in Fabric. Key Benefits Private, Secure Access – Access S3 data privately without opening public endpoints or relaxing firewall rules. Zero-Trust Identity Control – Maintain strong identity governance using Microsoft Entra ID and OIDC, eliminating reliance on long-lived AWS access keys. Simplified Management – No secrets, no key rotation, and no IAM user lifecycle to manage. Unified Governance and Auditing – Maintain full cross-cloud visibility as Fabric governs usage and AWS CloudTrail logs all S3 access and Microsoft Entra role assumptions. Setting This Up This capability builds on two existing flows: Identity & OIDC setup using Microsoft Entra service principals for Amazon S3 shortcuts. Private network connectivity to VPC-protected S3 buckets using the on-premises data gateway. Below is the high-level sequence, with links to the detailed step-by-step guides. Step 1: Configure Entra–AWS trust for S3 shortcuts First, set up Microsoft Entra ID and AWS IAM so that Fabric can use a Microsoft Entra service principal with OIDC to assume an AWS role and access your S3 bucket. Register a service principal in Microsoft Entra ID. Configure an OIDC trust relationship between Entra and AWS IAM. Create an IAM role and policy that grants the required S3 permissions. To configure the identity trust between Entra and AWS, follow the steps outlined in the Access Amazon S3 Shortcuts Securely and Seamlessly with Microsoft Entra Service Principals (Preview) blog post. Step 2: Enable private access to your VPC-protected S3 bucket Next, make sure Fabric can reach your S3 bucket over a private network path via the on-premises data gateway. Configure your VPC endpoint / network path to S3. Install and configure the on-premises data gateway in your network. Verify that the gateway can reach the S3 endpoint privately. For a full walkthrough, refer to the Creating a shortcut to a VPC-protected Amazon S3 bucket blog post. Step 3: Create the Fabric S3 shortcut using both identity and network paths Finally, bring it all together in Fabric: Create (or reuse) a Fabric connection that uses your Entra service principal–based integration. When creating the Amazon S3 shortcut, select the connection and the on-premises data gateway. Browse to the VPC-protected bucket or folder and complete the shortcut creation. At this point, identity flows through Microsoft Entra and AWS IAM, network traffic flows through your private gateway, and Fabric can query S3 data securely without exposing it to the public internet. Security Best Practices Use a unique service principal per AWS IAM role for strong isolation and auditability. Rotate service principal secrets regularly and store them securely. Monitor AWS CloudTrail logs for STS activity and role assumptions. Current Limitations This Microsoft Entra integration currently supports only the service-principal based approach. OAuth and workspace identity support are not yet available. Try it Today If your organization is already using Microsoft Entra and S3, we encourage you to try it out and see how it can simplify your data access and governance. Setup is straightforward, and you’ll be able to take advantage of secure, efficient access to your data from day one. Get started now!Re: Identity Federation for S3 shortcuts for tokenless Authentication based on Workspace Identity
Thank you for the feedback. The feature was released last week in public preview. Docs - https://learn.microsoft.com/en-us/fabric/onelake/amazon-storage-shortcut-entra-integration Blog - https://blog.fabric.microsoft.com/en-US/blog/24780/Re: Issue with Mirrored Azure Databricks Unity Catalog Tables: Data Preview Unavailable
Hello everyone, I wanted to check if the issue still persists where data preview fails intermittently. Appreciate if you can share the support tickets raised and I am happy to follow up. We are also in parallel actively trying to reproduce and look at the issue. Thank you.
