Best practices for integrating the Fabric ML Model endpoint with an application (preferably .NET)?
Hi, I'm working on a ML use case in Microsoft Fabric where we train models using Fabric notebooks, track experiments through MLflow, and register models for real-time predictions. We have already registered a model and activated its real-time inference endpoint in Fabric. We're now planning to integrate this endpoint with an application (preferably .NET). I'd appreciate some guidance on the recommended integration approach, particularly around authentication, permissions, and API invocation. A few questions: Authentication: What is the recommended way for the backend (.NET) to authenticate with a Fabric ML Model endpoint? Does it support OAuth 2.0 Client Credentials using a Microsoft Entra service principal, and is Managed Identity also supported? Permissions: What Fabric workspace or ML Model permissions are required for a service principal to invoke an activated endpoint? Are any additional tenant-level settings required? API integration: What is the recommended approach for calling the endpoint from .NET application, including the request payload format, response handling, and access-token management? Endpoint behavior: Are there any considerations around synchronous/asynchronous responses, latency, timeouts, or concurrent requests when integrating the endpoint with an application? Model versioning: If we train and register an updated model version later, can we activate a new endpoint while keeping the existing version available? What is the recommended approach for switching between versions without affecting the application? If anyone has implemented a similar integration, I'd appreciate any recommended architecture, documentation, or working. Also would like to understand whether there is a secure Fabric-native approach that would be more appropriate. Thanks in advance!Microsoft Fabric Notebook Alternatives to Private Endpoints and Custom Live Pools?
We are experiencing challenges connecting Microsoft Fabric notebooks to Azure Key Vault when the Key Vault is configured with public network access disabled. Our organization follows strict cybersecurity policies requiring Azure Key Vault to remain private, without exposing it to public networks. CURRENT CHALLENGES 1) Dynamic Outbound IP Addresses Microsoft Fabric notebooks use outbound IP addresses that can change periodically. Maintaining an IP allowlist for Fabric is not a sustainable solution, as the IP addresses may change frequently (approximately weekly in our experience). Our cybersecurity team does not support enabling public network access to Key Vault or continuously updating firewall rules. 2) Private Endpoint Connectivity and Notebook Startup Time We have explored using Microsoft Fabric Managed Private Endpoints to securely access Azure Key Vault. However, this approach introduces a significant performance concern. In our testing, notebook execution takes approximately 5 minutes to initialize, creating substantial overhead for workloads that otherwise complete quickly. We have multiple scheduled notebooks and pipelines, so this additional startup time affects overall processing efficiency 3) Custom Live Pools and Additional Cost We understand that Custom Live Pools may help reduce Spark session startup latency by keeping compute resources available. However, maintaining these resources introduces additional capacity consumption and potentially higher operational costs. We would prefer to avoid paying for continuously available Spark compute solely to retrieve secrets securely from Azure Key Vault. OUR REQUIREMENTS Azure Key Vault must remain private, with public network access disabled. No dependency on maintaining dynamic Fabric outbound IP allowlists. Secure authentication using managed identity or another Microsoft Entra ID-based approach. Minimal notebook startup overhead, especially for short-running notebooks. Cost-effective architecture that supports multiple production notebooks and scheduled pipelines. Is there a recommended architecture for Microsoft Fabric notebooks to securely access a private Azure Key Vault without requiring Managed Private Endpoints and Custom Live Pools together? Is there a way to use Managed Private Endpoints while retaining the faster startup experience of standard Fabric Starter Pools? Does Microsoft Fabric provide any alternative private networking mechanism, such as trusted service access, workspace identity, or another supported integration, that would allow notebooks to retrieve Key Vault secrets without public network exposure? Are there any recommended approaches for reducing Spark session initialization time when accessing private Azure resources? Our primary objective is to maintain security compliance without introducing significant notebook startup delays or unnecessary Fabric capacity costs. Any recommendations, architectural guidance, or real-world implementation experiences would be greatly appreciated. Thanks, Suresh Guddanti
Clinical and IoT data on OneLake: Fabric data science or Azure ML for predictive models?
Hi all, While working on a Fabric healthcare analytics platform at DataToBiz, we made some design choices on clinical, patient monitoring and IoT data that I’d like to compare with how people here would handle them. The sources were SQL Server, SharePoint, clinical systems, ERP/CRM systems, APIs, files and IoT devices, which had been feeding legacy BI tools through manual ETL with reporting delays of 2 to 3 days. We pulled everything into OneLake with Data Factory pipelines, then used Data Factory and Synapse for transformation, validation and preparation. For ML, we used Synapse Data Science and Azure ML for predictive modeling on patient outcomes, resource forecasting and anomaly detection. Power BI handles reporting with automated refreshes and row-level security, Power Automate covers workflows and alerts, and Purview provides lineage and auditing, with role-based access on OneLake to meet HIPAA and GDPR requirements. It works, but a few decisions are still open for me. On the ML side, we ended up using both Synapse Data Science and Azure ML. If you were starting now, would you do everything in Fabric’s data science experience, or keep Azure ML for deployment and monitoring? Where do you draw the line, especially for models that regulators or auditors might ask about? For the IoT data and anomaly detection, we processed it in batches. Would you move that to Eventstreams and an Eventhouse for near real-time detection, or is batch good enough for most monitoring use cases? What made you decide? For sensitive data, how do you split security between OneLake access controls, the Warehouse or Lakehouse, and RLS in the semantic model? I’m particularly interested in how you prove access and lineage to an auditor in practice, whether Purview and the audit logs are enough or you need to build more. And for the move from SQL Server and the legacy BI tools, did you use mirroring or Data Factory copy pipelines? I’d like to hear how you handled the cutover without disrupting reporting. If you’ve built something similar, or tried an approach and dropped it, I’d really like to know what worked and what didn’t. Thanks, and I’d love to hear how you’ve handled it.
Best approach to load report-style Excel files from a SharePoint folder into Lakehouse tables
Hi all, I’d like advice on the best pattern for loading business-maintained Excel files from SharePoint into a Fabric Lakehouse. Scenario Business users upload a monthly Excel file per department to a SharePoint Online document library folder (HSE first; Finance and Internal Audit later). Each file is year-to-date (Jan up to the reported month). The plan is one file per month, named like HSE_Performance_YYYY_MM.xlsx. The sheet is a report layout, not a flat table: two header rows: month names merged across 3 columns, then HQ / Marine / Contractor section header rows inside the data (e.g. “Incident Rates”, “Minor Incidents”) ~120 KPI rows, YTD total columns at the end formula cells that return #DIV/0! The target is about 19 Delta tables (one per KPI section, with KPIs as columns), plus a load log table. Current solution (working in DEV) A PySpark notebook reads the workbook with openpyxl, detects sections, unpivots and pivots, and writes the 19 tables. This works well when the file is uploaded manually to Lakehouse Files. Planned logic: each run, pick the latest file per year; skip files already loaded (name + modified time); replace only that year’s rows; write all tables or none; copy the processed file to an archive folder in Lakehouse Files; log success or failure and alert on failure. Where I need advice OneLake SharePoint shortcut: our service account has “Can view” access directly on the target folder only, not on the parent folders or the library. The shortcut wizard connects at the site URL and browses down. Is folder-level access enough to create the shortcut, or does the identity need access at library or parent-folder level? Has anyone used folder-only permissions successfully? Moving files in SharePoint: as I understand it, SharePoint shortcuts are read-only (writing returns “This operation is not supported through shortcuts of account type OneDriveSharePoint”). Is that correct? Is the Graph API (service principal with Sites.Selected) the recommended way to move processed files to Archive / Error folders in SharePoint? Dataflow Gen2 as a fetch step: Copy activity doesn’t support the SharePoint folder connector, only lists. Has anyone used Dataflow Gen2 just to land the raw sheet into a staging table (all columns as text, errors replaced), then a notebook for the transformation? Any issues with merged cells, error values or column types? Production authentication: for unattended runs, what do you recommend for SharePoint access: workspace identity, a service account (organizational account) or a service principal? Any limitations with workspace identity for SharePoint shortcuts or the SharePoint folder connector? Overall pattern: for monthly, business-maintained, report-style Excel files, would you choose: (a) shortcut + notebook, (b) Dataflow Gen2 fetch + notebook, or (c) notebook calling the Graph API directly? Or is there a better pattern you’ve used? Volumes are small (one file per department per month, under 1 MB), so simplicity and reliability matter more than performance. Thanks in advance!
Connect to an Azure SQL Server with no public access
We have an Azure SQL Server that doesn't permit public access. When accessing it from a Fabric workspace, it looks like we have, essentially, 2 options: either setup private endpoints and configure these. This has the (huge) disadvantage that startup time for spark environments goes from 5 seconds to 3 minutes. That means every time I use a spark notebook (even when it doesn't actually use the endpoints), it warms up for 3 minutes, which is a lot of overhead (we historically have a lot of short running notebooks, each now facing this slowdown). or use a data gateway (virtual or on premise). However, when we connect to this same SQL Server from a notebook (using jdbc), it appears that this connection does not pick up on the data gateway (I don't see how it could, anyway). So this means my notebooks cannot communicate to this SQL Server? (Note: when accessing an ADLS Gen 2 storage (also with public access disabled), there is the option to allow access to it with the workspace identity. A similar option seems not to be present for Azure SQL Server (there is only on option to "Allow Azure Services to access this", but that grants way too much access and probably doesn't affect Fabric connections)?) Can anybody enlighten me on what is the right approach for this scenario? So I hope you can tell me: either you inform me that private endpoints are the way to go, and then have a solution to avoid the long warmup for notebooks either you say a gateway is the solution and then I hope you can confirm that connections from a notebook straight to this SQL Server will indeed utilize this gateway or who knows: you may have another option I'm not seeing right nowBest approach to load Sybase views into a Bronze Lakehouse: overwrite, append or daily snapshots?
Hi all, We’re loading data from an on-premises Sybase (SQL Anywhere) database into a Fabric Bronze Lakehouse using a metadata-driven pipeline with the Copy activity over an ODBC connection. The sources are views, not tables, so we have no primary keys or modified-date columns for incremental loads. We’re unsure how Bronze should be maintained for these views: Full overwrite: replace the table with the current view data on every run. Append daily snapshots: load the full view every day with a load date. For example, 100,000 rows today and the same 100,000 again tomorrow under tomorrow’s date, so the table keeps growing. Something else: for example, overwrite the table and also keep dated raw files for history. Questions: What is the recommended or best-practice approach for Bronze when the sources are views without change tracking? Is keeping daily snapshots in Bronze common, or is Delta time travel considered enough for history and recovery? If you use full overwrite, how do you protect against a bad load, such as the view returning 0 rows? Thanks!
Notebook Execution Fails When Using Workspace Identity in Microsoft Fabric Pipeline
I want to execute the notebook using the Workspace Identity, as it is required to authenticate with Azure Key Vault (No other soloution). I have already enabled the Workspace Identity and granted the required Member permissions. Then, through Manage Connections and Gateways, I created a new cloud connection with the connection type set to Notebook. {I have checked bothway (Allow Code-First Artifacts..................................... ☑️ and 🔲)} Then, within the pipeline, the notebook activity connection was configured as shown below. I added the notebook that needs to be executed using Workspace Identity into the pipeline. But When I run the pipeline I got following error. { Failed to get workspace details with http error code - 'Unauthorized', error message - '{"requestId":"requestId","errorCode":"Unauthorized", "message":"The caller is not authenticated to access this resource","isRetriable":false}' } Is there a way to correct this error?
