Shared data, one consistent view across teams.
Healthcare analytics is a good example of why this matters. Clinical teams, operations teams, and management may need different reports, but those reports still depend on reliable, consistent patient data. If information sits across operational systems and SharePoint files, preparing it for reporting can become a challenge of integration and data quality before visualization even begins. One detail I found interesting in a Microsoft Fabric healthcare analytics project we recently took for datatobiz is the use of OneLake alongside automated data processing and Power BI dashboards. The architectural point is worth discussing: bringing data into a common environment is only the first step. The next is making sure the data is appropriately prepared, governed, and structured for the people using it. For those working with Fabric, how are you organizing data from multiple sources before it reaches the reporting layer? Do you prefer a Lakehouse-first approach, a Warehouse-first approach, or a combination based on the workload?
How to get a Custom JSON Response Instead of the Default DataFrame Format from a Fabric ML Endpoint?
Hi , I'm currently exploring Fabric ML Model Endpoints. The endpoint currently returns predictions in a DataFrame-oriented JSON structure, similar to: { "formatType": "dataframe", "orientation": "values", "predictions": [ [ 0.92, "Category A" ] ] } However, I would like the endpoint to return a JSON response with named fields, such as: { "predictions": [ { "probability": 0.92, "category": "Category A" } ] } I understand that the model's output can be modified, but the REST API appears to use a predefined response structure. I wanted to clarify: Is there any supported way to customize the HTTP response structure directly from a Fabric ML Model Endpoint? Can the formatType or orientation parameters be configured to return named JSON objects instead of positional arrays? Would modifying the model's predict() output or MLflow signature affect the HTTP response format? Is there any Fabric-native alternative to achieve this without introducing an external API layer? I'm particularly interested in whether this is supported natively within Fabric, rather than transforming the response in the consuming application. Any guidance, documentation, or recommended approaches would be appreciated. Thanks!Shortcut to Lakehouse table does not reflect newly added columns after schema change
Hi all! I'm seeing what looks like a metadata synchronization issue with Lakehouse shortcuts across workspaces. Scenario Workspace A Lakehouses: Silver and Gold Table: test_table A new column was introduced upstream to Silver lakehouse using: ALTER TABLE test_table ADD COLUMNS (country STRING) The data is then propagated from Silver to Gold through a Dataflow Gen2. In the Gold Lakehouse, the final table contains the new column (renamed to pais during the transformation). Verification in the SQL Analytics Endpoint of Workspace A: SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'test_table'; Result includes: pais Workspace B A Lakehouse shortcut points directly to the same test_table table in Workspace A. Running the same query: SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'test_table'; returns all expected columns except pais. Troubleshooting already performed Refreshed Lakehouse and SQL Analytics Endpoint Refreshed browser session Closed and reopened Fabric Waited several hours Deleted and recreated the shortcut multiple times Confirmed the shortcut points to the actual table (not a view) Confirmed the column exists and is queryable in the source workspace Question Are schema changes introduced after shortcut creation (for example via ALTER TABLE ... ADD COLUMNS) expected to propagate automatically to Lakehouse shortcuts? Has anyone seen shortcuts keeping an old schema even after recreation and refresh? Thanks!
FabricPipelineWorker.exe crash with SAP connection
Hello all, We are experiencing recurring crashes of FabricPipelineWorker.exe when using Microsoft Fabric Data pipelines to extract data from SAP through the SAP Table Application Server connector and an On-premises Data Gateway. The issue appears to be similar to the one reported in this discussion: https://community.fabric.microsoft.com/discussions/df_pipelines/faulting-application-fabricpipelineworker-exe/5364506 For some reason I cannot reply on mentioned post with message: "This content was not accepted by moderation." However, despite updating the relevant components and changing the gateway configuration, the issue persists. Issue description The interesting part is that Copy activities complete successfully in Microsoft Fabric, and we do not see any errors or failed pipeline executions in Fabric itself. In the On-premises Data Gateway logs, the Copy activities also return: Status: Succeeded However, FabricPipelineWorker.exe appears to crash after the data transfer has completed, potentially during the cleanup phase when existing SAP connections are being closed or disposed of. This is currently our assumption based on the observed behavior, rather than a confirmed root cause. Although the operations complete successfully, these crashes generate a very large number of error logs on the SAP side, creating additional operational overhead and making monitoring more difficult. Environment On-premises Data Gateway: 3000.334.4 Microsoft C++ Runtime 14.x: 14.50.35719.0 SAP .NET Connector (NCo): 3.1.8.0 Troubleshooting performed We have already taken the following steps: Updated the On-premises Data Gateway to version 3000.334.4. Updated the SAP .NET Connector to version 3.1.8.0. Verified the Microsoft C++ Runtime version. Disabled gateway load balancing to rule out potential issues with connections being distributed across gateway nodes. Reviewed the gateway logs and confirmed that Copy activities report successful completion. Unfortunately, none of these changes has resolved the issue. Since the data transfers themselves are successful, this appears to be more related to the gateway worker process lifecycle or SAP connection cleanup than to SAP connectivity or data extraction. Any insights from the Microsoft Fabric engineering team or others experiencing similar behavior would be greatly appreciated. Thank you!Fabric ML Model Endpoint returns 401 from application but works in Postman with the same token
Hi , Now, I'm currently integrating the Fabric ML Model endpoint with a .NET application and encountering a 401 Unauthorized error when calling the scoring API. The endpoint is activated and working as expected when tested directly through Postman. However, when the application calls the same Fabric scoring API, it returns HTTP 401. As far as we've verified, the same Bearer token and authentication details are being used. Endpoint pattern: POST /v1/workspaces/{workspaceId}/mlmodels/{modelId}/endpoint/score Error response: { "errorCode": "RequestFailed", "message": "Unable to process the request", "isRetriable": false } Additional details from the application logs: HTTP status: 401 Unauthorized Authorization header present on final request: True WWW-Authenticate header: Empty Postman request: Successful Application request: Fails with 401 The error response doesn't provide any additional information to identify the exact cause. I'm trying to understand whether there are any specific authentication, authorization, endpoint configuration, or HTTP client requirements when calling Fabric ML Model endpoints from an external application. Has anyone encountered a similar issue where the same token works in Postman but fails when used in the application? Are there any additional headers, permissions, redirect behaviors, or diagnostic steps that we should check to identify the cause? Any suggestions? Thanks!Microsoft Fabric SAP Table Copy Job Fails During Incremental Load with `__rowMarker__` Column Error
Hey Everyone, I am currently working on a Microsoft Fabric Copy Job, using SAP Table Application Server as the source to fetch data from SAP HANA. I have enabled the ABAP Add-on option and configured a Fabric SQL Database as the destination, with the copy mode set to Incremental Load. The initial full load completes successfully, and the data is loaded into the destination table. However, when I modify an existing document in SAP and rerun the Copy Job, it fails with the following error: Error: SqlColumnNameNotExist: Column '__rowMarker__' does not exist in the target table '[##InterimTable_db7db620-2727-4b15-8095-e5aab4f179f8]'. Has anyone encountered this issue with Microsoft Fabric SAP Table Copy Jobs while using incremental loading with the ABAP Add-on? Any suggestions on how to resolve this issue would be appreciated. I have attached a screenshot of the error for reference. Any help in identifying the root cause and resolving this issue would be appreciated. Regards, DarshanWorkspace Private Link Copy data activity to Lakehouse error (DeniedByPolicy)
I have a workspace level Private link configured and have a lakehouse on this workspace. Designed a pipeline that connects to a source SQL server DB and copy data activity that copies over the data into lakehouse in the workspace level private link. I am the admin on the workspace. The pipeline run on the same workspace. The issue that I am encountering is that when I try copying the data my copy activity errors out "You do not have enough permission to access the Lakehouse." The workspace level private link was configured following microsoft document and guidlines. The workspace is accessible only from internal network as public access is blocked. Are there additional checks and configuration neccessary? How do i resolve this error? What additional checks needs to be done?
Relationships disappearing after refreshing in my semantic model
Hi all, I have a semantic model in Fabric that connects to my sql endpoint of a Lakehouse. The semantic model has all of the tables stored in import mode. Now when I refresh schema and data, all of my relationships disappear. When I only refresh the data, it works fine and the relationships remain intact, but a column that I added in my MLV's will not be added. How is this possible and does anyone know a method to fix this? Furthermore, I have not changed the tables and datatypes of the columns at all, I only added one test column in one specific dimension. But for every fact and dimension the relationships will dissappear. I edit the semantic model within the service environment. I have already turned off these settings: Thank you in advance!
How do I permanently delete OneLake soft-deleted data right away to stop paying for it?
According to the OneLake soft delete documentation , deleted files stay in soft delete for 7 days and are billed at the same rate as active data. The page explains how to restore these files (Azure Storage Explorer, PowerShell or REST API) but not how to delete them permanently before the 7 days are up. I don't need to recover this data and would like to avoid paying for it. Is there a way to permanently delete soft-deleted files immediately (Fabric portal, Storage Explorer, PowerShell or REST API)? Can soft delete be turned off or shortened for a workspace, item or folder? The docs briefly mentioned configurable OneLake soft delete, but that change was later reverted. Is this feature planned? If I delete the whole item or workspace, is the data still billed for 7 days? Thanks!
Best practice for managing OAuth Client Secrets in Fabric Pipelines without Azure Key Vault
Hi, I am currently building a Microsoft Fabric Pipeline that retrieves data from the Marketo REST API using OAuth authentication. Since OAuth authentication requires a Client ID and Client Secret, I am looking for the recommended and secure way to manage the Client Secret in Microsoft Fabric. I understand that one common approach is to store the Client Secret in Azure Key Vault (AKV) and retrieve it during the Pipeline execution. However, before introducing AKV, I would like to understand whether Microsoft Fabric provides a native way to securely manage credentials or secrets for this type of scenario. Specifically: Is there a recommended way to securely store and use an OAuth Client Secret within Fabric, for example through Connections, Credentials, Workspace settings, or another Fabric feature, without using Azure Key Vault? For a Fabric Pipeline calling an external REST API such as Marketo, is Azure Key Vault considered the recommended or best-practice approach for managing the Client Secret? If both approaches are possible, what are the main security and operational differences between using Fabric-native credential management and Azure Key Vault? Are there any Microsoft documentation or reference architectures that describe the recommended approach for this scenario? I would like to understand whether introducing Azure Key Vault is necessary for this use case, or whether there is a secure Fabric-native approach that would be more appropriate. Any advice or examples from similar implementations would be greatly appreciated. Thank you!
