rss.livelink.threads-in-node

Best Practices for Handling Incremental Data Loads in Dataflow Gen2

codeautomation — Fri, 18 Sep 2026 15:26:56 GMT

Hi everyone,

I am exploring different approaches for handling incremental data loads with Dataflow Gen2 in Microsoft Fabric.

For larger datasets, refreshing the entire dataset every time can become inefficient, so I am interested in understanding how others are designing their dataflows to process only new or changed records.

A few questions:

How are you identifying and filtering changed records between dataflow runs?

Is it better to manage incremental logic directly inside Dataflow Gen2, or use a Fabric pipeline to control the process?

How do you handle failed runs or partially processed data without creating duplicate records?

Are there any recommended patterns for maintaining good performance as the volume of historical data grows?

I would appreciate hearing about approaches that have worked well in real Fabric environments.

Office 365 Outlook activity cannot bind Workspace Identity or Service Principal connection

vicenteajm — Tue, 15 Sep 2026 23:47:21 GMT

Hi Fabric Community,

I am testing the Office 365 Outlook / Office 365 Email activity in Microsoft Fabric Data Factory for a CI/CD scenario.

According to the current Microsoft Learn documentation, the Office 365 Outlook activity supports User Authentication OAuth, Workspace Identity WI, and Service Principal SPN.

I tested both Service Principal and Workspace Identity.

The Fabric workspace has an active Workspace Identity.

The corresponding Microsoft Entra application and service principal exist.

The Office 365 Email / MicrosoftOutlook cloud connection can be created successfully.

The connection is visible in Manage connections and gateways.

From the pipeline Outlook activity, the connection is also visible through Browse all.

The issue is that when I select the WI or SPN connection from Browse all, Fabric shows the loading indicator but does not bind the connection to the Outlook activity.

No error is displayed.

The problem occurs before pipeline execution, so this is not a Microsoft Graph Mail.Send runtime error.

Expected behavior:

The selected Workspace Identity or Service Principal connection should be associated with the Office 365 Outlook activity.

Actual behavior:

The connection is visible and discoverable, but selecting it does nothing and the activity remains without a connection.

This blocks our CI/CD scenario because we want to avoid user OAuth connections that must be recreated or rebound between DEV, TEST and PROD.

I also noticed a documentation inconsistency.

The current Office 365 Outlook activity documentation explicitly says that Workspace Identity and Service Principal are supported.

However, the general Service Principal support page does not currently list Outlook as a supported data source.

The general Workspace Identity support page also does not list Outlook among the supported data sources.

Can Microsoft confirm whether Workspace Identity and Service Principal support for the Office 365 Outlook activity is fully GA, under staged rollout, or currently affected by a known issue?

Microsoft Learn references:

Office 365 Outlook activity

learn.microsoft.com/en-us/fabric/data-factory/outlook-activity

Service Principal support in Data Factory

learn.microsoft.com/en-us/fabric/data-factory/service-principals

Workspace Identity support in Data Factory

learn.microsoft.com/en-us/fabric/data-factory/workspace-identity

I can provide additional diagnostics or tenant details privately if required.

Thanks.

Configure Dataflow Gen2

UncleLewis — Tue, 15 Sep 2026 20:31:27 GMT

Hi,

Using DataFlow Gen2
I don't see a gear icon anywhere near default destination so I can configure

What am I missing?
Thanks,
w

Bug: QuickBooks connector (QuickBooks for Pipeline) fails OAuth login

Vitaliy3 — Fri, 11 Sep 2026 06:19:22 GMT

When attempting to create a connection of type QuickBooks for Pipeline (Cloud category) via Manage Connections and Gateways in Microsoft Fabric, with the authentication method automatically set to OAuth 2.0, clicking the Edit credentials button results in an error when trying to initiate the OAuth login flow.

Error message:

Unable to start OAuth login for this data source. Connection string is not supported.

Details: Query contains unsupported function. Function name: QuickBooksForPipeline.Contents

Session Id: 6fb86dae-dc71-4a06-8f7f-b99aee7885df

RequestId: 710bfcb8-ff7a-444d-90dc-c2dcc311e0f9

Cluster URI: https://api.powerbi.com

Status code: 400

Time: Fri Sep 11 2026 07:54:51 GMT+0300 (Eastern European Summer Time)

Steps to reproduce:

In a Fabric workspace, open a Copy Data activity (Source) or go to Manage Connections and Gateways.

Click "+ New" to create a new connection.

Select connection type Cloud.

Search for and select QuickBooks for Pipeline as the data source.

Authentication method is automatically set to OAuth 2.0.

Click Edit credentials.

The error above appears instead of the expected QuickBooks OAuth consent screen.

Expected behavior: Clicking Edit credentials should open the standard Intuit OAuth 2.0 consent screen, as described in the QuickBooks connector documentation for Data Factory / Fabric pipelines.

Impact: It is currently impossible to create a working QuickBooks Online connection in a Fabric pipeline using this connector, which blocks migrating an existing pipeline (where tokens are currently passed manually as parameters) to native OAuth authentication.

Environment: Microsoft Fabric (trial capacity), browser: Chrome, date: 2026-09-11.

SAP ECC into a Microsoft Fabric Lakehouse using Fabric Data Factory.

Murtaza_Ghafoor — Tue, 08 Sep 2026 13:40:48 GMT

We are currently extracting data from SAP ECC into a Microsoft Fabric Lakehouse using Fabric Data Factory.

*SAP ECC side:* We are using the SAP .NET Connector (NCo), which communicates with SAP through RFC calls.

*Fabric side:* We are using the SAP Table – Application Server connector, which leverages the Open Data Gateway (ODG) to retrieve data from SAP ECC.

*Small tables:* Data extraction and loading work successfully with relatively small SAP tables.

*Large tables:* When extracting larger tables, the process fails with a "Not enough space" error.

*Partitioning / Chunking:* We have tried the partitioning and RFC table options available in the Fabric connector to split large-table extraction into smaller chunks/batches based on indexed fields.

*Performance issue:* Although chunking allows us to process the data, the resulting throughput is very low, and extracting large tables takes an excessive amount of time.

*Key Challenge*

We need to identify a scalable and performant approach for extracting large SAP ECC tables into Microsoft Fabric Lakehouse, while avoiding the "Not enough space" error and significantly improving extraction throughput.

In particular, we need to determine whether the bottleneck is related to SAP RFC/NCo processing, Open Data Gateway, Fabric's SAP Table connector, partitioning strategy, batch size, or the underlying data transfer mechanism, and what configuration or architecture would provide better performance for large-volume SAP extractions.

Fabric Monday 121: Human in The Loop

DennesTorres — Mon, 07 Sep 2026 13:22:18 GMT

🚨 A Business Event can start a Fabric pipeline automatically. But should every event be allowed to complete the process without a human decision?

Business Events turn important data conditions into governed, reusable signals. The new Approval activity puts a human decision directly inside a Fabric Data Factory pipeline.

Combine them, and you get an event-driven workflow that stays automatic until judgment is actually required.

📡 FROM DATA TO A BUSINESS EVENT

In this video, a Fabric notebook checks incoming sales records for orders with a quantity greater than 15.

When it finds one, it publishes a `HighQuantityOrderDetected` Business Event containing the invoice, item, quantity, and total value.

The event uses a registered schema, so publishers and consumers exchange the same business context in a predictable format.

⚡ FROM BUSINESS EVENT TO PIPELINE

Activator monitors the Business Event in Real-Time Hub.

For each matching event, Activator starts a Fabric pipeline and passes the event fields into pipeline parameters. The pipeline can then enrich the request with additional data before asking for approval.

👤 HUMAN IN THE LOOP

The Approval activity pauses the pipeline and sends the request to a designated reviewer through Outlook or Microsoft Teams.

The reviewer sees the business context, opens the Review experience in Monitoring Hub, and chooses whether to approve or reject the request.

👉 Approved — the pipeline continues through the success path
👉 Rejected — the pipeline continues through the failure path
👉 Timeout — the activity fails and follows the rejection path

After the decision, the pipeline resumes automatically.

💾 KEEPING THE DECISION

Both branches call a notebook that records the result in the Lakehouse.

The saved record includes the event values, approval result, pipeline run ID, and decision timestamp. The original signal, the human decision, and the downstream action remain connected and traceable.

🎯 WHY THIS COMBINATION MATTERS

Business Events remove the need for every producer and consumer to build a private integration contract.

The Approval activity adds a controlled decision point without moving the process outside Fabric.

Together they enable workflows that are:

👉 Event-driven — the process starts when a meaningful business condition occurs
👉 Context-aware — the event payload travels into the approval request
👉 Governed — only the step that needs judgment is paused
👉 Auditable — the pipeline records who decided, the outcome, and what happened nextreti

This is where real-time detection becomes an operational business process: automatic detection, human judgment, and an automated response in one Fabric workflow.

⚠️ The Business Events alert integration and the Approval activity shown in this video are currently in Preview.

Sharepoint lists mirroring issue

Gllme — Fri, 04 Sep 2026 10:01:19 GMT

Hello,

We currently face an issue with sharepoint lists mirroring in microsoft fabric and we have no clue what can happen because the error message is pretty useless :

In fact, we are mirroring more than 20 lists but only half of them succeed. The others failed with the following error :

Internal system error occurred. ArtifactId: daa35143-...-5aa6accfec68, SequenceNumber: 9

I have linked an eventhouse but the error message is still the same, no more information:

Internal system error occurred. ArtifactId: daa35143-...-5aa6accfec68, SequenceNumber: 9

How can I have more information about the issue ?

Thank you very much for your help,

Regards

Sharepoint lists mirroring issue

Gllme — Fri, 04 Sep 2026 09:29:51 GMT

Hello,

We currently face an issue with sharepoint lists mirroring in microsoft fabric and we have no clue what can happen because the error message is pretty useless :

In fact, we are mirroring more than 20 lists but only half of them succeed. The others failed with the following error :

Internal system error occurred. ArtifactId: daa35143-...-5aa6accfec68, SequenceNumber: 9

I have linked an eventhouse but the error message is still the same, no more information:

Internal system error occurred. ArtifactId: daa35143-...-5aa6accfec68, SequenceNumber: 9

How can I have more information about the issue ?

Thank you very much for your help,

Regards

Faulting application FabricPipelineWorker.exe

ThomasPluemper — Thu, 03 Sep 2026 08:40:03 GMT

Hello,
we installed an On-premise data gateway version 3000.330.1 (August 2026) on a Windows Server 2025.

Every time we connect to SAP over SAP Table Application Server to our SAP System on SAP Cloud, we get the following error:

Application: FabricPipelineWorker.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.InvalidOperationException
at Microsoft.DataTransfer.Common.ProcessManagement.Communication.MessageHandlerSet.Dispatch(Microsoft.DataTransfer.Common.ProcessManagement.Communication.IMessageChannel, Microsoft.DataTransfer.Common.ProcessManagement.Communication.Message)
at Microsoft.DataTransfer.Common.ProcessManagement.Communication.MultiplexingMessenger.OnMessageToOtherChannel(Microsoft.DataTransfer.Common.ProcessManagement.Communication.IMessageChannel, MessageToOtherChannel)
at Microsoft.DataTransfer.Common.ProcessManagement.Communication.MessageHandlerSet.Dispatch(Microsoft.DataTransfer.Common.ProcessManagement.Communication.IMessageChannel, Microsoft.DataTransfer.Common.ProcessManagement.Communication.Message)
at Microsoft.DataTransfer.Common.ProcessManagement.Communication.IMessengerExtensions+<WaitForAsync>d__0`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at Microsoft.DataTransfer.Common.ProcessManagement.Communication.IMessengerExtensions+<WaitForAsync>d__1`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at Microsoft.DataTransfer.Common.ProcessManagement.RemoteService.RemoteServiceEnvironment+<DisposeServiceStubsAsync>d__4.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at Microsoft.DataTransfer.DataMovementGateway.WorkerProcess.WorkerContainer+<StartAsync>d__11.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at Microsoft.DataTransfer.DataMovementGateway.WorkerProcess.Program+<Main>d__1.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at Microsoft.DataTransfer.DataMovementGateway.WorkerProcess.Program.<Main>(System.String[])

AND:

Faulting application name: FabricPipelineWorker.exe, version: 5.68.9706.2, time stamp: 0x900a871d
Faulting module name: KERNELBASE.dll, version: 10.0.26100.33158, time stamp: 0x4332201f
Exception code: 0xe0434352
Fault offset: 0x00000000000c75fa
Faulting process id: 0x2234
Faulting application start time: 0x1DD3B1901C52B57
Faulting application path: C:\Program Files\On-premises data gateway\FabricIntegrationRuntime\5.0\Shared\FabricPipelineWorker.exe
Faulting module path: C:\WINDOWS\System32\KERNELBASE.dll
Report Id: a3bc643f-227b-4f9e-b7b7-45ff005eb05f
Faulting package full name:
Faulting package-relative application ID:

Has anyone an idea?

BR Thomas

Fabric Pipeline stuck on Loading - pipeline editor does not open

NaraayaPC — Mon, 31 Aug 2026 01:22:35 GMT

Hi, I am facing an issue with Microsoft Fabric Data Pipelines. The pipeline editor remains stuck on Loading and the activities/canvas never appear. The issue happens in multiple workspaces in the same tenant. I also created a new pipeline in another Fabric trial workspace, but it has the same issue. Lakehouse is working normally, but Pipeline remains stuck on Loading. I have tried refreshing and reopening Fabric and testing in another workspace, but the issue persists. Is anyone else experiencing this issue? Could this be a Fabric service-side issue? I have attached screenshots showing the Loading screen. Thanks

A bug in the Data pipeline expression builder

tarektm17 — Mon, 31 Aug 2026 00:28:37 GMT

Since last week's Fabric update, the Expression Builder is generating incorrect expressions when selecting pipeline parameters. For example, instead of generating:

It writes @linkedService().<parameter_name> instead of @pipeline.parameters.<parameter_name>.

The expression is immediately flagged with an "UnsupportedConcept" error. Additionally, the value cannot be saved even after manually correcting the expression to the proper pipeline parameter syntax. The issue occurs with both newly created and existing parameters, suggesting a regression introduced in the latest Fabric update.

[BUG] Dataflow Gen2 destination picker 404 — api.powerbi.com lakehouses UK region

sahillihas20 — Wed, 26 Aug 2026 23:16:25 GMT

Hi community,

We have a complete blocker with Dataflow Gen2 (CI/CD) destination configuration in our UK-region Fabric tenant.

The Error:
When clicking "Add default destination" and selecting a Fabric Lakehouse from the OneLake catalog, this error fires immediately for ALL users including the
workspace Admin:
DataSource.Error: Downstream service call to url 'https://api.powerbi.com/v1/workspaces/{WORKSPACE_ID}/lakehouses' failed with status code 404. (Region: uk)

The Lakehouse IS visible in the OneLake catalog — it appears in the picker. The 404 fires the moment it is clicked.

Root cause hypothesis:
The destination picker calls the legacy Power BI REST API: api.powerbi.com/v1/workspaces/{id}/lakehouses ← returns 404

For Fabric-native workspaces in the UK region, the correct endpoint should be: api.fabric.microsoft.com/v1/workspaces/{id}/lakehouses

The OneLake catalog listing (which uses a different path) resolves the lakehouse correctly — it's only the destination configuration call that uses the wrong endpoint.

Workarounds attempted — all failed:
1. UI — Admin user attempted same steps → same 404
2. queryMetadata.json via Git — Added destination fields, pushed to prod branch → Fabric silently strips loadEnabled/destination on every sync, creating a permanent conflict loop (workspace ≠ Git on every update)
3. Authentication methods — Before the error began firing immediately, a "Connect to default data destination" screen appeared offering two authentication options for the Lakehouse connection:
- Organizational account (Sahil Singh / user account)
- Service principal

Both were attempted. Neither succeeded — the 404 followed in both cases.

Subsequently, this connection credentials screen stopped appearing altogether. The 404 now fires immediately upon clicking lh_dhl_prod in the OneLake catalog, before the connection setup dialog is even shown. This regression suggests the failure point has moved earlier in the destination configuration flow.

Questions:
1. Is this a known issue for UK-region tenants?
2. Is there a way to configure the Dataflow Gen2 destination via Fabric REST API directly to bypass the broken UI?
3. Are there other file-based approaches (mashup.pq metadata) for CI/CD Dataflow Gen2 destinations?

Happy to provide Session IDs and further details to the Fabric team.

I have attached the relevant images, PFA...

Thanks

Fabric Pipeline Lakehouse sink fails across multiple workspaces with invalid quoted UTF-8 charset

ShayneY75 — Tue, 25 Aug 2026 13:30:05 GMT

We are unable to use Microsoft Fabric Pipelines to copy data from an on-premises SQL Server into either the Tables or Files area of a Fabric Lakehouse.

The SQL Server source is reached successfully, authentication succeeds, the query executes, and the Pipeline reads and copies the returned rows. However, the Lakehouse sink fails before writing any data or creating a file.

The consistent error is:

ErrorCode=LakehouseOperationFailed

The character set provided in ContentType is invalid.
Cannot read content as string using an invalid character set.

'"utf-8"' is not a supported encoding name.
For information on defining a custom encoding, see the documentation
for the Encoding.RegisterProvider method.

Parameter name: name

The quoted encoding value "utf-8" does not appear anywhere in the Pipeline JSON configuration. The exception appears to be raised when the Microsoft Data Transfer client processes a response during the Lakehouse write operation.

Reproduction steps

Create a new Fabric workspace assigned to the F2 capacity.
Create a new Lakehouse in that workspace.
Create a Fabric Data Factory Pipeline.
Add a Copy Data activity.
Use the existing SQL Server Database connection through the on-premises gateway.
Use SELECT 1 AS TestID as the source query.
Configure the destination as Bronze_LH_1/Files/Test1/TestID.
Run the Pipeline.
The source row is read successfully, but the Lakehouse sink fails before writing the file.

Full Pipeline Error

Code: 27370

ErrorCode=LakehouseOperationFailed,
Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,
Message=Lakehouse operation failed for: The character set provided in
ContentType is invalid. Cannot read content as string using an invalid
character set.

Source=Microsoft.DataTransfer.ClientLibrary,

Type=System.InvalidOperationException,
Message=The character set provided in ContentType is invalid.
Cannot read content as string using an invalid character set.
Source=mscorlib,

Type=System.ArgumentException,
Message='"utf-8"' is not a supported encoding name.
For information on defining a custom encoding, see the documentation
for the Encoding.RegisterProvider method.

Parameter name: name
Source=mscorlib

Troubleshooting Already Completed

The following tests have already been performed:

Confirmed Microsoft Fabric is enabled in tenant settings.
Confirmed users can create Fabric items.
Confirmed the workspace is assigned to an active Fabric F2 capacity.
Confirmed the capacity is in the correct region
Confirmed the SQL Server Database connection is Online and Ready.
Confirmed the on-premises data gateway shows no warnings.
Confirmed Windows authentication credentials are valid.
Successfully executed and read results from SELECT 1 AS TestID.
Confirmed the Pipeline read and copied one row from SQL Server.
Tested a source view named [DataWarehouse].[BI_vw].[AccountTypeNames].
Reduced the source query to the literal query SELECT 1 AS TestID.
Tested a Lakehouse Table destination.
Tested a Lakehouse Files destination.
Tested the default dbo destination schema.
Tested a custom BI_vw destination schema.
Tested the Copy Job assistant.
Tested a manually configured Pipeline Copy Data activity.
Created new Lakehouses and repeated the test.
Created an entirely new Fabric workspace and repeated the test.
Reproduced the identical error in both the original and new workspaces.
Successfully uploaded a file manually through the Lakehouse browser interface.
Confirmed manual OneLake/Lakehouse file access works.
Confirmed no Pipeline JSON property specifies charset="utf-8".

Findings

The testing indicates:

On-premises gateway connectivity: Successful
Windows authentication: Successful
SQL Server query execution: Successful
Source data read: Successful
Row transfer from source: Successful
Manual Lakehouse file upload: Successful
Pipeline Lakehouse sink write: Failed

The issue is reproducible across multiple workspaces, Lakehouses, destination types, and source queries. The original workspace’s history as a Power BI workspace was eliminated as a cause by reproducing the issue in a newly created Fabric workspace.

Workspace Identity works in one workspace but fails in another when executing Notebook Activity

santiago-marano — Thu, 20 Aug 2026 00:12:06 GMT

Hi everyone,

I'm troubleshooting an issue with Notebook Activity + Workspace Identity (WI) in Microsoft Fabric Data Factory and would like to know if anyone has experienced something similar.

Scenario

I'm trying to execute a notebook from a pipeline using a Workspace Identity connection.

The notebook itself is very simple and does not require:

Lakehouse
Environment
External connections

The execution fails before Spark starts.

Notebook Error

An unexpected error encountered while preparing spark operation execution.

The user access token was not found.
The user session may have expired or the user is no longer valid.
Please sign in again.

Additional observations:

In the notebook run history, the "Submitted By" column appears as "--"
The notebook never reaches Spark execution

Pipeline Error

Notebook execution failed at Notebook service with http status code - '200'

Failed to create session for executing notebook.

Notebook execution is in Failed state.

What I have already validated

✅ Workspace Identity exists and is active in Entra ID

✅ Workspace Identity is included in the Security Group allowed to use Fabric APIs

✅ Tenant setting "Service principals can call Fabric public APIs" is enabled

✅ Workspace Identity is Workspace Admin

✅ Workspace Identity is successfully used to access Azure Data Lake Storage Gen2

✅ Notebook Activity is configured to use a Workspace Identity connection

✅ Notebook owner account is active and valid

✅ Pipeline owner account is active and valid

✅ No Environment attached

✅ No Lakehouse attached

✅ No external connections inside the notebook

✅ Created a brand-new notebook containing only:

print("hello")

✅ Created a brand-new pipeline containing only a Notebook Activity

Same result.

✅ The same Notebook Connection with Workspace Identity works successfully in another workspace

Comparison with another workspace

I have another workspace with:

The same tenant
Different Fabric capacity
The same geography
Different Security Group
The same Fabric API tenant settings
The same Notebook Connection owner

and the exact same configuration works correctly there.

Additionally:

The same Notebook Connection was tested in both workspaces
The same Workspace Identity pattern is used
The working workspace can access Key Vault and execute notebooks through pipelines without issues

The only variable that seems to change is the workspace itself.

Additional Observation

The Workspace Identity appears to be functioning correctly because it is already being used successfully to authenticate against Azure resources such as ADLS Gen2.

Because of this, the issue does not appear to be related to Entra permissions, Fabric API permissions, or Workspace Identity access to Azure resources.

Instead, it seems related to one of the following:

Notebook Activity + Workspace Identity integration
Session creation
Workspace metadata
Workspace-specific configuration

Question

Has anyone encountered a situation where:

Workspace Identity authentication works for ADLS and Key Vault
The same Notebook Connection works in another workspace
Notebook Activity only fails in one specific workspace
The error is related to:
- "The user access token was not found"
- "Failed to create session for executing notebook"
- Empty "Submitted By" field

I'm trying to determine whether this is:

A workspace-specific metadata issue
A Notebook Activity bug
A Workspace Identity provisioning issue
An undocumented prerequisite for Workspace Identity execution

Any insights would be greatly appreciated.

Thanks!

Mirroring SQL Endpoint Capacity Usage

FabricEnjoyer — Mon, 17 Aug 2026 11:19:29 GMT

Hi All,

I've been working extensively with the Open Mirroring Database item in Microsoft Fabric, but I'm still trying to fully understand how its capacity (CU) consumption is calculated.

From my understanding, the internal operations include components such as:

OneLake Write via Redirect
OneLake Iterative Read via Proxy
OneLake Read via Redirect
OneLake Read via Proxy
OneLake Write via Proxy
OneLake Other Operations

My assumption is that these workloads account for activities such as data ingestion, CSV-to-Parquet conversion, data maintenance, vacuum operations, and other background processing required to keep the mirrored database operational.

What I do not understand is the Warehouse SQL Endpoint Query Usage metric.

On 11 August, this metric increased dramatically in our environment and has remained significantly elevated ever since. For one particular Open Mirroring Database item, usage increased from approximately 7K CU to 34K CU, and I am trying to determine what contributes to this workload and why it changed so suddenly.

Before: Screenshot

After: Screenshot

Another observation is related to the Run By filter in the Capacity Metrics App.

Prior to 10 August, I was able to filter consumption by individual user identities and clearly distinguish activity generated by specific users from activity generated by the system.

However, from around 10/11 August, all user-generated activity appears to be grouped under a generic "User" identity, and individual users can no longer be distinguished or filtered.

Before 10 August: Screenshot showing identifiable users

After 10 August: Screenshot showing only "User"

I don't understand what this new User identity represents. Is it an aggregation of all user-generated activity, a new classification introduced by the metering changes, or something else entirely?

My current theory is that both the increase in Warehouse SQL Endpoint Query Usage and the change in the Run By filter may be related to the recent Fabric capacity metering changes, but I have not been able to find any documentation that clearly explains this behavior.

Can anyone provide insight into the following?

What specifically contributes to Warehouse SQL Endpoint Query Usage for an Open Mirroring Database?
Do any of the internal mirroring processes execute through the SQL Endpoint and therefore appear under this metric?
Did the recent Fabric metering changes alter how these operations are classified or billed?
What exactly does the "User" identity represent in the Run By filter?
Why did individual user identities disappear after 10 August and become a generic "User" value?
Is the new User Identity dimension connected to the recent metering changes?
Has the attribution model for user-generated workloads changed, resulting in individual users being grouped into a single category?

Any clarification would be greatly appreciated, as these changes make it difficult to understand capacity consumption patterns and compare usage before and after 10 August.

Thanks!

iifnull() failing on non-null input

cmjcf — Mon, 17 Aug 2026 03:40:08 GMT

The documentation for iifNull() says this:

Returns the first not null item when given two or more inputs.

We have a dataflow which handles the contents of a number of files, using a specified CSV format. One of the columns is named 'Dealer'. Due to an error in an upstream system, a significant number of files came in with the header misspelled 'Delaer'. There are so many of these files that we cannot simply edit them manually to fix. I have the following expression in a Derived Column transform:

iifNull('Dealer',byName('Delaer'))

I understand this expression to mean "Use the value of the column 'Dealer' if it is not null, else use the value of the drifted column 'Delaer'." (That name is not in the list set by the dataset, so it is drifted, and leaving out byName() causes validation to fail.)

When the file does not contain the spelling error, my expected result is that it simply retains the correct value in 'Dealer' because this is not null. However, when actually run, this expression actually returns null, and causes the column in the Derived Column transform to not be created, resulting in a "Resolved attribute(s) [name] missing from [names]" error.

Passing the Parameter while calling the Data Pipeline using REST API

12_shashank — Wed, 12 Aug 2026 04:31:22 GMT

Hello Readers,

I am trying to trigger the REST API to trigger an on-demand job on my datapipeline:
POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/dataPipelines/{pipelineId}/jobs/execute/instances

Body/Payload:

{

"parameters": [

{

"name": "filed_based_run",

"value": true,

"type": "Boolean"

{

"name": "fail_whole_load",

"value": true,

"type": "Boolean"

{

"name": "config_file",

"value": "on_demand",

"type": "Text"

}

]

}

Error:
Status Code: 403 Forbidden
Error Message:

{

"requestId": "cc5e44f7-cda1-4adc-9e18-ef19513b15d5",

"errorCode": "FeatureNotAvailable",

"message": "Parameter is not allowed for this item type or this item job type",

"isRetriable": false

}

My question : Why we can't pass the parameter while calling the datapipeline and is there another I can do this like using Power Automate or Logic apps?

nested Array Issue in Mapping the key Values Part

ShayanSiddiqueI — Mon, 10 Aug 2026 14:28:02 GMT

I have a nested list of the array within the API Response in the Azure Data Factory. When I am mapping it with the column its is showing me null value instead of giving me the actual value which exist. Here rthe Product List Part part of the reposnse that I wanted to map

"products": [
{
"id": 1,
"catalogItem": {
"id": 1,
"identifier": "1",
"_info": {
"catalog_href": "https://example.com/1"
}
},
"description": "1",
"sequenceNumber": 1,
"quantity": 1,
"price": 1,
"cost": 1,
"agreementAmount": 1,
"billableOption": "1",
"agreement": {
"id": 1,
"name": "1",
"_info": {
"agreement_href": "https://example.com/1"
}
},
"locationId": 1,
"businessUnitId": 1,
"taxableFlag": false,
"dropshipFlag": false,
"specialOrderFlag": false,
"phaseProductFlag": false,
"cancelledFlag": false,
"quantityCancelled": 1,
"customerDescription": "1",
"productSuppliedFlag": false,
"invoice": {
"id": 1,
"identifier": "1",
"_info": {
"invoice_href": "https://example.com/1"
}
},
"warehouseId": 1,
"warehouseBinId": 1,
"calculatedPriceFlag": false,
"calculatedCostFlag": false,
"warehouse": "1",
"warehouseBin": "1",
"company": {
"id": 1,
"identifier": "1",
"name": "1",
"_info": {
"company_href": "https://example.com/1"
}
},
"productClass": "1",
"needToPurchaseFlag": false,
"minimumStockFlag": false,
"calculatedPrice": 1,
"calculatedCost": 1,
"_info": {
"lastUpdated": "2026-01-01T00:00:00Z",
"updatedBy": "1"
}
}
]

I am trying the catlogitem.id and the the Same Format you can see here but still cant get an data in the rows of the SQL Table How to refrenece it or how to flat it?

Copy Job does an Append for FULL Load, cannot be changed

ToddChitt — Fri, 07 Aug 2026 13:12:34 GMT

I played around with Copy Job when it first came out. Now I am trying to set one up for real. I'm starting with one table of about a million records that needs a Full load every night. The job has run three times and I have 3 million records. It is configured for FULL Load. In the Advanced settings, under Write settings, it is set to Append. Ah, THAT is the issue. But I cannot select anything else. Where is the OVERWRITE option? What am I missing? Do I have to manually TRUNCATE the destination tables before running the Copy Job?

I sort of assumed that a FULL LOAD implied overwrte.

Semantic Model Refresh failing in Pipeline but not manually

Jakob_H — Thu, 06 Aug 2026 08:58:19 GMT

Hello everyone,

we have a Fabric F8 environment with a pipeline that triggers multiple pipelines to load data into bronze, silver and gold layer. After the Gold pipeline is succeeded, a Semantic Model Refresh activity should refresh a Semantic Model in another workspace.

After it had worked for a few days, the pipeline now fails at the Semantic Model Refresh step. Restarting the pipeline from that step or triggering the Semantic Model refresh from a another pipeline or manually via the Refresh Button all succeed.

The error message states, that a column is missing in the rowset. The column is created immediately before the semantic model refresh starts.

What I have tried so far:

- adding a 15 minutes waiting time to solve any metadata refresh activities

- rebuild the whole pipeline from scratch

- adding an invoke pipeline step, that starts a pipeline which only contains the Semantic Model Refresh activity

None of that has solved the issue.

Any suggestion would be highly welcome.

Error message:

Semantic model refresh execution failed, error message received from semantic model refresh operation - '{"errorCode":"ModelRefresh_ShortMessage_ProcessingError","errorDescription":"Retry attempts for failures while executing the refresh exceeded the retry limit set on the request.\n0xC11200B0: The '<oii>is_anonym<\/oii>' column does not exist in the rowset.\n0xC112001C: The operation was canceled.. The exception was raised by the IDbCommand interface.\n0xC11C0006: The current operation was cancelled because another operation in the transaction failed.\n0xC112001C: The operation was canceled.. The exception was raised by the IDbCommand interface.\n0xC112001C: The operation was canceled.. The exception was raised by the IDbCommand interface.\n0xC14700F0: {\"RootActivityId\":\"46081B1B-9B40-4967-9402-297B0D45D710\"}"}'