[Azure Data Factory] Data Flow Sink is extremely slow
I'm copying about 800 items from SAP to Azure Fabric. I'm running pipeline with data flow in Azure Data Factory (ADF). We are using Self-hosted integration runtime. The Data Flow is not complex, it just copies data and adds an additional column. There are no complex transformations. However, the Sink activity takes around 10 minutes! Sink configuration: Sink type: dataset Dataset: Fabric warehouse Allow schema drift: true Allow insert: true Batch size: 100 Enable staging: false (I cannot turn it on; otherwise, I get an error in my pipeline: "Fabric DW supports staging only via copy command - EXE-0001, Dataflow cannot be analyzed as a graph.") I also tried using Copy activity as suggested here, but it keeps failing with a BULK insert error: Failure happened on 'Sink' side. ErrorCode=SqlOperationFailed, 'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException, Message=A database operation failed. Please search error to get more details., Source=Microsoft.DataTransfer.Connectors.MSSQL, ''Type=Microsoft.Data.SqlClient.SqlException, Message=Incorrect syntax near 'INSERT BULK'., Source=Framework Microsoft SqlClient Data Provider,' Some other observations: Sink processing time: 10m 33s Stage time: 7m 36s Table operations SQL duration: 2m 30s I also checked in the warehouse's long_running_queries and saw that INSERT was called 840 times, with each insert taking around 450ms. That means 800 * 450ms = 360,000ms (6 minutes)! My Question: Why is ADF inserting items one by one and ignoring the batch size of 100? Since I cannot enable Staging without errors and BULK insert fails, what is the correct configuration to load data from SAP to Fabric efficiently?