[Azure Data Factory] Data Flow Sink is extremely slow
I'm copying about 800 items from SAP to Azure Fabric. I'm running pipeline with data flow in Azure Data Factory (ADF). We are using Self-hosted integration runtime. The Data Flow is not complex, it just copies data and adds an additional column. There are no complex transformations. However, the Sink activity takes around 10 minutes! Sink configuration: Sink type: dataset Dataset: Fabric warehouse Allow schema drift: true Allow insert: true Batch size: 100 Enable staging: false (I cannot turn it on; otherwise, I get an error in my pipeline: "Fabric DW supports staging only via copy command - EXE-0001, Dataflow cannot be analyzed as a graph.") I also tried using Copy activity as suggested here, but it keeps failing with a BULK insert error: Failure happened on 'Sink' side. ErrorCode=SqlOperationFailed, 'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException, Message=A database operation failed. Please search error to get more details., Source=Microsoft.DataTransfer.Connectors.MSSQL, ''Type=Microsoft.Data.SqlClient.SqlException, Message=Incorrect syntax near 'INSERT BULK'., Source=Framework Microsoft SqlClient Data Provider,' Some other observations: Sink processing time: 10m 33s Stage time: 7m 36s Table operations SQL duration: 2m 30s I also checked in the warehouse's long_running_queries and saw that INSERT was called 840 times, with each insert taking around 450ms. That means 800 * 450ms = 360,000ms (6 minutes)! My Question: Why is ADF inserting items one by one and ignoring the batch size of 100? Since I cannot enable Staging without errors and BULK insert fails, what is the correct configuration to load data from SAP to Fabric efficiently?
Training Resources for Manufacturing and Quality
Can anyone recommend any YouTube or other training resources for power bi reporting with a focus on maufacturing, technical and quality issues? Seems most of the training out there is geared towards sales and sales trending. Okay some of the senarios can be adapted but it would be easier for me to garner information where it relates more to what I am doing. Currently trying to do complaint trending using two different raw data sources, SAP and SQL (although with the SQL data I am only allowed to connect via access databases!). So having to merge the two lots of production data and skus etc. But would like to see ideal example of star schemas used for this type of trending, to give me some pointers. Currently using a denormalised table scraped from a report in access, along with data provided via launch bi.