Gen2 Dataflow failures when loading medium sized datasets
I've been trying to get a medium-sized (100,000 records for now) dataset to load from a SQL database (on prem and Azure) to a Fabric Warehouse from a Gen2 dataflow. I can get it to load 1000 or 10000 records, but once I step up to higher numbers, I get the error below. And yes--port 1433 is open. Also--I can load all the data no problem with a native Power BI dataset or a Gen1 dataflow. It's almost like the connection to the source database times out after a certain period of time. Mashup Exception Data Source Error Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Microsoft SQL: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.) Details: DataSourceKind = Lakehouse;DataSourcePath = Lakehouse;Message = A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.);ErrorCode = -2146232060;Number = 10054;Class = 20Dataflow Gen2 Issue
Hi, I'm trying to run a very simple Gen2 Dataflow from an on prem sql server. All setups fine but when I ask it to refresh I get: "Error: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: We don't support creating directories in Azure Storage unless they are empty." Thanks Ben
Lakehouse Add or Remove columns from table
Not sure if this is the right forum or not - but here is the issue. We are loading data to a Lakehouse using gen 2 data flows (for now they are just pointing at exisitn gen 1 dataflows then doing the lakehouse insert - we will recify this later on). Over time it is typical for columns to be added, removed and / or updated in a dataflow - with a datamart these changes are reflected automatically in the schema - however with a laehouse when adding a new column to the dtaflow i can see no way to bring that into the lakehouse. What do i need to do here - only options i can see are 1: import it as a new table but that seems to be very clunky as you would need to update queoroes / stored procedures on your sql end point to cater for this 2: Delete exisitng table in lakehouse and then add a new one with the same name. Am i missing something ?Fabric compute capacity exceeded its limits. How to know how much CU each dataflowgen2 consumes
Our dataflowgen2 got Failed status after exceeding the Fabric compute capacity limit At the moment, I'm trying to access Fabric Capacity Metrics to see if it can be viewed there but cant open it as well with message "Unable to load model due to reaching capacity limits" - How to know how much CU each activity(dataflowgen2,notebook,etc) in a datapipeline consume so we can avoid hitting the CU limit? - Do we have other way to view fabric capacity metrics?
Getting On-Prem data from SQL Server to Microsoft Fabric using Dataflow (gen2)
Hi, I am trying to get the On-prem data from SQL Server to fabric using data factory-> Dataflows(gen2). Below are the steps which I am following: 1. Hvae installed Gateway with latest version which is working fine. 2. Created one lakehouse. 3. Creating Dataflows(gen2) ->selected Get Data->SQL server database -> connecting with server and database using credentials->selected one table-> chose lakehouse in data destination -> go with Lakehouse(none) option.(attached in snapshot) 4.In choose destination target pane I am selecting the lakehouse which I have created earlier in step 2. And when I click on next I am getting error : Something went wrong while retrieving the list of tables.(atached in snapshot). So why I am getting this error? It would be great if you guide.
