DataflowGen2 Slow Performance during design is HORRIBLE

[ToddChitt]

I have a Fabric Workspace that is currently under a Free Trial Capacity (F64). I am designing several Dataflow Gen2 objects and performance during the design process is abysmal! I am pulling data from SharePoint Excel files. To get from one simple step to another sometimes takes as much as 10 minutes, or times out entirely, or I get frustrated and start it again. FYI, the data in question is 2 columns and 1000 rows. By exiting and saving the DF every time, usually the next time it picks up where I left off and I can continue. Right now, I am trying to set up a Destination to a Warehouse in the same Workspace. I click on Add Destination, select Warehouse, then finally the warehouse in question, and it just hangs. Like 10 minutes or more. 

I have also tried uploading the file in question to a Lakehouse and creating a DF from there. Currently, it is trying to figure out the files in the Lakehouse (of which there are only 4 so far) and it has been "Loading..." for over 5 minutes. 

This is CRAZY. It is a HUGE IMPEDIMENT as Fabric simply cannot complete the simplest of tasks for me to get the work done. 

What is going on?

Yes, I have tried refreshing the browser, clearing the cache, even re-booting. Nothing helps.

[ToddChitt]

@v-kpoloju-msft 

Thanks for your reply. I am EXTREMELY curious about your statement: A Trial Capacity (at F64) and a purchased F64 Capacity are not the same. Do you have documentation to this fact? To be honest, it feels a little like dishonest advertising on the part of Microsoft to promise and F64 Capacity during the Free Trail, but not really deliver it.

Everything I have read, like this from MS: Microsoft Fabric concepts - Microsoft Fabric | Microsoft Learn says that it is the same. Please provide something to refute this.

Regarding the main issue: I did a side-by-side test with another workspace in a different tenant, also under Trial Capacity. For that, I was able to navigate the Excel file and the Lakehouse destination options just fine. During that development time, the original development efforts were hung up on simply navigating the list of Lakehouses available for a Destination. A second paralell test of the same functionality in a new Workspace in the SAME tenant tenant also produced good results. CONCLUSION: The Workspace is at fault. Side note on this Workspace: Every time I went to create a new Dataflow Gen2, it showed up like this:

("Enable Git integration option CHECKED, AND locked out.) To get around it, I always have to switch to a different Workspace in the same tenant, then back again and then I get this (which is what I want)

Those two items together lead me to believe there is something seriously wrong with the Workspace.

My response: Create a new Workspace and move on. I'm not that far into the development work that this is a major effort. 

[v-kpoloju-msft]

Hi @ToddChitt,

After thoroughly reviewing the details you provided, I have identified alternative workarounds that may help resolve the issue. Please follow these steps:

Although Git integration is locked, please create a new workspace without it and check if it functions properly. An admin should review the settings in the Fabric Admin Portal > Workspaces and compare them with a functional workspace.

Remove stored credentials by navigating to Fabric Home > Settings > Data Sources. Delete any existing credentials for SharePoint, Lakehouse, and Warehouse, then re-authenticate. Reconnect Data Sources by manually adding SharePoint, Lakehouse, and Warehouse connections again and verifying if they work.

In the Fabric Admin Portal > Capacity Metrics, check CPU and Memory Usage during periods of slow performance. If high usage is detected, schedule Dataflow work during low-usage hours.

If the issue persists after attempting all the workarounds, please consider raising a Microsoft support ticket. You can create a support ticket using the following link:
https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket
 
If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Thank you.

[v-kpoloju-msft]

Hi @ToddChitt,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-kpoloju-msft]

Hi @ToddChitt,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[ToddChitt]

I did some testing:

First, a parallel test of the one workspace against another tenant: same operation(s). The other tenant performed just fine. Second, I created a new Workspace in the same tenant (and under the same Trial Capacity) and THAT worked fine. My conculsion is that there was something about the workspace. 

Since I was very early on in the development process, I simply recreated the artifacts in the new workspace and have now deleted the one that was having performance issues. 

We meay never know what the cause of the slow performance was. As I said, the issues came around some very basic operations, like while setting up a Dataflow Gen2 destination, it would take 10 minutes to 'browse' the list of available lakehouses. 

But now I am EXTREMELY curous about your statement:

• Yes, that’s correct. A Trial Capacity (at F64) and a purchased F64 Capacity are not the same.

Everything I read from Microsoft documentation says that a Free Trial is the same as an F64 Capacty, just that you don't have to pay for it. Please provide documentation to support your statement.

[v-kpoloju-msft · Accepted solution]

Hi @ToddChitt,

I apologize for any inconvenience. Here are the two documents that explain Trail vs Premium. Please review the links below for a better understanding:

Trial Capacity (F64)

• Typically time-limited (e.g., 60 days).
• May have restrictions or limitations compared to a paid capacity.
• Designed for evaluation purposes.

Purchased F64 Capacity

• A fully paid, production-ready capacity.
• Includes service level agreements (SLAs) and comprehensive support.
• Ensures uninterrupted access as long as the subscription remains active.

While both options provide comparable compute power, please note that the trial version is temporary and may have certain limitations compared to the purchased capacity.

Please refer to the below links for better understanding:

Fabric trial capacity - Microsoft Fabric | Microsoft Learn
What is Power BI Premium? - Power BI | Microsoft Learn

Also, kindly refer the below link for more information:

Microsoft Fabric concepts - Microsoft Fabric | Microsoft Learn

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Thank you.

[v-kpoloju-msft]

Hi @ToddChitt,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-kpoloju-msft]

Hi @ToddChitt,

We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?

If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.

Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

Thank you for your understanding and participation.

[andrewsommer]

The only documented limitations of the trial capacity I have seen is the trials do not support autoscaling and they do not support AI skills. 

[v-kpoloju-msft]

Hi @ToddChitt,

Thank you for reaching out to the Microsoft Fabric Community Forum.

After reviewing the details you provided, I have identified a few workarounds that may help resolve the issue. Please follow these steps:

• Free Trial capacities (F64) have limited resources, which may affect performance. High usage of the workspace (by multiple users or processes) can lead to performance issues. Navigate to Fabric Admin Portal > Capacity Settings > Metrics. If usage is high, try working during off-peak hours. For extensive use, consider upgrading to a paid capacity (F2 or higher).
  Plan your capacity size - Microsoft Fabric | Microsoft Learn
• Instead of loading the file directly from SharePoint, please download it and store it in a Lakehouse first. Then, reference it in the Dataflow. If the file needs to be in SharePoint, ensure it is stored in OneDrive for Business rather than a SharePoint site, as ODFB offers better integration. Additionally, avoid unnecessary transformations in the Power Query Editor while designing the Dataflow.
• Instead of connecting Dataflow Gen2 directly to the Warehouse, follow these steps: First, load data into a Lakehouse. Then, use a Notebook or Pipeline to transfer data from the Lakehouse to the Warehouse. If you must use Dataflow Gen2, try creating a new, empty Dataflow and connect it solely to the Warehouse to check if the issue lies with the Dataflow creation itself.
• To enable fast copy, navigate to the appropriate Fabric endpoint, access a premium workspace, and create a Dataflow Gen2. On the Home tab, select Options, then go to the Scale tab and check the “Allow use of fast copy connectors” checkbox.
  Fast copy in Dataflows Gen2 - Microsoft Fabric | Microsoft Learn

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Best Regards.

[ToddChitt]

@v-kpoloju-msft 

• Are you saying that a Trial Capacity (at F64) is NOT the same as a purchased F64 Capacity?
• I tried downloading the file (360 KB) and then uploading into a Lakehouse in the same Workspace. I can get to it, but for the Dataflow to 'figure out' the contents and show me the available sheets and tables (of which there are less than 10 total), i.e. the [Content] step, it just hangs. Maybe, once it did finish reading it, simply showed an icon of the file in the space where data is usually displayed. I was not able to take any further actions. 
• For one of the Destinations, I wanted to send it to a Warehouse. The wizard will typicaly show me a list of Warehouses in the same Workspace. It does not, it just hangs. 
• For another DF, I was trying to load data from Dataflow Gen1. To expand the "Dataflows" node during initial design, it just spins.

[v-kpoloju-msft]

Hi @ToddChitt,

After reviewing the details you provided, I have identified alternative workarounds that may help resolve the issue. Kindly follow these steps:

• Yes, that’s correct. A Trial Capacity (at F64) and a purchased F64 Capacity are not the same.
• Use a Notebook or Data Pipeline to upload data from the Lakehouse to the Warehouse instead of Dataflow Gen2. Convert Excel files to CSVs for faster uploads. If using Excel, keep it as a Delta table in the Lakehouse before referencing in Dataflow.
• Load data into a Lakehouse table or Notebook (SQL or PySpark) to write to the Warehouse and create a new Dataflow Gen2 to check if the issue is Dataflow-related. Check connectivity in the Fabric Admin Portal under Capacity Metrics for throttling or high usage.
• Store Dataflow Gen1 output in a Lakehouse table as an alternative to Dataflow Gen2. Use a Notebook to retrieve Dataflow Gen1 results and transfer them to a Lakehouse or Warehouse. If necessary, create a new, simple Gen2 Dataflow to see if the issue is related to Gen1 connectivity.

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Best Regards.