Dataflow Gen2 refresh extremly slow (compared to Gen1)

[JayJay11]

Hi all,

performing various tests using Fabric, we are encountering different issues. The topic of this post is that it seems that Gen2 Dataflows are much slower when refreshing than the exact same queries run in a Gen1 Dataflow (PPU workspace), see the screenshots.

We are using a Fabric Trial capacity, can this be issue, i.e. is this maybe a sort of shared capacity in the background?

[v-cboorla-msft]

Hi @JayJay11 

Thanks for usng Microsoft Fabric Community.

Apologies for the inconvenience.

Here are some possible reasons and solutions for your problem:

Dataflow Gen2 has some limitations. For example, it does not support Data Factory Fast Copy, duration and binary columns, or spaces or special characters in column or table names. These limitations may affect the speed and efficiency of your dataflows.

Please refer to the documentation for more details.

Dataflow Gen2 may require some optimization. You can use the refresh statistics to analyze and improve the performance of your dataflows. For example, you can check the source and destination throughput, the partitioning strategy, the staging option, and the query folding level.

Please refer to the documentation for additional information.

No, Microsoft Fabric Free Trial Capacity is not a sort of shared capacity. It is a dedicated capacity that you can use for 60 days to test out the Fabric experiences with 64 capacity units (CUs) of throughput.

For more details about Microsoft Fabric Free Trail please refer this document.

If the issue still persists please do let us know. Glad to help.

I hope this information helps. 

Thanks.

[JayJay11]

To add here: I learned from another thread that disabling staging the query in the dataflow significantly increases speed of refreshes (ca. halving refresh time) (given that there is a destination for the queries). Still slower than Gen1 though.

[v-cboorla-msft]

Hi @JayJay11 

Apologies for the delay in response and the inconvenience that you are facing here.

Appreciate if you could share the feedback on our feedback link , which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.
Feedback Link : Home (microsoft.com)

Hope this helps. Please let us know if you have any further questions.

Thanks.

[JayJay11]

Thank you for your reply. Where can I see the refresh statistics you mentioned (not in the downloadable CSV, not in the refresh history)?