Forum Discussion

JayJay11's avatar
JayJay11
Resolver II
2 years ago

Dataflow Gen2 refresh extremly slow (compared to Gen1)

Hi all,

 

performing various tests using Fabric, we are encountering different issues. The topic of this post is that it seems that Gen2 Dataflows are much slower when refreshing than the exact same queries run in a Gen1 Dataflow (PPU workspace), see the screenshots.

 

We are using a Fabric Trial capacity, can this be issue, i.e. is this maybe a sort of shared capacity in the background?

 

data engineering

4 Replies

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi JayJay11 

     

    Thanks for usng Microsoft Fabric Community.

    Apologies for the inconvenience.

    Here are some possible reasons and solutions for your problem:

    Dataflow Gen2 has some limitations. For example, it does not support Data Factory Fast Copy, duration and binary columns, or spaces or special characters in column or table names. These limitations may affect the speed and efficiency of your dataflows.

    Please refer to the documentation for more details.

    Dataflow Gen2 may require some optimization. You can use the refresh statistics to analyze and improve the performance of your dataflows. For example, you can check the source and destination throughput, the partitioning strategy, the staging option, and the query folding level.

    Please refer to the documentation for additional information.

    No, Microsoft Fabric Free Trial Capacity is not a sort of shared capacity. It is a dedicated capacity that you can use for 60 days to test out the Fabric experiences with 64 capacity units (CUs) of throughput.

    For more details about Microsoft Fabric Free Trail please refer this document.

     

    If the issue still persists please do let us know. Glad to help.

    I hope this information helps. 

     

    Thanks.

    • JayJay11's avatar
      JayJay11
      Resolver II
      2 years ago

      Thank you for your reply. Where can I see the refresh statistics you mentioned (not in the downloadable CSV, not in the refresh history)?

    • JayJay11's avatar
      JayJay11
      Resolver II
      2 years ago

      To add here: I learned from another thread that disabling staging the query in the dataflow significantly increases speed of refreshes (ca. halving refresh time) (given that there is a destination for the queries). Still slower than Gen1 though.

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Microsoft Employee
        2 years ago

        Hi JayJay11 

         

        Apologies for the delay in response and the inconvenience that you are facing here.

        Appreciate if you could share the feedback on our feedback link , which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.
        Feedback Link : Home (microsoft.com)

         

        Hope this helps. Please let us know if you have any further questions.

         

        Thanks.