Forum Discussion

bcdobbs's avatar
bcdobbs
Community Champion
3 years ago

Dataflow Gen2 Issue

Hi,   I'm trying to run a very simple Gen2 Dataflow from an on prem sql server.   All setups fine but when I ask it to refresh I get: "Error: Couldn't refresh the entity because of an issue ...
Dataflows
vitaly's avatar
vitaly
Advocate II
3 years ago

miguelYes, I saw that document, thanks.

 

I do think it needs to be integrated into the main data gateway networking document (Adjust communication settings for the on-premises data gateway | Microsoft Learn), otherwise folks will keep misconfiguring their firewall.

 

I actually inquired with Microsoft on these forums just a few weeks ago about this topic, when we had another firewall issue. The rep claimed that official data gateway is complete and that no new rules are needed for Fabric  😀

 

Now, the key thing in our scenario is that this error only appears for larger data volumes. Data volumes below around 150k rows work fine. What can be causing this? I would've expected the firewall rules to either work or not, regardless of the data volumes (unless there is a fancy feature in the firewall that's based on data transferred (DLP?), connection times, etc.)

 

Also interestingly, everything works fine when staging is disabled. Even larger data volumes of around 1M rows transfer fine.

 

I do understand from Data Factory Spotlight: Dataflow Gen2 | Microsoft Fabric Blog | Microsoft Fabric that staging is inherently more complicated architecturally. Does Fabric automatically disable staging at runtime for small data volumes, even if staging is enabled in the data flow?

 

That would explain the behaviour we are seeing, although I was unable to locate any reference to such a possible Fabric feature.

  • AnthonyGenovese's avatar
    AnthonyGenovese
    Resolver III
    3 years ago

    How do you disable staging in the gen2 dataflow? When I click on the enable staging button, it grays out PUBLISH and won't let me save.

    • SidJay's avatar
      SidJay
      Microsoft Employee
      3 years ago

      Hi Anthony,

       

      We currently have a restriction where at least one query needs to be staged, but that restriction is being lifted. Depending on your region you may already have the change. We expect the change to be in all regions by next week.

       

      If your region doesn't have the change yet, a temporary workaround would be to add one dummy query with staging enabled.

       

      Thanks