Dataflow Gen2 Issue

miguelYes, I saw that document, thanks.

I do think it needs to be integrated into the main data gateway networking document (Adjust communication settings for the on-premises data gateway | Microsoft Learn), otherwise folks will keep misconfiguring their firewall.

I actually inquired with Microsoft on these forums just a few weeks ago about this topic, when we had another firewall issue. The rep claimed that official data gateway is complete and that no new rules are needed for Fabric  😀

Now, the key thing in our scenario is that this error only appears for larger data volumes. Data volumes below around 150k rows work fine. What can be causing this? I would've expected the firewall rules to either work or not, regardless of the data volumes (unless there is a fancy feature in the firewall that's based on data transferred (DLP?), connection times, etc.)

Also interestingly, everything works fine when staging is disabled. Even larger data volumes of around 1M rows transfer fine.

I do understand from Data Factory Spotlight: Dataflow Gen2 | Microsoft Fabric Blog | Microsoft Fabric that staging is inherently more complicated architecturally. Does Fabric automatically disable staging at runtime for small data volumes, even if staging is enabled in the data flow?

That would explain the behaviour we are seeing, although I was unable to locate any reference to such a possible Fabric feature.