Microsoft - new gateway server network diagram?

[ScottPowellECU]

Hi folks, Microsoft had published a network diagram for how the gateway servers worked in Power BI ( https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-onprem-indepth )

From my understanding this doesn't apply to Fabric - it seems that the gateway server needs to be able to talk directly to the Power BI aka Fabric service on port 1433 to load data to a warehouse or lakehouse. I'm understanding this due to the fact that loads are failing if port 1433 is blocked from the gateway server.

Just wanted Microsoft to be aware that this diagram needs updated. Are there other changes than a new direct connection between the gateway servers and the SQL endpoints?

Thanks,

Scott

[nikkiwaghani]

This should be addressed with the July release of the gateway. Can you update and try again?

Also, I can confirm that this does apply to Fabric - we will update the image, good catch on that.

[AnthonyGenovese]

It looks like the new July 2023 Gateway server addresses a lot of these issues.

[motoray]

Oh man, I hope you're right. I've had really spotty "success" trying to get onprem data into Fabric so far and am feeling like maybe it's time to set this aside until it matures a bit more.

[AnthonyGenovese]

I was in the exact same boat. Bashing my head with issue after issue. 3 new errors in the past 24 hours. But then I just refreshed a dataflow today to get a baseline, and it worked. Only thing that changed from previous attempts was an updated gateway.  Fingers crossed for you. 

[motoray]

It's better. I'd been able to get around 25k rows to load consistently(ish). That number is more like 50k rows now. The table I want to load has 5 million rows. At this point I'm not even loading from on-prem but am hitting an Azure SQL database. Maybe Gen2 dataflows just aren't ready for primetime yet. 

The new results are after updating to the July 2023 gateway.

[marcoG]

Hi @ScottPowellECU ,

I found this article:

https://learn.microsoft.com/en-us/fabric/data-factory/gateway-considerations-output-destinations

And this fabric blog article:

https://blog.fabric.microsoft.com/it-it/blog/data-factory-spotlight-dataflows-gen2?ft=All:

So i will attend for next release to disable staging area so I don’t need to use 1433 port.

I think this will work for me.

Bye

MG

[motoray]

This is new and I don't really understand why. I've been able to get data loaded into staging warehouses and lakehouses so it doesn't make much sense to me that the step of loading data to the actual warehouse needs to suddenly connect to the gateway servers on that port.

[AnthonyGenovese]

I totally agree. 

[ScottPowellECU]

Agree 100%!

Scott