Forum Discussion
Dataflow refresh started failing today MashupException.Error
I aggree. Gremlins everywhere in the MS backend codebase, it seems. 😁
Good to know the architecture of your pipeline, but the question was directed v-cboorla-msft because s/he wrote:
The underlying problem occurs when the gateway is unable to connect to the dataflow staging Lakehouse in order to read the data before copying it to the desired data destination.
sentence which, by its use of pronouns, means the data GW is trying to write to a Fabric destination after pulling data from the staging LH. In other words, he says this:
GW-->staging LH-->back to GW-->any type of destination , so a roundtrip to the on-prem data GW
which I find very very odd to do.
I thought the more logical and efficient path, after having loaded the data from the on-prem DB via the GW would be as follows:
GW-->staging LH-->pipeline or dataflow (no more going back to the on-prem GW)-->destination (LH or DW)
If the latter, then v-cboorla-msft explanation makes no sense, no offense Boorla. 😉
GW-->staging LH-->back to GW-->any type of destination , so a roundtrip to the on-prem data GW
Yes this roundtrip is very weird. It seems to be that this second round to the GW is not working with our current firewall setup on the on-prem server, but the first load used to (and might still?) work fine.
TBH I'm not convinced there's a roundtrip... waiting for v-cboorla-msft or someone else at Microsoft to confirm one way or the other.
When staging is enabled and On Prem Data Gateway is used, the data will flow as such:
1. [Data Source] ---OPDG---> [Staging Lakehouse]
2. [Staging Lakehouse] --- OPDG ---> [Final Output]
It may not be the most optimal path, hence if you do not need staging of entities, please turn it off. Without staging, the path should simply be
[Data Source] ---OPDG---> [Output]
pqian_MSFT Thanks. Agreed. However, here's a question for you:
I am forced to use this Microsoft suggested workaround because atm using one DFg2 to extract data from an on-prem DB does not work (well, at least for us it doesn't--case# 2401190040000098). However, using the chained DFs workaround does work.
So my question is: When will this [Data Source] ---OPDG---> [Output] become available for on-prem datasources?
