Forum Discussion
Dataflow refresh started failing today MashupException.Error
Hi JE_test
Thanks for using Microsoft Fabric Community.
Apologies for the inconvenience that you are facing here.
When using Microsoft Fabric Dataflow Gen2 with an on-premises data gateway, you might encounter issues with the dataflow refresh process. The underlying problem occurs when the gateway is unable to connect to the dataflow staging Lakehouse in order to read the data before copying it to the desired data destination. This issue can occur regardless of the type of data destination being used.
For more details, please refer : On-premises data gateway considerations for data destinations in Dataflow Gen2.
If you are using Microsoft Fabric through an on-premises data gateway, be sure to update to the latest version of the gateway. Some errors related to output destinations or queries might be resolved by upgrading the gateway and also this ensures that any updates to Fabric features and known issues are propagated through the gateway.
For details please refer : Currently supported monthly updates to the on-premises data gateways.
We're adding support to cancel ongoing Dataflow Gen2 refreshes from the workspace items view, timeline for the cancel feature, please refer : Cancel refresh support in Dataflow Gen2.
I hope this information helps. Please do let us know if you have any further questions.
Thanks.
Hold on a sec... are you saying that the following is the path the data takes? This is the sentence that bothers me:
The underlying problem occurs when the gateway is unable to connect to the dataflow staging Lakehouse in order to read the data before copying it to the desired data destination.
1__on-prem DB-->GW-->DefaultStagingLakehouse
followed by
2__DefaultStagingLakehouse-->GW-->user lakehouse
Why is the data path instead not like this? Or is it?
1__on-prem DB-->GW-->DefaultStagingLakehouse
followed by
2__DefaultStagingLakehouse -->[ internal Fabric bulk copy mechanism ] --> user lakehouse
This is the current setup:
1__on-prem DB-->GW-->DefaultStagingLakehouse
followed by
2__DefaultStagingLakehouse-->Notebook-->user lakehouse
followed by
3__user lakehouse-->(Stored Procedure)-->user Warehouse (this is where the transformations happen, from one schema to the next, using SPs)
But today I looked at my data in my warehouse and it seems I have new data there, even though the status of my Gateways says failed for the last 3 days. Very interesting/weird...
I aggree. Gremlins everywhere in the MS backend codebase, it seems. 😁
Good to know the architecture of your pipeline, but the question was directed v-cboorla-msft because s/he wrote:
The underlying problem occurs when the gateway is unable to connect to the dataflow staging Lakehouse in order to read the data before copying it to the desired data destination.sentence which, by its use of pronouns, means the data GW is trying to write to a Fabric destination after pulling data from the staging LH. In other words, he says this:
GW-->staging LH-->back to GW-->any type of destination , so a roundtrip to the on-prem data GW
which I find very very odd to do.
I thought the more logical and efficient path, after having loaded the data from the on-prem DB via the GW would be as follows:
GW-->staging LH-->pipeline or dataflow (no more going back to the on-prem GW)-->destination (LH or DW)
If the latter, then v-cboorla-msft explanation makes no sense, no offense Boorla. 😉
GW-->staging LH-->back to GW-->any type of destination , so a roundtrip to the on-prem data GW
Yes this roundtrip is very weird. It seems to be that this second round to the GW is not working with our current firewall setup on the on-prem server, but the first load used to (and might still?) work fine.
