Forum Discussion

JE_test's avatar
JE_test
Resolver I
2 years ago

Dataflow refresh started failing today MashupException.Error

Hi!   I've had about 90 dataflows set up for a few months now. Last night all but 2 of them failed with error MashupException.Error The two that succeeded were very small tables with less than 100...
Dataflows
JE_test's avatar
JE_test
Resolver I
2 years ago

I just noticed that one of my dataflows were still running (or parts of it, before clicking into it it just said "failed"). trying to load 5 rows of data and it has been running for 36 hours now... and no easy way to cancel it.

When will you implement cancellation of dataflows?

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi JE_test 

     

    Thanks for using Microsoft Fabric Community.

    Apologies for the inconvenience that you are facing here.

    When using Microsoft Fabric Dataflow Gen2 with an on-premises data gateway, you might encounter issues with the dataflow refresh process. The underlying problem occurs when the gateway is unable to connect to the dataflow staging Lakehouse in order to read the data before copying it to the desired data destination. This issue can occur regardless of the type of data destination being used.

    For more details, please refer : On-premises data gateway considerations for data destinations in Dataflow Gen2.

     

    If you are using Microsoft Fabric through an on-premises data gateway, be sure to update to the latest version of the gateway. Some errors related to output destinations or queries might be resolved by upgrading the gateway and also this ensures that any updates to Fabric features and known issues are propagated through the gateway.

    For details please refer : Currently supported monthly updates to the on-premises data gateways.

     

    We're adding support to cancel ongoing Dataflow Gen2 refreshes from the workspace items view, timeline for the cancel feature, please refer : Cancel refresh support in Dataflow Gen2.

     

    I hope this information helps. Please do let us know if you have any further questions.

     

    Thanks.

    • Element115's avatar
      Element115
      Memorable Member
      2 years ago

      Hold on a sec... are you saying that the following is the path the data takes? This is the sentence that bothers me:  

       

      The underlying problem occurs when the gateway is unable to connect to the dataflow staging Lakehouse in order to read the data before copying it to the desired data destination.

       

       

      1__on-prem DB-->GW-->DefaultStagingLakehouse

      followed by 

      2__DefaultStagingLakehouse-->GW-->user lakehouse

       

      Why is the data path instead not like this? Or is it?

      1__on-prem DB-->GW-->DefaultStagingLakehouse

      followed by 

      2__DefaultStagingLakehouse -->[ internal Fabric bulk copy mechanism ] --> user lakehouse

      • JE_test's avatar
        JE_test
        Resolver I
        2 years ago

        This is the current setup:

        1__on-prem DB-->GW-->DefaultStagingLakehouse

        followed by 

        2__DefaultStagingLakehouse-->Notebook-->user lakehouse

        followed by

        3__user lakehouse-->(Stored Procedure)-->user Warehouse (this is where the transformations happen, from one schema to the next, using SPs)

         

        But today I looked at my data in my warehouse and it seems I have new data there, even though the status of my Gateways says failed for the last 3 days. Very interesting/weird...