Forum Discussion

DataPne's avatar
DataPne
Frequent Visitor
2 years ago

Gen2 Dataflows - Errors and Optimization

Hi all, Currently, our org is really struggling with Gen2 dataflows writing to a Warehouse. The speed at which they execute (sometimes 8 hours and then a timeout), success rates, general stalling ...
Dataflows
General Question
v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi DataPne 

 

Thanks for using Microsoft Fabric Community.

Apologies for the issue that you are facing.

This might require a deeper investigation from our engineering team about your workspace, the pipeline details and the logic behind it to properly understand what might be happening. 

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

 

Thank you.

v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi DataPne 

 

Apologise for the inconvenience that you are facing here.

 

Just as a work around, instead of loading on prem data to Dataflow Gen2 and then to warehouse (On-Prem -> DFg2 -> Warehouse) can you please try to test by loading the data from on-prem to Dataflow Gen2 and then to Lakehouse.(On-Prem -> DFg2 -> Lakehouse) (files or tables, would test both)
Then create a Data Pipeline to move data from Lakehouse to Warehouse.

Just like the medallion architecture we see often, we want to do a quick load of the source into a Raw layer, then work our way to the Warehouse which would be Silver/Gold. 


The issue which you are facing while loading the data into warehouse is just temporary. This won't be a thing once Data Pipelines can use On Prem Data Gateway which is in development.

 


Hope this helps. Please let me know if you have any further questions.

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi DataPne 

     

    We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
    Otherwise, will respond back with the more details and we will try to help.


    Thanks

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Microsoft Employee
      2 years ago

      Hi DataPne 


      We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
      In case if you have any resolution please do share that same with the community as it can be helpful to others.
      If you have any question relating to the current thread, please do let us know and we will try out best to help you.
      In case if you have any other question on a different issue, we request you to open a new thread .

       

      Thanks