Forum Discussion

DataPne's avatar
DataPne
Frequent Visitor
2 years ago

Gen2 Dataflows - Errors and Optimization

Hi all, Currently, our org is really struggling with Gen2 dataflows writing to a Warehouse. The speed at which they execute (sometimes 8 hours and then a timeout), success rates, general stalling ...
Dataflows
General Question
v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi DataPne 

 

Apologise for the inconvenience that you are facing here.

 

Just as a work around, instead of loading on prem data to Dataflow Gen2 and then to warehouse (On-Prem -> DFg2 -> Warehouse) can you please try to test by loading the data from on-prem to Dataflow Gen2 and then to Lakehouse.(On-Prem -> DFg2 -> Lakehouse) (files or tables, would test both)
Then create a Data Pipeline to move data from Lakehouse to Warehouse.

Just like the medallion architecture we see often, we want to do a quick load of the source into a Raw layer, then work our way to the Warehouse which would be Silver/Gold. 


The issue which you are facing while loading the data into warehouse is just temporary. This won't be a thing once Data Pipelines can use On Prem Data Gateway which is in development.

 


Hope this helps. Please let me know if you have any further questions.

v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi DataPne 

 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.


Thanks