Iirresponsible?

[yjh]

Many people have said that DataFlow gen2 writing to WareHouse from various sources will continue abnormally for many hours with a high probability of failure, why can't Microsoft solve this problem for a long time? Now that Fabric is in GA state, when did Microsoft become so irresponsible?

I'm also facing this issue now.

[Joshrodgers123]

Hi @v-nikhilan-msft - if this is a known issue, why hasn't it been added to the known issues list? It seems many customers are having similar issues. According to the GA status of Fabric, we expect these things will work and there is no documentation or posted known issue that says otherwise.

[v-nikhilan-msft]

Hi @Joshrodgers123 
Please refer to these links for Known Issues: Microsoft Fabric Known Issues
The known issue is stated here: Known issue - Gateway Refreshes longer than an hour fail - Microsoft Fabric | Microsoft Learn

[Joshrodgers123]

This issue is related to writing to a warehouse and is not applicable to the known issue you linked. 

Here are 2 recent posts from others with similar issues.

Gen2 Dataflow to Warehouse - really slow performan... - Microsoft Fabric Community

Gen2 Dataflows - Errors and Optimization - Microsoft Fabric Community

[v-nikhilan-msft]

Hi @Joshrodgers123 
Just as a work around, instead of loading on prem data to Dataflow Gen2 and then to warehouse (On-Prem -> DFg2 -> Warehouse) can you please try to test by loading the data from on-prem to Dataflow Gen2 and then to Lakehouse.(On-Prem -> DFg2 -> Lakehouse) (files or tables, would test both)
Then create a Data Pipeline to move data from Lakehouse to Warehouse.

Just like the medallion architecture we see often, we want to do a quick load of the source into a Raw layer, then work our way to the Warehouse which would be Silver/Gold. 

The issue which you are facing while loading the data into warehouse is just temporary. This won't be a thing once Data Pipelines can use On Prem Data Gateway which is in development.

Hope this helps. Please let me know if you have any further questions.

[v-nikhilan-msft]

Hi @Joshrodgers123 
Thanks for letting me know. I have informed the same to the internal team. 
I will get back to you once I hear back from them. 
Appreciate your patience.

[v-nikhilan-msft · Accepted solution]

Hi @yjh 
Thanks for using Fabric Community. 
Apologies for the issue you have been facing. We understand that this issue has caused problems to your operations, and we sincerely apologize for the inconvenience. We want to assure you that Microsoft is taking this issue very seriously and is actively working on a fix. Our engineering teams have identified the root cause of the problem and are currently developing a solution. 

This is a known issue and the team is aware about this. But we do not have any ETA regarding the fix. We will continue to keep you updated on the progress of the fix and provide you with additional information as it becomes available. We appreciate your patience and understanding as we work to resolve this issue.

Thank you.