Forum Discussion

Martins1234's avatar
Martins1234
Regular Visitor
8 months ago

Dataflow Gen2 Timeout When Loading Databricks Tables

I created a Dataflow Gen2 to get data from Databricks. I can see the preview data very quickly (around 5 seconds). But when I run the dataflow, it takes 8 hours and then cancels with a timeout. I’m t...
Data Pipeline
v-pnaroju-msft's avatar
v-pnaroju-msft
Community Support
6 months ago

Hi Martins1234,

Thankyou for the followup.

Based on my understanding, the behaviour may be caused by a Dataflow Gen2 execution limitation in Microsoft Fabric when loading certain Databricks Delta tables. Although the preview runs quickly and Databricks processes the query in milliseconds, a full Dataflow Gen2 refresh can fall back to the mashup engine and fully materialise all rows within Fabric capacity. This can result in high memory usage, capacity pressure, and eventual timeouts, even for relatively small tables of 40 to 50 MB. Applying a filter reduces the amount of data processed, which explains why the operation succeeds. Therefore, this might not be an issue with Databricks or the data quality.

Given the consistent timeout behaviour, please raise a support ticket using the provided link so that backend execution logs can be reviewed for confirmation and validation Microsoft Fabric Support and Status | Microsoft Fabric

We hope this information helps to resolve the issue. Should you have any further queries, please feel free to contact the Microsoft Fabric community.

Thank you.