Forum Discussion

Martins1234's avatar
Martins1234
Regular Visitor
8 months ago

Dataflow Gen2 Timeout When Loading Databricks Tables

I created a Dataflow Gen2 to get data from Databricks. I can see the preview data very quickly (around 5 seconds). But when I run the dataflow, it takes 8 hours and then cancels with a timeout. I’m t...
data pipeline
mabdollahi's avatar
mabdollahi
Advocate IV
8 months ago

Hi Martins1234 ,

In addition what ssrithar mentioned, 

Also worth checking query folding and staging behavior in Dataflow Gen2. The preview only samples data, but during a full run any non-folding step (data type change, rename, reorder, custom column) can force Fabric to process all rows in the mashup engine, which can lead to long runtimes and timeouts.

A few practical additions:

  • Verify folding stays intact for the two failing tables all the way to the source step.

  • Disable staging for those queries if it’s enabled.

  • Load the tables independently (one dataflow per table) to rule out cross-query contention.

  • Check Fabric capacity pressure during the run — even small tables can stall if the capacity is throttled.

Together with schema alignment and Databricks OPTIMIZE, this usually resolves “fast preview, slow refresh” issues.

Regards,
Mehrdad Abdollahi