Dataflow Gen2 Timeout When Loading Databricks Tables

[Martins1234]

I created a Dataflow Gen2 to get data from Databricks. I can see the preview data very quickly (around 5 seconds). But when I run the dataflow, it takes 8 hours and then cancels with a timeout. I’m trying to get 8 tables with the same schema. Six of them work fine with no problems, but with two of them I’m experiencing the issue I just described. The table sizes are around 50 MB.

What can I do to solve this issue?

[v-pnaroju-msft]

Thankyou, @ssrithar and @mabdollahi  for your responses.

Hi Martins1234,

We appreciate your inquiry through the Microsoft Fabric Community Forum.

We would like to inquire whether have you got the chance to check the solutions provided by @ssrithar and @mabdollahito resolve the issue. We hope the information provided helps to clear the query. Should you have any further queries, kindly feel free to contact the Microsoft Fabric community.

Thank you.

[mabdollahi]

Hi @Martins1234 ,

In addition what @ssrithar mentioned, 

Also worth checking query folding and staging behavior in Dataflow Gen2. The preview only samples data, but during a full run any non-folding step (data type change, rename, reorder, custom column) can force Fabric to process all rows in the mashup engine, which can lead to long runtimes and timeouts.

A few practical additions:

• Verify folding stays intact for the two failing tables all the way to the source step.

• Disable staging for those queries if it’s enabled.

• Load the tables independently (one dataflow per table) to rule out cross-query contention.

• Check Fabric capacity pressure during the run — even small tables can stall if the capacity is throttled.

Together with schema alignment and Databricks OPTIMIZE, this usually resolves “fast preview, slow refresh” issues.

Regards,
Mehrdad Abdollahi

[ssrithar]

A mismatch between the dataflow output and the destination table is the leading cause of such timeouts.Ensure the column order in your Dataflow exactly matches the column order in the destination table.

Since 6 tables work but 2 fail despite having the same schema, there may be data-level issues in the specific 50 MB files:

• Hidden Characters: Check for special characters or extremely long strings in the two failing tables that might not be visible in the 5-second preview.
• Optimize Databricks Source: Run the OPTIMIZE command on the two problematic tables in Databricks to ensure they are not composed of thousands of tiny files, which can slow down the reading process during a full refresh even if the total size is small.

If your destination is a Fabric Lakehouse, you can use the "Fast Copy" feature to bypass the standard mashup engine.

Open the Fabric Capacity Metrics app to see if your workspace is hitting a "Throttling" limit