Forum Discussion
Dataflow Gen2 Timeout When Loading Databricks Tables
I created a Dataflow Gen2 to get data from Databricks. I can see the preview data very quickly (around 5 seconds). But when I run the dataflow, it takes 8 hours and then cancels with a timeout. I’m t...
A mismatch between the dataflow output and the destination table is the leading cause of such timeouts.Ensure the column order in your Dataflow exactly matches the column order in the destination table.
Since 6 tables work but 2 fail despite having the same schema, there may be data-level issues in the specific 50 MB files:
- Hidden Characters: Check for special characters or extremely long strings in the two failing tables that might not be visible in the 5-second preview.
- Optimize Databricks Source: Run the OPTIMIZE command on the two problematic tables in Databricks to ensure they are not composed of thousands of tiny files, which can slow down the reading process during a full refresh even if the total size is small.
If your destination is a Fabric Lakehouse, you can use the "Fast Copy" feature to bypass the standard mashup engine.
Open the Fabric Capacity Metrics app to see if your workspace is hitting a "Throttling" limit