Forum Discussion
Gen2 Dataflow (CI/CD)
jp_here2026
This is issue more probably due to increase in datasize.
and the problem occurs when you try to write this into lakehouse.
Steps to follow:
1-To confirm this only try to limit the load of data like first 100 rows for eachTable.FirstN(Source,100)If this solve the issue it means you have data limitation constraint.
if not then it means it is definitely related to Schema or Fabric Related issue.
2- Next recommendation is try to write data for failed piplines in another lakehouse or in another warehouseThis helps to identy whether the problem is with the source or destination,
further write to in a new table, if succeed then it means you need to create new Delta tables in lakehouse.
Check the fabric capacity size.My suggestion:
- Lakehouse write timeout due to increased data volume or slower writes- Most Likely
- Recent Microsoft Fabric backend issue/regression, Most likely
- API response has changed (size or schema) due to schema drift.-Less Likely coz pipelines were working previously.
- Corrupted Delta Table metadata, need to create new table.- Less Likely
If this helps, ✓ Mark as Kudos | Help Others
Dev_Dholakia Thanks for suggestion, i tried to Limit first 100 records, the flow works fine. I beleive this is fabric capacity error.
Resolver IVhi jp_here2026 ,
Glad the First 100 test confirmed it — as the accepted solution already covered, this is a Lakehouse write timeout driven by data volume, not a schema issue.
Just adding a couple of practical follow-ups for anyone hitting the same:
- Move to Incremental Refresh / Append instead of full Replace — full loads will keep failing as data keeps growing. This is the real long-term fix.
- For the initial historical load, use a Pipeline Copy activity → Lakehouse (it's much faster and doesn't hit mashup timeouts), then let the Dataflow handle only the daily delta.
- Check the Fabric Capacity Metrics app during the failing window — if you see smoothing/overload, scaling up the SKU briefly for one clean full load usually gets you unblocked.
- Pull data from the API in chunks (date range / pagination) rather than one big pull, if the source supports it.
Disclosure: My earlier response above was drafted with the assistance of an AI tool (Microsoft Copilot) and cross-checked against Microsoft Learn documentation. Sharing this note for transparency as per the Community AI Usage Policy