Forum Discussion
Gen2 Dataflow (CI/CD)
jp_here2026
This is issue more probably due to increase in datasize.
and the problem occurs when you try to write this into lakehouse.
Steps to follow:
1-To confirm this only try to limit the load of data like first 100 rows for eachTable.FirstN(Source,100)If this solve the issue it means you have data limitation constraint.
if not then it means it is definitely related to Schema or Fabric Related issue.
2- Next recommendation is try to write data for failed piplines in another lakehouse or in another warehouseThis helps to identy whether the problem is with the source or destination,
further write to in a new table, if succeed then it means you need to create new Delta tables in lakehouse.
Check the fabric capacity size.My suggestion:
- Lakehouse write timeout due to increased data volume or slower writes- Most Likely
- Recent Microsoft Fabric backend issue/regression, Most likely
- API response has changed (size or schema) due to schema drift.-Less Likely coz pipelines were working previously.
- Corrupted Delta Table metadata, need to create new table.- Less Likely
If this helps, ✓ Mark as Kudos | Help Others
jp_here2026
This is issue more probably due to increase in datasize.
and the problem occurs when you try to write this into lakehouse.
Steps to follow:
1-To confirm this only try to limit the load of data like first 100 rows for each
Table.FirstN(Source,100)
If this solve the issue it means you have data limitation constraint.
if not then it means it is definitely related to Schema or Fabric Related issue.
2- Next recommendation is try to write data for failed piplines in another lakehouse or in another warehouse
This helps to identy whether the problem is with the source or destination,
further write to in a new table, if succeed then it means you need to create new Delta tables in lakehouse.
Check the fabric capacity size.
My suggestion:
- Lakehouse write timeout due to increased data volume or slower writes- Most Likely
- Recent Microsoft Fabric backend issue/regression, Most likely
- API response has changed (size or schema) due to schema drift.-Less Likely coz pipelines were working previously.
- Corrupted Delta Table metadata, need to create new table.- Less Likely
If this helps, ✓ Mark as Kudos | Help Others
Murtaza_Ghafoor Thanks for the idea. I tried with First 100, where the dataflow Works and data stored in Lakehouse. But if i tried with entire data it throws timeout error.
jp_here2026
The next step would be check the capacity throtlling issue from fabric metric capacity apps.
If the issue is with capacity sizing then you need to increase the capacity.
Other option would be implment CDC incremental load that will not choke the capacity.
If this helps, ✓ Mark as Kudos | Help Others