Forum Discussion
Data pipeline reports failure but Gen2 Dataflow completes
I have ~45 Gen2 Dataflows i'm trying to refresh. I'm using several data pipelines to orchestrate this (due to the limit of 40 items per pipeline). each pipeline is refreshing ~20 dataflows at once. ...
Thanks. I have ended up reconfiguring the table extracts into 2 dataflows (instead of the ~45, one for each table) and am running these intead. This seams a lot more stable. No issues so far. I'm still using pipelines to trigger the dataflows.
😨😨😨
Even worse, it looks like some of the failed Gen2 Dataflows runs ended up created parquet files which are now causing duplicates in my datalake tables...
Hello Anonymous ,
Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.