Data pipeline reports failure but Gen2 Dataflow completes

[samaguire]

I have ~45 Gen2 Dataflows i'm trying to refresh. I'm using several data pipelines to orchestrate this (due to the limit of 40 items per pipeline). each pipeline is refreshing ~20 dataflows at once.

However, the pipeline(s) 'randomly' (as in different dataflows each run) report failure "Dataflow refresh transaction failed with status Unknown". Checking the dataflows, this is occuring where the dataflow refresh is automatically retried by the service (presumably due to an internal timeout, no detail is given in the refresh history of the dataflow). The pipeline settings for all dataflow items are default; retries are set to 0

Seems like this is a bug. And, does anyone have a workaround (other than schedule the ~45 datflows individually on the dataflow)?

Error message in the pipeline:

Refresh history for the same dataflow:

[samaguire · Accepted solution]

Thanks. I have ended up reconfiguring the table extracts into 2 dataflows (instead of the ~45, one for each table) and am running these intead. This seams a lot more stable. No issues so far. I'm still using pipelines to trigger the dataflows.

[Anonymous]

Hi @samaguire ,

Glad to know that you find some resolution on your query. Please continue using Fabric Community for your further queries.

[samaguire]

😨😨😨

Even worse, it looks like some of the failed Gen2 Dataflows runs ended up created parquet files which are now causing duplicates in my datalake tables...

[Anonymous]

Hello @samaguire ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.