topic Data pipeline reports failure but Gen2 Dataflow completes in Pipelines

Data pipeline reports failure but Gen2 Dataflow completes

Anonymous — Wed, 24 Jan 2024 22:58:49 GMT

I have ~45 Gen2 Dataflows i'm trying to refresh. I'm using several data pipelines to orchestrate this (due to the limit of 40 items per pipeline). each pipeline is refreshing ~20 dataflows at once.

However, the pipeline(s) 'randomly' (as in different dataflows each run) report failure "Dataflow refresh transaction failed with status Unknown". Checking the dataflows, this is occuring where the dataflow refresh is automatically retried by the service (presumably due to an internal timeout, no detail is given in the refresh history of the dataflow). The pipeline settings for all dataflow items are default; retries are set to 0

Seems like this is a bug. And, does anyone have a workaround (other than schedule the ~45 datflows individually on the dataflow)?

Error message in the pipeline:

Refresh history for the same dataflow:

Re: Data pipeline reports failure but Gen2 Dataflow completes

Anonymous — Thu, 25 Jan 2024 01:45:42 GMT

😨😨😨

Even worse, it looks like some of the failed Gen2 Dataflows runs ended up created parquet files which are now causing duplicates in my datalake tables...

Re: Data pipeline reports failure but Gen2 Dataflow completes

Anonymous — Thu, 25 Jan 2024 12:57:54 GMT

Hello Anonymous ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Re: Data pipeline reports failure but Gen2 Dataflow completes

Anonymous — Thu, 25 Jan 2024 20:09:54 GMT

Thanks. I have ended up reconfiguring the table extracts into 2 dataflows (instead of the ~45, one for each table) and am running these intead. This seams a lot more stable. No issues so far. I'm still using pipelines to trigger the dataflows.

Re: Data pipeline reports failure but Gen2 Dataflow completes

Anonymous — Mon, 29 Jan 2024 08:56:02 GMT

Hi Anonymous ,

Glad to know that you find some resolution on your query. Please continue using Fabric Community for your further queries.