Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Data pipeline reports failure but Gen2 Dataflow completes

I have ~45 Gen2 Dataflows i'm trying to refresh. I'm using several data pipelines to orchestrate this (due to the limit of 40 items per pipeline). each pipeline is refreshing ~20 dataflows at once.

 

However, the pipeline(s) 'randomly' (as in different dataflows each run) report failure "Dataflow refresh transaction failed with status Unknown". Checking the dataflows, this is occuring where the dataflow refresh is automatically retried by the service (presumably due to an internal timeout, no detail is given in the refresh history of the dataflow). The pipeline settings for all dataflow items are default; retries are set to 0

 

Seems like this is a bug. And, does anyone have a workaround (other than schedule the ~45 datflows individually on the dataflow)?

 

Error message in the pipeline:

 

 

Refresh history for the same dataflow:

 

 

pipelines
  • Anonymous's avatar
    Anonymous
    2 years ago

    Thanks. I have ended up reconfiguring the table extracts into 2 dataflows (instead of the ~45, one for each table) and am running these intead. This seams a lot more stable. No issues so far. I'm still using pipelines to trigger the dataflows.

4 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    😨😨😨

     

    Even worse, it looks like some of the failed Gen2 Dataflows runs ended up created parquet files which are now causing duplicates in my datalake tables...

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hello Anonymous ,

      Thanks for using Fabric Community.
      At this time, we are reaching out to the internal team to get some help on this .
      We will update you once we hear back from them.

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Thanks. I have ended up reconfiguring the table extracts into 2 dataflows (instead of the ~45, one for each table) and am running these intead. This seams a lot more stable. No issues so far. I'm still using pipelines to trigger the dataflows.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi Anonymous ,

      Glad to know that you find some resolution on your query. Please continue using Fabric Community for your further queries.