Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
samaguire
Advocate II
Advocate II

Data pipeline reports failure but Gen2 Dataflow completes

I have ~45 Gen2 Dataflows i'm trying to refresh. I'm using several data pipelines to orchestrate this (due to the limit of 40 items per pipeline). each pipeline is refreshing ~20 dataflows at once.

 

However, the pipeline(s) 'randomly' (as in different dataflows each run) report failure "Dataflow refresh transaction failed with status Unknown". Checking the dataflows, this is occuring where the dataflow refresh is automatically retried by the service (presumably due to an internal timeout, no detail is given in the refresh history of the dataflow). The pipeline settings for all dataflow items are default; retries are set to 0

 

Seems like this is a bug. And, does anyone have a workaround (other than schedule the ~45 datflows individually on the dataflow)?

 

Error message in the pipeline:

 

samaguire_0-1706136589269.png

 

Refresh history for the same dataflow:

 

samaguire_1-1706136664484.png

samaguire_2-1706136966235.png

 

1 ACCEPTED SOLUTION
samaguire
Advocate II
Advocate II

Thanks. I have ended up reconfiguring the table extracts into 2 dataflows (instead of the ~45, one for each table) and am running these intead. This seams a lot more stable. No issues so far. I'm still using pipelines to trigger the dataflows.

View solution in original post

4 REPLIES 4
samaguire
Advocate II
Advocate II

Thanks. I have ended up reconfiguring the table extracts into 2 dataflows (instead of the ~45, one for each table) and am running these intead. This seams a lot more stable. No issues so far. I'm still using pipelines to trigger the dataflows.

Anonymous
Not applicable

Hi @samaguire ,

Glad to know that you find some resolution on your query. Please continue using Fabric Community for your further queries.

samaguire
Advocate II
Advocate II

😨😨😨

 

Even worse, it looks like some of the failed Gen2 Dataflows runs ended up created parquet files which are now causing duplicates in my datalake tables...

Anonymous
Not applicable

Hello @samaguire ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Helpful resources

Announcements
Sept Fabric Carousel

Fabric Monthly Update - September 2024

Check out the September 2024 Fabric update to learn about new features.

Expanding the Data Factory Forums

New forum boards available in Data Factory

Ask questions in Apache Airflow Job and Mirroring.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

Sept NL Carousel

Fabric Community Update - September 2024

Find out what's new and trending in the Fabric Community.