Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Dataflow updated without errors but the result is not complete

Hi, we created several Dataflows Gen2, which are to be executed regularly by a Data Pipeline. While setting up the Pipeline, we checked several times that all dependencies such as data sources, ...
dataflow
  • v-karpurapud's avatar
    v-karpurapud
    1 year ago

    Hi Anonymous 

    May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

    Thank you

Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi,

In my opinion, race conditions should not be a problem in our system.
We use a data pipeline in which we execute 15 activities depending on each other.
Our system follows the medallion scheme, where we transform data from two independent sources at bronze and silver level in parallel and then combine them in our gold level.
The pipeline covers the entire workflow from data ingestion in Azure Data Factory to transformations in several data flows and updating the semantic model.
All dataflows are waiting for the (successful) completion of their predecessors.
Lakehouse tables are never read or written to by multiple dataflows at the same time in our pipeline.

You mentioned that Fabric does not always log partial errors, is there a way to get more detailed insights into the execution of dataflows or pipeline activities beyond the logs of the monitoring tab? Is it possible to see the prioritization of dynamic resource allocation? We already use the Microsoft Fabric Capacity Metrics App.
v-karpurapud's avatar
v-karpurapud
Icon for Community Support rankCommunity Support
1 year ago

Hi Anonymous 


We understand that race conditions are unlikely in your case and appreciate your detailed explanation of the Medallion architecture and pipeline execution dependencies. Since Fabric does not always log partial error. we recommend the following steps to consider :

Open Dataflow Gen2 → Monitor → Execution History to review execution times, warnings, and table updates. Since the Monitoring tab only shows high-level logs, enable detailed execution logging at the pipeline level:

 

Check Throttle Events & CPU/Memory Spikes in Fabric Capacity Metrics App to detect resource prioritization issues.Compare execution order & delays across multiple runs to see if specific Dataflows are consistently deprioritized.

 

Thank you.