Forum Discussion

etane's avatar
etane
Icon for Helper V rankHelper V
8 months ago
Solved

Stop Refresh if it Exceeds X Minutes

Hello.

 

I have lots of dataflows that are linked.  And, they are scheduled to refresh 30 minutes apart.  It usually takes about 5 minutes for a refresh to complete.  However, sometimes, for unknown reasons, the refresh can take longer than 30 minutes.  And, this prevents the next dataflow's refresh from starting.  

 

Is there a way to stop a refresh that has already started but exceeds a maximum time?  In effect, I want to abandon the dataflow's refresh that exceeds a time limit so that 1) the next dataflow's refresh can start without interuption and 2) the Fabric CU doesn't take too much of a hit.

 

Thanks.

Need Help
  • Natarajan_M's avatar
    Natarajan_M
    8 months ago

    Hi etane , 

    There is no built-in method to check/stop the refresh time of any object in fabric . 

    you can use either pipeline/notebook to orchestrate the entire process of refreshing and monitoring the pipelines .

    Note bookapproach :

    Run A → wait max 30 minutes  max
                  If completed before 30 mins → Run B
                  If exceeded 30 mins → cancel + notify + STOP A -> go ahead with B

    Run B → wait max 30 minutes  max
                  If completed before 30 mins → Run C
                  If exceeded 30 mins → cancel + notify + STOP B -> go ahead with C
    Run C → wait max 30 minutes  max
                  If completed before 30 mins → notify user all process compelted
                  If exceeded 30 mins → cancel + notify + STOP C 

    Use the Rest API calls to get the dataflow run status

    Also any reason for running the data flows in sequential? is there any dependency or just to limit the stress on the capacity the dataflows are ran in sequential

    Thanks 

5 Replies

  • GilbertQ's avatar
    GilbertQ
    Icon for Super User rankSuper User
    8 months ago

    Hi etane 

     

    What you could do is to look at using Power Automate to monitor the refresh of a data flow and if it exceeds a certain amount of time you could then stop it. To do this you would need to use the Power BI REST Apis. 


    Here is the API to get start the refresh: Dataflows - Refresh Dataflow - REST API (Power BI Power BI REST APIs) | Microsoft Learn

     

    Here is the API to monitor the Refresh: Dataflows - Get Dataflow Transactions - REST API (Power BI Power BI REST APIs) | Microsoft Learn

     

    And here is the API to cancel the refresh: Dataflows - Cancel Dataflow Transaction - REST API (Power BI Power BI REST APIs) | Microsoft Learn

  • Natarajan_M's avatar
    Natarajan_M
    Icon for Super User rankSuper User
    8 months ago

    Hi etane , 

    There is no built-in method to check/stop the refresh time of any object in fabric . 

    you can use either pipeline/notebook to orchestrate the entire process of refreshing and monitoring the pipelines .

    Note bookapproach :

    Run A → wait max 30 minutes  max
                  If completed before 30 mins → Run B
                  If exceeded 30 mins → cancel + notify + STOP A -> go ahead with B

    Run B → wait max 30 minutes  max
                  If completed before 30 mins → Run C
                  If exceeded 30 mins → cancel + notify + STOP B -> go ahead with C
    Run C → wait max 30 minutes  max
                  If completed before 30 mins → notify user all process compelted
                  If exceeded 30 mins → cancel + notify + STOP C 

    Use the Rest API calls to get the dataflow run status

    Also any reason for running the data flows in sequential? is there any dependency or just to limit the stress on the capacity the dataflows are ran in sequential

    Thanks 

    • etane's avatar
      etane
      Icon for Helper V rankHelper V
      8 months ago

      Hello.

       

      Some data is loading from on premise via data gateway.  This data cannot directly load into dataflow to merge with fact table.  It has to first load into its own dataflow.  So, the on premise data first loads into dataflow 1.  Dataflow 1 is linked to dataflow 2, and when dataflow 1 starts refreshing (pulling data from on premise erp), it loads data into dataflow 2.  (dataflow 2 is also linked to dataflow 3 as it appends to data in dataflow 3).

  • v-hashadapu's avatar
    v-hashadapu
    Icon for Community Support rankCommunity Support
    8 months ago

    Hi etane , Thank you for reaching out to the Microsoft Community Forum.

     

    We find the answer shared by GilbertQ  is appropriate. Can you please confirm if the solution worked for you. It will help others with similar issues find the answer easily.

     

    Thank you GilbertQ  for your valuable response.

  • v-hashadapu's avatar
    v-hashadapu
    Icon for Community Support rankCommunity Support
    8 months ago

    Hi etane , Hope you are doing well. Kindly let us know if the issue has been resolved or if further assistance is needed. Your input could be helpful to others in the community.