Forum Discussion

JibinSebastian's avatar
JibinSebastian
Icon for Advocate II rankAdvocate II
4 months ago
Solved

Scheduling a Microsoft Fabric Pipeline with Specific Time Intervals

I would like to configure a Microsoft Fabric pipeline to run on a recurring schedule with the following requirements: Execute every 10 minutes Start time: 7:10 AM EST End time: 6:50 PM EST Frequ...
data engineering
  • deborshi_nag's avatar
    deborshi_nag
    4 months ago

    Hello JibinSebastian 

     

    I am assuming your Fabric capacity is paused outside business hours (7AM - 7PM) to save costs. If you set your pipeline to trigger every 10 mins, the trigger will run fine but your pipeline will fail because your capacity is found Paused. 

     

    Secondly, Power BI semantic model enforces a 30 min minimum interval window, so the service backend will block it, regardless we tried a 10 min interval. Maximum 8 scheduled refreshes per day per dataset is allowed. 

     

    Considering the above, I suggest you design it the following way:

     

    Orchestrator pipeline

    Notebook activity (checks against weekends) 
    Scheduled every 10 minutes, business hours only

    Pipeline 1 – Data ingestion (trigger if parent Notebook passes as True)
    Copy Data

    Pipeline 2 – Semantic model refresh (runs after pipeline 1 success, engine may skip or throttle if run within 30 mins)
    Semantic model refresh activity

     

    Schedule as discussed in previous post:

    • Frequency: every 10 minutes
    • Time window: 07:10–18:50
    • Time zone: EST

     

deborshi_nag's avatar
deborshi_nag
Icon for Super User rankSuper User
4 months ago

Hello JibinSebastian 

  

Unfortunately Fabric pipelines currently don't have support for cron expression or weekday only trigger. What you could use is set one trigger that covers rest and apply the weekday logic inside a Notebook in the pipeline. 

 

So, effectively your pipeline trigger would look something like this:

Frequency: Minute - Interval: 10 - Time zone: Eastern Standard Time - Start time: 07:10 - End time: 18:50
 
Your notebook logic could include something like this:
 
# skip if it is not a weekday

if now.weekday() >= 5:
    print("Pipeline not scheduled to run on weekends – exiting ...")
    dbutils.notebook.exit("SKIPPED")
 
  • JibinSebastian's avatar
    JibinSebastian
    Icon for Advocate II rankAdvocate II
    4 months ago

    Hi @debroshi, thank you for your response.

    I currently have two pipelines:

    1. The first pipeline uses Copy Data to move a small amount of data from a large SQL Server table.
    2. The second pipeline is for refreshing a semantic model.

    I’m new to Microsoft Fabric and ran into a couple of issues:

    • I tried scheduling the pipeline to run every 10 minutes. It works during active hours, but when Fabric is suspended (outside 7 AM to 7 PM), the pipeline fails to execute.
    • For the semantic model refresh, the scheduler only allows a minimum interval of 30 minutes. However, I need it to refresh every 10 minutes on weekdays between 7 AM and 7 PM.

    Could you please guide me on how to set up the trigger properly? It would really help if you could explain it in simple, beginner-friendly terms.

    Thank you!

    • deborshi_nag's avatar
      deborshi_nag
      Icon for Super User rankSuper User
      4 months ago

      Hello JibinSebastian 

       

      I am assuming your Fabric capacity is paused outside business hours (7AM - 7PM) to save costs. If you set your pipeline to trigger every 10 mins, the trigger will run fine but your pipeline will fail because your capacity is found Paused. 

       

      Secondly, Power BI semantic model enforces a 30 min minimum interval window, so the service backend will block it, regardless we tried a 10 min interval. Maximum 8 scheduled refreshes per day per dataset is allowed. 

       

      Considering the above, I suggest you design it the following way:

       

      Orchestrator pipeline

      Notebook activity (checks against weekends) 
      Scheduled every 10 minutes, business hours only

      Pipeline 1 – Data ingestion (trigger if parent Notebook passes as True)
      Copy Data

      Pipeline 2 – Semantic model refresh (runs after pipeline 1 success, engine may skip or throttle if run within 30 mins)
      Semantic model refresh activity

       

      Schedule as discussed in previous post:

      • Frequency: every 10 minutes
      • Time window: 07:10–18:50
      • Time zone: EST

       

      • JibinSebastian's avatar
        JibinSebastian
        Icon for Advocate II rankAdvocate II
        4 months ago

        This worked Thank you for your time and efforts. may be in future fabric will implement cron into pipeline which makes everybodies life easy

    • lbendlin's avatar
      lbendlin
      Icon for Super User rankSuper User
      4 months ago
      For the semantic model refresh, the scheduler only allows a minimum interval of 30 minutes. However, I need it to refresh every 10 minutes on weekdays between 7 AM and 7 PM.

       

      That is way too frequent.  Consider using Direct Query connection or streaming datasets/RTI.

       

      Note:  If you request a refresh via API (for example through Power Automate) there is no such limit - but you must implement your own polling to find out if there is currently a refresh in progress before you submit another request.