Nuanced Scheduling in Fabric Data Factory

[Bshin]

I have created an ingestion pipeline that ETL's data from a source system, but am having trouble scheduling the pipeline as its a bit nuanced. 

The requirements:

- On Mondays, Wednesdays, and Fridays: Pipeline runs every day at 9AM

- On Tuesdays and Thursdays: Pipeline runs at 9AM as well, but also needs updating at 12PM and 2PM too.

It appears I can only attach one schedule to each pipeline in Fabric Data Factory. So if I create a daily schedule at 9AM, I can no longer create the second schedule to handle the 12PM and 2PM trigger on Tuesdays and Thursdays. 

Is there a recommended workaround for nuanced scheduling? Honestly quite absurd that something this basic is not available in Fabric.

[FabianSchut]

You could also take a look at Apache Airflow and its wide capability of scheduling. It is code based and a relatively big setup just to fix this issue, but you will not have to modify your current pipelines with any if statements. You can take a look at this example from Microsoft how to trigger an pipeline with Apache Airflow: https://learn.microsoft.com/en-us/fabric/data-factory/apache-airflow-jobs-run-fabric-item-job. There are also some plugins to make the setup a bit easier: https://github.com/ambika-garg/apache-airflow-microsoft-fabric-plugin. 

With Apache Airflow, you can create your custom schedule with code. 

[nilendraFabric]

Posted an idea for this   Please vote.

https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=abe68d7e-83de-ef11-95f5-000d3a0fe9f3

[Bshin]

So lets say I use that solution, you suggest I create a schedule to run my pipeline every day, every hour, just to filter out most of those executions? Would that not be a needless waste of my CU's?

[nilendraFabric]

This method avoids creating multiple pipelines but can complicate the pipeline design and cost bit extra as we discussed 

[nilendraFabric · Accepted solution]

@Bshin 100% agreed.

but since we are only running one if statement with simple condition it won't be much wastage. Not ideal suitation though.

Thanks

[nilendraFabric]

Hello @Bshin 

This was discussed in this forum earlier as well. 

yes its not great to unable to attach multiple custom schedules to pipeline

but this approach worked for many users

Use an If activity at the beginning of the pipeline to check the current day and time.
• Configure the logic to execute specific activities only if the conditions match (e.g., run additional updates at 12 PM and 2 PM on Tuesdays and Thursdays).

Hope this works 

Pleasr accept the solution if this is helpful 

thanks