Forum Discussion
Programmatically pause pool
It is indeed based on up time, but the consumption is CU. You already buy a set of CU's with your Microsoft Capacity. So if the CU consumption falls within the capacity limit, you won't be charged extra.
FabianSchut - I tested this out today and it looks like the scheduler does not pick up the jobs correctly if the pool is auto-paused.
To test this I setup an Airflow job to run hourly against the Starter Pool and insert a record into a database table every run to show when it ran. The Fabric capacity stayed active the entire time, but the pool was allowed to auto-pause since there was no other activity against it except this hourly job. The first hour worked even when it was paused, but the two subseqent hours did not run the job. When I went back into Fabric and resumed the pool manually it triggered the schedule to run.
This means that we can't reliably have the pool auto-pause, which goes back to my original question. If I could programatically pause/resume the pool then I would be able to save costs because I could set the pool to pause when I know there won't be jobs running. With the current setup I have to have an always running capacity and an always running pool for a job that only runs once in the morning for 20 minutes. This limits the usefulness of the Airflow job.
Hi, you are right, auto-pause does not resume itself. Microsoft suggests to use the starter pool only for development and a custom pool for production: https://learn.microsoft.com/en-us/fabric/data-factory/apache-airflow-jobs-workspace-settings. I've done a little research and it seems that it is not possible to programmatically pause and resume the airflow. For Spark pools, there are already some (preview) API's available, but none for the Apache Airflow pools: https://learn.microsoft.com/en-us/rest/api/fabric/spark/custom-pools/create-workspace-custom-pool?tabs=HTTP
You could create an idea for this om Microsoft: https://ideas.fabric.microsoft.com/
One note: if the Apache Airflow pool is running, it does not automatically increase the costs of Fabric. It will consume Capacity Units (CU), but if you have an F-capacity that is not fully consumed by the tasks, the custom cool can be up-and-running without increasing the F-capacity.
Thanks for the reply, I will submit an idea to Microsoft to be able to do this.
I got my pricing info from here, and it says that pricing is based on pool up time. That's why I think I'm charged for the pool being resumed even if nothing is running.
It is indeed based on up time, but the consumption is CU. You already buy a set of CU's with your Microsoft Capacity. So if the CU consumption falls within the capacity limit, you won't be charged extra.