I have setup an apacheAirflowJob using a starter pool as this offers extremely good and efficient use of compute resource and manages costs well (as detailed in the launch blog post). However, I have found that after pausing and resuming my compute instance (which I do to save costs when not using the platform) that the starter pool does not resume. This is a problem if I want to automate my DAG job run. I have synced the pool to a git repo to store my dags in (repo is under 3MB). I have a straight forward medallion architecture which I want to run once per day. I have a copy job that separately retrieves data from an SQL server instance and then I use airflow to process the data from a lakehouse to a warehouse. The whole airflow job takes no more than 30 minutes on an F8 instance (which my workspace is licensed on).
My query is that if I automate the airflow task via terraform, I need to be able to check the status of the starter pool, resume it if it is stopped and potentially stop it when finished for the day (although it will idle after 20 minutes of non-use, it would be useful to be able to tell the service explicitly when not needed). Because it can take up to 30 minutes to resume I could account for this with my terraform automation.
Currently I may only resume a starter pool if stopped when logged into fabric. I can see the following APIs when I have a apacheAirfowJob open in Fabric:
https://[region-id].pbidedicated.windows.net/webapi/capacities/[capacity-id]/workloads/DI/DiService/...
https://[region-id].pbidedicated.windows.net/webapi/capacities/[capacity-id]/workloads/DI/DiService/...
These are authenticated via an MvcToken which appears to be a wrapper token that checks if the request is being initiated by a user (as opposed to an app). I have a service principle set up in Entra and added as a member to my workspace. Ideally I can use this service principle to call these apis so that I can check in my terraform pipeline if the start pool is available. The issue currently is that trying to run an airflow job if the start pool doesn’t exist at the IP will fail. The only way to fix it is to manually resume the starter pool. Given that this resume operation is being achieved in Fabric via an api call I want to be able to use that same API in my terraform to check if the pool is available and to start it if it isn’t before then going on to run the dags.
Has anyone had any success with this. Currently I can reach all the apis and if I copy the token I may run them from curl in the commandline, but I cannot yet generate an MvcToken from outside fabric that will allow me to automate this. There is an API here:
https://wabi-uk-south-c-primary-redirect.analysis.windows.net/metadata/v201606/generatemwctoken
which is being used to sign a token from Fabric, but cannot be used with a service principle because it includes "idtyp": "app", which is rejected by fabric with 401.
Is it possible to call the getStatus or start API endpoints using either managed identity or service principle? If so I would really appreciate some guidance as this is fairly standard business pipeline automation stuff and I need it for my project.
Thank you.
