Hi @BoNDoK01 ,
You can parameterize pipelines or read the datasets resp. tables you want to refresh in a pipeline step from a configuration table that you maintain, so you don't need to change the pipeline necessarily. You can also implement something like get all datasets from Power BI REST API or get all datasets within a workspace and then refresh this list, this way you can automatically include new datasets, but somehow the refresh activity needs to know which datasets to refresh. You can also get from the Power BI REST API the datasets and their data sources and once your pipeline has updated a data source you refresh all depending datasets. But data source in Power BI is a database, not e.g. a single table, so this might be a too high granularity for controlling refreshes.
If you have multiple pipelines that need to be executed before dataset refresh, you can build an orchestration pipeline, running these multiple piplines and then do one refresh of Power BI datasets. You can also have multiple pipelines that each do the refresh. It depends on your requirements and your ETL design what is more appropriate in order to prevent too many redundant refreshes.
The Azure equivilant for what used to be the Windows Task Scheduler is Azure Automation https://www.linkedin.com/pulse/azure-automation-vs-task-scheduler-jamie-lawson/ You can write a small PowerShell script there that does the refreshes an you can trigger execution from time events/schedules or webhook called events https://learn.microsoft.com/en-us/azure/automation/automation-webhooks?tabs=portal Your pieplines can call such webhook or any other event that you can turn into a webhook call. What events do you have in mind, technically?
BR
Martin
Modified: Corrected wrong link.
