Forum Discussion
How to execute multiple dataflows(Dataflow Gen2 CI/CD Enabled) dynamically in Fabric?
Hello rohit_motwani ,
Yes, you can trigger dataflows with the REST API without any problems.
https://learn.microsoft.com/en-us/rest/api/power-bi/dataflows/refresh-dataflowHowever, I don't think it's the Data Pipeline Orchestration that consumes a lot (it does nothing except trigger dataflows), but data flows are not the fastest elements.
Transferring data directly via a Data Pipeline (copy data), a Python script or other method will be much more efficient.
However, it depends on the context. Dataflows are very practical, but when there is a lot of data, they consume a lot of resources.
Feel free to give me a kudo and accept my answer as the solution if it suits you.
Have a nice day,
VivienHi rohit_motwani,
In my environment, I just add the dataflow to the pipeline and run everything through the pipeline API. This gives me a more stable system that requires less changes as there is only the one API I am interacting with, instead of also interacting with the notebook and dataflow APIs respectively.
That being said, there is an API available for dataflows: Dataflows - Refresh Dataflow - REST API (Power BI Power BI REST APIs) | Microsoft Learn
You should be able to implement it much the same way as the pipeline API.
If you found this useful, consider giving Kudos. If I solved your problem, mark this post as the solution.
Hello rohit_motwani
You can create a Data Pipeline in which you organise your Dataflows as you wish (sequential, parallel) and then schedule only this Data Pipeline, which will trigger all child Dataflows.
Feel free to give me a kudo and accept my answer as the solution if it suits you.
Vivien
Hi vivien57 ,
Thank you for your response. Yes, I’m currently following the approach of organizing Dataflows within a Data Pipeline and scheduling it. However, I was exploring whether there might be an alternative method for execution—ideally one that’s more optimized, since I’ve noticed that sometimes the pipeline gets overloaded and the overall run takes longer.
Do you know if there’s any supported way to execute Dataflows directly (for example, via API or Python) that could help reduce the execution time?
Best regards,
Rohit
Hello rohit_motwani ,
Yes, you can trigger dataflows with the REST API without any problems.
https://learn.microsoft.com/en-us/rest/api/power-bi/dataflows/refresh-dataflowHowever, I don't think it's the Data Pipeline Orchestration that consumes a lot (it does nothing except trigger dataflows), but data flows are not the fastest elements.
Transferring data directly via a Data Pipeline (copy data), a Python script or other method will be much more efficient.
However, it depends on the context. Dataflows are very practical, but when there is a lot of data, they consume a lot of resources.
Feel free to give me a kudo and accept my answer as the solution if it suits you.
Have a nice day,
Vivien