Forum Discussion
How to execute multiple dataflows(Dataflow Gen2 CI/CD Enabled) dynamically in Fabric?
Hello rohit_motwani ,
Yes, you can trigger dataflows with the REST API without any problems.
https://learn.microsoft.com/en-us/rest/api/power-bi/dataflows/refresh-dataflowHowever, I don't think it's the Data Pipeline Orchestration that consumes a lot (it does nothing except trigger dataflows), but data flows are not the fastest elements.
Transferring data directly via a Data Pipeline (copy data), a Python script or other method will be much more efficient.
However, it depends on the context. Dataflows are very practical, but when there is a lot of data, they consume a lot of resources.
Feel free to give me a kudo and accept my answer as the solution if it suits you.
Have a nice day,
VivienHi rohit_motwani,
In my environment, I just add the dataflow to the pipeline and run everything through the pipeline API. This gives me a more stable system that requires less changes as there is only the one API I am interacting with, instead of also interacting with the notebook and dataflow APIs respectively.
That being said, there is an API available for dataflows: Dataflows - Refresh Dataflow - REST API (Power BI Power BI REST APIs) | Microsoft Learn
You should be able to implement it much the same way as the pipeline API.
If you found this useful, consider giving Kudos. If I solved your problem, mark this post as the solution.
Hi rohit_motwani,
You can execute a pipeline fron a notebook using the API using the following code:
import sempy.fabric as fabric
from sempy.fabric import FabricRestClient
_client = FabricRestClient()
start_resp = client.post(f"v1/workspaces/{workspace_id}/items/{item_id}/jobs/instances?jobType=Pipeline")
run_id = start_resp.headers['Location'].split('/')[-1]
sleep(15)
# Wait for visibility
while client.get(f"v1/workspaces/{workspace_id}/items/{item_id}/jobs/instances/{run_id}").status_code != 200:
print(f"Waiting for job {job_id} to become visible...")
sleep(15)
If you found this useful, consider giving Kudos. If I solved your problem, mark this post as the solution.
Hi tayloramy ,
Thanks a lot for sharing this! Yes, the code you provided works well for executing a pipeline from a notebook. My query was more around whether a similar approach is possible for executing Dataflows using Python. Could you confirm if Dataflow execution is supported via the API, or if it’s currently limited to Pipelines?
Hi rohit_motwani,
In my environment, I just add the dataflow to the pipeline and run everything through the pipeline API. This gives me a more stable system that requires less changes as there is only the one API I am interacting with, instead of also interacting with the notebook and dataflow APIs respectively.
That being said, there is an API available for dataflows: Dataflows - Refresh Dataflow - REST API (Power BI Power BI REST APIs) | Microsoft Learn
You should be able to implement it much the same way as the pipeline API.
If you found this useful, consider giving Kudos. If I solved your problem, mark this post as the solution.