Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now
We have a Fabric data factory pipeline that accepts a month parameter and then picks data from view in a Fabric datawarehouse for that specific month and loads to a Azure SQL table that belongs to an external/3rd party platform.
Our business users would like the ability to trigger this pipeline on-demand. What's the simplest way to achieve that?
Solved! Go to Solution.
Hi @VickyDev18 ,
Thanks for using Fabric Community. Currently we don't have any ETA for REST API feature. But a rough timeline is it might be available by Q2 2024. I cannot confirm you with the timeline as this might change due to some priority tasks.
Hope this is helpful. Please let me know incase of further queries.
Update: it looks as though the API is now available https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler/run-on-demand-item-job?tabs=HTT...
I note that the original question didn't request an API specifically - it is possible to allow your business users to execute a pipeline from within the Fabric experience. If you give the users contributor access to the workspace the pipeline is contained in, they'll be able to run it.
However, I'm presuming you don't want to give access to users to potentially modify the pipelines as you have created them, so the solution here is to create a separate premium/fabric capacity workspace that your business users have contributor or above access to, and create a new pipeline in there that contains an Invoke Pipeline activity referencing the original pipeline.
One thing to watch out for here is permissions - while your business users don't need access to the original workspace, they do at the very least need to have access to a Fabric Data Pipelines cloud connection that uses credentials that do have contributor access or above on the original workspace. This can result in some... interesting quirks in the pipelines UI for your business users - if they don't have access to the original workspace at all, they will see errors on the Invoke Pipeline activity about the workspace and pipeline not existing, but still be able to run it (the activity displays a GUID):
The workspace list displayed uses the user's credentials, and not the credentials for the selected connection. This means users won't be able to develop their own pipelines invoking yours, but doesn't stop them from running this apparently broken activity if you create it for them:
If you do need to enable your users to create their own Invoke Pipeline activity, you can grant them viewer access on the original workspace (the Invoke Pipeline activity now displays the pipeline name rather than GUID). If you do this, when they go to execute it, they may get a message saying they don't have access to all the connections in the pipeline if there are those downstream in your pipeline that hasn't been shared with them. Despite this, it still appears to have no problems executing (I've tested the below where the original pipeline uses a connection that the 'business user' doesn't have access):
You may wish to configure the connection shared with your business users to use some sort of dedicated service account (not a service principal - they're not yet supported by the API), otherwise if you e.g. grant access to your own connection, you may be granting them access to a lot more than they should have.
I haven't tested the API yet, but I don't think there's any reason to believe that the permissions required are going to be different to when executing via the Fabric experience
Hello @VickyDev18
I am agreement with what @NandanHegde mentioned .
But if you have a liberty of update the extsting pipeline and add a activity to check a SQL Table or may be a blob and if this activity passes the validation you can proceed with the pipeline . This pipeline will be running on may be 30mins schedule or even lower .
So once the users updated the SQL table with the right values then the pipeline will run all the steps otherwise it will only run the first step and exit .
Thanks
Himanshu
Hey,
For now the only way is to give the users access within the workspace to trigger the data pipelines on adhoc execution.
https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-runs
There is no REST API as of now to trigger them
Thanks for the response. Is it fair to assume that a REST API will be available at some point in the near future? If yes, do you happen to know any rough timelines on when that could be so I can communicate back to the business users.
Hi @VickyDev18 ,
Thanks for using Fabric Community. Currently we don't have any ETA for REST API feature. But a rough timeline is it might be available by Q2 2024. I cannot confirm you with the timeline as this might change due to some priority tasks.
Hope this is helpful. Please let me know incase of further queries.
Hi @VickyDev18 ,
We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .
Hi @VickyDev18 ,
We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .
Since a REST API is expected to available at some point, we've just decided to wait until then.
Hi @VickyDev18 ,
Glad to know that you got some insights over your query. Please continue using Fabric Community for your further queries.
User | Count |
---|---|
6 | |
6 | |
4 | |
3 | |
3 |
User | Count |
---|---|
14 | |
10 | |
7 | |
7 | |
6 |