Forum Discussion
pipeline trigger using metadata table
Hi DiKi-I,
For execution order, I am using the dependencies column in my registry table. It contains a dataset like so: {bronze:1, silver:3}
this would mean the job depends on bronze job 1 and silver job 3.
Then I am building a directed acyclic graph to determine the order of jobs to run, and I am executing the jobs in that order, waiting for them to finish, then executing the next later.I also have my notebook set up to be able to run jobs concurrently, so if there's 15 jobs with no dependencies, they will all start at the same time. If you're doing something similar, keep in mind your capcaity usage. I am using F64 capacities, if you're using a smaller capacity or your pipelines are very complex, you might run into CU limits.
If you found this helpful, consider giving some kudos. If I answered your quesiton or solved your problem, mark this post as the solution so future community members can find it easily.
also this solution will work across different workspaces?
Hi DiKi-I,
The function that actually starts the job, _start_job_instance_raw, takes a workspace id, item id, and job type.
For pipelines, the job type is Pipeline.
It will start the pipeline in the specificed workspace as long as you have access to it.
As for the parameters, I have not implemented that in my solution yet, that is next on my todo list to do.
Take a look at the job scheduler documentation: https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler/run-on-demand-item-job?tabs=HTTP#run-item-job-instance-with-request-body-containing-executiondata-example
If you found this helpful, consider giving some kudos. If I answered your quesiton or solved your problem, mark this post as the solution so future community members can find it easily.
I tested the functions and its working fine. Next I will see how I can set execution order and pass parmeter. Thank you so much for your help.
Hi DiKi-I,
For execution order, I am using the dependencies column in my registry table. It contains a dataset like so: {bronze:1, silver:3}
this would mean the job depends on bronze job 1 and silver job 3.
Then I am building a directed acyclic graph to determine the order of jobs to run, and I am executing the jobs in that order, waiting for them to finish, then executing the next later.I also have my notebook set up to be able to run jobs concurrently, so if there's 15 jobs with no dependencies, they will all start at the same time. If you're doing something similar, keep in mind your capcaity usage. I am using F64 capacities, if you're using a smaller capacity or your pipelines are very complex, you might run into CU limits.
If you found this helpful, consider giving some kudos. If I answered your quesiton or solved your problem, mark this post as the solution so future community members can find it easily.