Documentation (or sample) for running pipeline from notebook?

[dbeavon3]

It sounds simple but I haven't found any explanation of how to run a pipeline (a single-activity pipeline) from a notebook.

Currently I can run a notebook from a pipeline.  But not the other way around.

I believe that the docs about the "job scheduler" may be relevant but I've reviewed them several times and don't see an meaningful examples:

"job scheduler"

https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler

https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler/run-on-demand-item-job?tabs=HTT...
(Run On Demand Item Job)

I suspect these docs were written by someone who does NOT understand the purpose of the "job scheduler" api, or perhaps they are not a native english-speaker.  There is no introduction or overview in these docs.  And for any given operation (eg Run-On-Demand-Item-Job) there is nothing to explain the options available for "job type", and nothing that explains how to prepare a "request body containing executionData".

It is almost like this section of docs is just a placeholder, where we may find some meaningful information some day in the future. 

Has anyone tried this API yet?  Can someone please share a small example of how to preemptively execute a data-pipeline from a single cell in a notebook, ideally the notebook can continue execution without losing the flow-control.   Ideally the notebook could block and wait for the completion of the pipeline, and then do some follow-up work afterwards.

[v-karpurapud]

Hi @dbeavon3 

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[dbeavon3]

I managed to find a couple more links  (THANKS REDDIT!)

https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-rest-api#run-on-demand-item-job

https://www.reddit.com/r/MicrosoftFabric/comments/1gxkq8h/quick_knowledge_bomb_fabric_job_scheduler_...

Here is an example of running a pipeline, for anyone else who may want to see one:

[v-karpurapud · Accepted solution]

Hi @dbeavon3 
Hope you are doing well!

We appreciate your insights on this topic, Running a pipeline from a notebook. We noticed that the reference document provided was updated today and is still in preview mode. There seems to be no practical implementation provide in official documentations or any other sources as of now. We will consider your valuable input and inform you once it is fully functional. 

I found the below blog please go through it once.
How Execute a Data Factory Pipeline from a Notebook in Fabric – Richard Mintz's BI Blog
 
I kindly request you to submit your ideas and feedback here Microsoft Fabric Ideas feedback submitted here is often reviewed by the product teams and can lead to meaningful improvement.

If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.

Thank You.

[dbeavon3]

Hi @v-karpurapud 

Thanks for pointing out that this is stil in preview.  That never came up in the prior discussion on Reddit.

Are you aware how long it will take for this to get to GA?  I think it is unfortunate that the Fabric team has regularly put stuff out in an unfinished state, and it can take years before things go to GA.  It feels like a state of purgatory, and we don't know whether we have support or not, and we don't know whether our solutions will need to be re-built from scratch or not.

.... The alternative is that Microsoft could be keeping their "preview" features to themselves.  Or they could publish them for internal use only, (their fellow Microsoft teams should work out the kinks until the software is good enough for a GA)

[v-karpurapud]

Hello @dbeavon3 

The discontent is understandable since there are no clear answers concerning a GA timeline. Unfortunately, we are still unsure about the time frame for getting General Availability status for this feature. Nevertheless, your feedback and use of our preview features become very important. Whenever you use the features and report any issues or bugs, you help us improve the product and make it better for everybody.

Thanks for your patience and support during these times because your input is essential to making sure that when the feature does reach GA, it has met very high quality and reliability standards. Thank you for being part of this journey with us!