topic Re: Pipeline containing 5 notebooks. can it be parameterised in Data Engineering

Pipeline containing 5 notebooks. can it be parameterised

DebbieE — Wed, 28 Aug 2024 10:19:29 GMT

I have the following pipeline

They have to be run asynchrnously in this order

I wonder if there was a way to parameterise so you only need one notebook that runs for each item?

I guess you would need the Workspace and Notebook in another file somewhere you can use? Is there any good documentation i can look at. I want to try doing it as simply as possible

Re: Pipeline containing 5 notebooks. can it be parameterised

frithjof_v — Wed, 28 Aug 2024 10:29:26 GMT

Any particular reason why you don't process all the tables inside one notebook?

Re: Pipeline containing 5 notebooks. can it be parameterised

Dronec — Wed, 28 Aug 2024 12:02:22 GMT

If you're processing them in the same way (i.e. no special logic for Dim Episode comparing to Dim Task), you generally don't need more than one notebook with parameters. Example of one of mine:

Notebook with parameters:

Pipeline task that sends parameters to this notebook:

Re: Pipeline containing 5 notebooks. can it be parameterised

AndyDDC — Wed, 28 Aug 2024 21:56:22 GMT

if you really need separate notebooks, how about running the notebooks in parallel using runmultiple?

https://pl.seequality.net/run-multiple-notebooks-in-parallel-using-runmultiple-in-microsoft-fabric/

Re: Pipeline containing 5 notebooks. can it be parameterised

DebbieE — Thu, 29 Aug 2024 11:12:56 GMT

Yes. To keep each notebook as simple as possible, Otherwise it would be really long. In the same way as I would process each as a stored procedure

Re: Pipeline containing 5 notebooks. can it be parameterised

DebbieE — Thu, 29 Aug 2024 11:20:11 GMT

I don't want to run them in parallel. i want to run them in sequence. But I can see that there is a way of using a master notebook to run them with this. So I might be able to use that.

I was expecting a solution where I would have a notebook with each parameter needed to run each notebook. And iterate through that until it was completed

Re: Pipeline containing 5 notebooks. can it be parameterised

Dronec — Thu, 29 Aug 2024 11:33:55 GMT

You can easily do exactly that with the solution I described. For-each task that provides parameters to the notebook inside

Re: Pipeline containing 5 notebooks. can it be parameterised

DebbieE — Fri, 13 Sep 2024 08:13:48 GMT

I have cracked it. I added a csv of the Notebooks WorkspaceName, WorkspaceID, NotebookName, NotebookID. Then I did a lookup to the csv file. Then a for each loop. Sequential. Items

@activity('GetNotebookNames').output.value

In the for each loop added a Notebook

Workspace ID @item().workspaceID

Notebook ID @item().notebookID

This seemed to do the trick