Forum Discussion
Pipeline containing 5 notebooks. can it be parameterised
I have the following pipeline
They have to be run asynchrnously in this order
I wonder if there was a way to parameterise so you only need one notebook that runs for each item?
I guess you would need the Workspace and Notebook in another file somewhere you can use? Is there any good documentation i can look at. I want to try doing it as simply as possible
7 Replies
If you're processing them in the same way (i.e. no special logic for Dim Episode comparing to Dim Task), you generally don't need more than one notebook with parameters. Example of one of mine:
Notebook with parameters:
Pipeline task that sends parameters to this notebook:
if you really need separate notebooks, how about running the notebooks in parallel using runmultiple?
https://pl.seequality.net/run-multiple-notebooks-in-parallel-using-runmultiple-in-microsoft-fabric/
I don't want to run them in parallel. i want to run them in sequence. But I can see that there is a way of using a master notebook to run them with this. So I might be able to use that.
I was expecting a solution where I would have a notebook with each parameter needed to run each notebook. And iterate through that until it was completed
You can easily do exactly that with the solution I described. For-each task that provides parameters to the notebook inside
I have cracked it. I added a csv of the Notebooks WorkspaceName, WorkspaceID, NotebookName, NotebookID. Then I did a lookup to the csv file. Then a for each loop. Sequential. Items
@activity('GetNotebookNames').output.valueIn the for each loop added a NotebookWorkspace ID @item().workspaceIDNotebook ID @item().notebookIDThis seemed to do the trickAny particular reason why you don't process all the tables inside one notebook?
Yes. To keep each notebook as simple as possible, Otherwise it would be really long. In the same way as I would process each as a stored procedure
