topic Re: How to run multiple code files from a Folder in a Notebook in Data Engineering

How to run multiple code files from a Folder in a Notebook

nilendraFabric — Thu, 30 Jan 2025 23:10:46 GMT

hello all,
I have folder named "Functions" in my workspace containing 5 Python notebooks.

I need to run these notebooks together sequetially, but it has to been read from "Functions" Folder
I have tried mssparkutils.notebook.runMultiple() function

Thanks

Re: How to run multiple code files from a Folder in a Notebook

spencer_sa — Fri, 31 Jan 2025 00:21:38 GMT

If my understanding is correct, currently folders are just a way of organising items in the front end. They certainly aren't saved in source control (and cause merry hell when you deploy to a new workspace)
You could manually build a Directed Acyclic Graph using .runmultiple.
If I was set on doing things automatically from a notebook, I think what I might consider doing is;

using a notebook naming convention and prefix my function notebooks "Funct 01 - " (increasing the number each time
use sempy with FabricRESTClient and the API to list all notebooks in the workspace, filter on the prefix, sort, and stick the names into a list
run a for loop over the list using notebookutils.notebook.run() to execute them one after the other.

Re: How to run multiple code files from a Folder in a Notebook

v-nmadadi-msft — Sat, 01 Feb 2025 13:49:36 GMT

Hi @nilendraFabric ,

As mentioned by @spencer_sa folders in fabric are just a way of organizing items in workspace so as to reduce the visual clutter.
If you want to run the notebooks which exist in one folder sequentially then make sure the notebooks are named exactly as the folder name suffixed with appropriate numbers.
Then run the mssparkutils.notebook.runMultiple() function over them.

If you find this post helpful, please mark it as an "Accept as Solution" and consider giving a KUDOS.
Thanks and Regards

Re: How to run multiple code files from a Folder in a Notebook

nilendraFabric — Sat, 01 Feb 2025 14:15:00 GMT

The requirement is that there are multiple module. And each module has different scripts.

existing setup is like Function module have all the functions and so on.

so customer is not looking to change the script name.
the idea is to place them the way they are in existing non Fabric system.

thanks @spencer_sa n @v-nmadadi-msft for the reply though

Re: How to run multiple code files from a Folder in a Notebook

AndersASorensen — Sat, 01 Feb 2025 15:43:55 GMT

Hi @nilendraFabric

Try the %run command instead

Re: How to run multiple code files from a Folder in a Notebook

nilendraFabric — Sat, 01 Feb 2025 15:49:26 GMT

@AndersASorensen %run command was not working in this scenario.

runmultiple was the only option, but seems like it can't take folder path

Re: How to run multiple code files from a Folder in a Notebook

v-nmadadi-msft — Tue, 04 Feb 2025 05:22:08 GMT

Hi @nilendraFabric ,
Thanks for reaching out to the Microsoft fabric community forum.
If that is the requirement then the possible solution you could use is to use runmultiple() using DAG
NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

It may be painful to hardcode notebook paths for all the existing notebooks but this will get what you want i.e to run all the notebooks existing in one folder
Example taken from the mentioned document:

# run multiple notebooks with parameters DAG = { "activities": [ { "name": "NotebookSimple", # activity name, must be unique "path": "NotebookSimple", # notebook path "timeoutPerCellInSeconds": 90, # max timeout for each cell, default to 90 seconds "args": {"p1": "changed value", "p2": 100}, # notebook parameters }, { "name": "NotebookSimple2", "path": "NotebookSimple2", "timeoutPerCellInSeconds": 120, "args": {"p1": "changed value 2", "p2": 200} }, { "name": "NotebookSimple2.2", "path": "NotebookSimple2", "timeoutPerCellInSeconds": 120, "args": {"p1": "changed value 3", "p2": 300}, "retry": 1, "retryIntervalInSeconds": 10, "dependencies": ["NotebookSimple"] # list of activity names that this activity depends on } ], "timeoutInSeconds": 43200, # max timeout for the entire DAG, default to 12 hours "concurrency": 50 # max number of notebooks to run concurrently, default to 50 } notebookutils.notebook.runMultiple(DAG, {"displayDAGViaGraphviz": False})

If you find this post helpful, please mark it as an "Accept as Solution" and consider giving a KUDOS.
Thanks and Regards

Re: How to run multiple code files from a Folder in a Notebook

v-nmadadi-msft — Fri, 07 Feb 2025 07:03:04 GMT

Hi @nilendraFabric

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

Re: How to run multiple code files from a Folder in a Notebook

v-nmadadi-msft — Wed, 12 Feb 2025 05:15:31 GMT

Hi @nilendraFabric ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If the community members response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thanks and regards

Re: How to run multiple code files from a Folder in a Notebook

v-nmadadi-msft — Mon, 17 Feb 2025 06:10:30 GMT

Hi @nilendraFabric ,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the community members for the issue worked. If our response addressed, please mark it as Accept as solution and click Yes if you found it helpful.

Thanks and regards