Forum Discussion
Notebook activity in Data Pipeline
As you stated that the notebooks need to be executed in a particular sequence,so you need to maintain the sequnce config somewhere.
In case if the requirement is to execute the notebooks in any sequnce but sequentailly, you can try the below route (which iI have not tested myself)
Get list of all notebooks within the workspace via REST API leveraging the web activity
pass that array to foreach activity
As you stated that the notebooks need to be executed in a particular sequence,so you need to maintain the sequnce config somewhere.
In case if the requirement is to execute the notebooks in any sequnce but sequentailly, you can try the below route (which iI have not tested myself)
Get list of all notebooks within the workspace via REST API leveraging the web activity
pass that array to foreach activity
There is another path that I think is closer to what the original poster is seeking.
You can use the file naming schema to sequence notebooks, which eliminates the need to maintain a configuration details. This approach also seemlessly scales across Dev/Prod deployement landscape
General design is to build a orchestration notebook:
1. Use notebookutils to get all objects in WS
2. Filter to Notebooks objects
3. order by sequence
4. Build out your DAG
5. Call run multiple
This is the approach we took and it works well. We bundle our notebooks into phases (ie: _p01, _p02). We also put each phase in a folder but unfortuatly at this time that attribute isnt retuened by notbook utils so its just to help my team stay organized.
Here is stubbed out py code for each step. Obv with this approach you need to repeat steps 3-5 for how every many phases you wanted to define. But if you want even more flexibility you could put the notebook Dependencies in the description of each notebook which is an attribute returned by notebookutils. Then update you DAG code to include the dependencies
1 .