Forum Discussion
Super UserNotebook activity in Data Pipeline
As you stated that the notebooks need to be executed in a particular sequence,so you need to maintain the sequnce config somewhere.
In case if the requirement is to execute the notebooks in any sequnce but sequentailly, you can try the below route (which iI have not tested myself)
Get list of all notebooks within the workspace via REST API leveraging the web activity
pass that array to foreach activity
Super UserHi spencer_sa , NandanHegde ,
Thanks for your reply. In both the cases i have hard code my notebook names, which i dont want to do. It should dynamically pick it up.
Regards,
Srisakthi
- NandanHegde
As you stated that the notebooks need to be executed in a particular sequence,so you need to maintain the sequnce config somewhere.
In case if the requirement is to execute the notebooks in any sequnce but sequentailly, you can try the below route (which iI have not tested myself)
Get list of all notebooks within the workspace via REST API leveraging the web activity
pass that array to foreach activity- Srisakthi
There is another path that I think is closer to what the original poster is seeking.
You can use the file naming schema to sequence notebooks, which eliminates the need to maintain a configuration details. This approach also seemlessly scales across Dev/Prod deployement landscape
General design is to build a orchestration notebook:
1. Use notebookutils to get all objects in WS
2. Filter to Notebooks objects
3. order by sequence
4. Build out your DAG
5. Call run multiple
This is the approach we took and it works well. We bundle our notebooks into phases (ie: _p01, _p02). We also put each phase in a folder but unfortuatly at this time that attribute isnt retuened by notbook utils so its just to help my team stay organized.
Here is stubbed out py code for each step. Obv with this approach you need to repeat steps 3-5 for how every many phases you wanted to define. But if you want even more flexibility you could put the notebook Dependencies in the description of each notebook which is an attribute returned by notebookutils. Then update you DAG code to include the dependencies
1 .
# Get all objects in wsartifacts_list = notebookutils.notebook.list()2.# Filter items to include only notebooksnotebooks = [item for item in artifacts_listif item['type'] == 'Notebook']3.# Filter items to include only phase01 notebooksp01_notebooks = [item for item in artifacts_listif '_p01' in item['displayName']]4.DAG_Array = {"activities": [{"name": item['displayName'],"path": item['displayName'],"timeoutPerCellInSeconds": 1800 # 90 is Default}for item in p01_notebooks],"timeoutInSeconds": 43200, # max timeout for the entire DAG 43200 (12 hrs) is Default"concurrency": 5 # max number of notebooks to run concurrently 50 is Default}5.#Run DAG Arraynotebookutils.notebook.runMultiple(DAG_Array)
