Notebook activity in Data Pipeline

Folders in workspaces are currently pretty much cosmetic - they don't persisted in Source Control (a separate concern).
If you didn't want to individually run notebooks you have a number of options, but most rely on having a list/array/table of notebook names;

1) Maintain a table of notebooks, perform a Lookup Activity to load them into an array, have a ForEach Activity to cycle over each notebook name executing each one.

2) Have a notebook that has this info to hand (either table, hardcoded, or API call/sempy_labs) and either uses .run/.runMultiple to excute them or passes them as a JSON string to the calling pipeline which them runs using a ForEach as per option 1) above.

You can design a meta data driven framework rather than multiple notebook activities as you have the option of callng the notebook dynamically:

So you can follow the below frame :

1) Create a parameter called NotebookSeq

and it would have notebook values semicolon seperated as below :

abc;def,ghi

2) use for each activity and make it sequential

and expression would be @split(parameter,';')

3) within for each, use the notebook activity and in notebook add dynamic expression as Item()

So whenever there are new notebooks or deletions or change of sequence, you just need to update the parameter value

Hi spencer_sa  , NandanHegde ,

Thanks for your reply. In both the cases i have hard code my notebook names, which i dont want to do. It should dynamically pick it up.

Regards,

Srisakthi