Forum Discussion

Srisakthi's avatar
Srisakthi
Icon for Super User rankSuper User
1 year ago
Solved

Notebook activity in Data Pipeline

Hello All,   In my workspace I have 5 notebooks in a folder, which needs to be triggered from Fabric Data Pipeline sequentially.  But I can select only one notebook at a time in a pipeline. is ther...
Pipeline
  • NandanHegde's avatar
    NandanHegde
    1 year ago

    As you stated that the notebooks need to be executed in a particular sequence,so you need to maintain the sequnce config somewhere.

    In case if the requirement is to execute the notebooks in any sequnce but sequentailly, you can try the below route (which iI have not tested myself)

    Get list of all notebooks within the workspace via REST API leveraging the web activity
    pass that array to foreach activity

spencer_sa's avatar
spencer_sa
Icon for Impactful Individual rankImpactful Individual
1 year ago

Folders in workspaces are currently pretty much cosmetic - they don't persisted in Source Control (a separate concern).
If you didn't want to individually run notebooks you have a number of options, but most rely on having a list/array/table of notebook names;

1) Maintain a table of notebooks, perform a Lookup Activity to load them into an array, have a ForEach Activity to cycle over each notebook name executing each one.

2) Have a notebook that has this info to hand (either table, hardcoded, or API call/sempy_labs) and either uses .run/.runMultiple to excute them or passes them as a JSON string to the calling pipeline which them runs using a ForEach as per option 1) above.