Issue with Dynamically Assigning Workspace ID in Notebook Activity in Fabric Data Factory

Hi Anonymous ,

Here is my personal opinion on your question:

To ensure that the Notebook activity correctly resolves the workspace, you should do these steps below:

1. When creating a pipeline, add a Notebook activity and configure it to point to the correct notebook within your workspace. Make sure to set the workspace and notebook parameters correctly in the activity settings.
2. You can dynamically retrieve the list of notebooks within a workspace using REST APIs. This can help in scenarios where you need to dynamically select notebooks based on certain conditions.
3. Enable high concurrency mode in your workspace settings to optimize the execution of multiple notebooks. This can help in minimizing the startup time by reusing existing Spark sessions.

Creating a config table to store notebook IDs for different environments (Dev, Test, Prod) is a good practice. This approach helps in managing and dynamically selecting the correct notebook ID based on the environment.

Here is also a ducument that you can read:

Microsoft Fabric: Dynamically Provide Notebook ID in Data Factory Pipeline

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.