Forum Discussion
Small notebook activity is very slow
I have a tiny notebook that I am using to return a string value at the beginning of my pipeline. I notice that every time I run the pipeline the notebook takes 30-40s to execute. Is there any way to speed this or cache the value? Thanks!
Hi alloowishus ,
If you are looking for ways to extract the workspace name, You can avoid using a notebook altogether by leveraging built-in pipeline context in Fabric. Fabric pipelines expose system variables such as the workspace ID and pipeline name, which can be accessed directly using dynamic expressions like@pipeline().workspaceId. While this gives you the workspace ID instantly without any execution overhead, you can easily derive the workspace name by combining it with a simple lookup or mapping (for example, from a small config table). This approach is significantly faster, more reliable, and eliminates the 30–40 second delay caused by spinning up a notebook.
Regards
10 Replies
Hi alloowishus ,
If you are looking for ways to extract the workspace name, You can avoid using a notebook altogether by leveraging built-in pipeline context in Fabric. Fabric pipelines expose system variables such as the workspace ID and pipeline name, which can be accessed directly using dynamic expressions like@pipeline().workspaceId. While this gives you the workspace ID instantly without any execution overhead, you can easily derive the workspace name by combining it with a simple lookup or mapping (for example, from a small config table). This approach is significantly faster, more reliable, and eliminates the 30–40 second delay caused by spinning up a notebook.
Regards
Only if you are willing to pay through the nose. That delay is the spark session starting up. If it were running 24x7 your notebook would be completing much faster but your bill would also be much (MUCH) higher.
Thanks, that's what I was worried about. Is there any other way to get the workspace name?
In a Dataflow Gen2 you can use Lakehouse.Contents() to harvest the workspace name.
Hi alloowishus ,
By the way, you mentioned caching.
If you don't need to regenerate the value every time, you can of course use variables or parameters as default values.But what exactly do you mean by caching?
Best regards
Unfortunately, I am using it figure out what envrionment I am in so I can't hard code it.
You can try this session tag to enable session sharing across all notebooks in the pipeline : you can also try to multiple notebooks in parallel, if that's not impacting your business logic and can definetely accelerate your running time of your pipeline, see the references below :
https://learn.microsoft.com/en-us/fabric/data-factory/notebook-activity
Hello alloowishus,
If your notebook is slow but only returns a string, you can cache the value outside the notebook to speed up your pipeline.
Steps:
import os # Ensure the folder exists os.makedirs("/dbfs/tmp", exist_ok=True) # Save the value once value = "your_string_value" with open("/dbfs/tmp/my_value.txt", "w") as f: f.write(value) # Later in your pipeline, read it quickly with open("/dbfs/tmp/my_value.txt", "r") as f: my_value = f.read() print(my_value)Why this works:
- You only write the file once.
- Reading the file in subsequent pipeline runs is almost instant.
- This avoids the 30–40s delay from running the notebook every time.
You just update the file if the string value ever changes.
Hi alloowishus
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.
Thank you.Hi alloowishus
May I check if this issue has been resolved? If not, Please feel free to contact us if you have any further questions.
Thank you
