Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.

Reply
Josh_Analysis
Frequent Visitor

Fabric Notebook too big

I have a Notebook that runs fine when I run it manually. But once I try to automate the notbook to run I get this error Failed to run notebook due to invalid request. [Error: Size of notebook content and exit value must be less than 8MB. Please try to write large outputs to files (Parameter 'resultSnapshot') Actual value was Microsoft.Analytics.SynapseNotebookExecService.Models.RunNotebookSnapshot.]

I've tried reducing the size of the notebook to remove the visuals and outputs, but the notebook is retrieving data from an API, loading it into a csv, I save the csv to a dataframe, transforming it, then loading the dataframe into the Lakehouse.

Any tips or advice to get the notebook to automate?

Thanks in advance. 

1 ACCEPTED SOLUTION

Hi @Josh_Analysis ,

Thanks for using Fabric Community and reporting this issue.

While going through your scenario team is in plan to extend size to 32M, same as the notebook size limit. This will increase output storage size and help users to work productively.
Unfortunately 32M may be not enough in your case, since notebook content will have a limit size and team can't extend this size to infinite.

In your case loading large data is fine, the problem is to display large dataframe in notebook output. 
Note: Display in notebook = storage a large notebook content in our service system storage

You should try to display large output in files and not directly displaying in console.

We can observe display dataframe code cell in your notebook.

When trying to run through pipeline or schedule, Notebook service will store all code cell outputs. But in manually run, it will show directly without saving outputs. That's why you are only facing this error in pipeline or schedule. The correct thing to do is check all code cells and aviod display or print large dataframe. Incase if requires to display the dataframe then you should display this output in files.! 

Hope this is helpful. Please let me know incase of any queries.

View solution in original post

5 REPLIES 5
ramonsuarez
Advocate II
Advocate II

Have you tried breaking up your notebook and then run those smaller notebooks from a master notebook using `%%run notebook-name`. This way you can spread the memory usage and show the visuals without the overhead of your massive dataframe.

v-gchenna-msft
Community Support
Community Support

Hi @Josh_Analysis ,

Thanks for using Fabric Community.

As I understand you were facing an issue - [Error: Size of notebook content and exit value must be less than 8MB. Please try to write large outputs to files (Parameter 'resultSnapshot') Actual value was Microsoft.Analytics.SynapseNotebookExecService.Models.RunNotebookSnapshot.] while retrieving data from an API.

 

Can you please share your workspace id, artifact id and activity id? We'd like to check if it's our issue or hit the error by design. It will be great if you can also share the code snippet along with it. We would like to understand why notebook requires snapshot more than 8M?

You can send us this information through email to AzCommunity[at]Microsoft[dot]com with the below details,
 

Email subject: <Attn - v-gchenna-msft  : Fabric Notebook too big>

Thread URL: Fabric Notebook too big - Microsoft Fabric Community

Thanks for the quick response. I've sent everything over. 

Hi @Josh_Analysis ,

Thank you for sharing the necessary details.
Team is looking into this issue and we will get back to you as soon as we have an update.

Hi @Josh_Analysis ,

Thanks for using Fabric Community and reporting this issue.

While going through your scenario team is in plan to extend size to 32M, same as the notebook size limit. This will increase output storage size and help users to work productively.
Unfortunately 32M may be not enough in your case, since notebook content will have a limit size and team can't extend this size to infinite.

In your case loading large data is fine, the problem is to display large dataframe in notebook output. 
Note: Display in notebook = storage a large notebook content in our service system storage

You should try to display large output in files and not directly displaying in console.

We can observe display dataframe code cell in your notebook.

When trying to run through pipeline or schedule, Notebook service will store all code cell outputs. But in manually run, it will show directly without saving outputs. That's why you are only facing this error in pipeline or schedule. The correct thing to do is check all code cells and aviod display or print large dataframe. Incase if requires to display the dataframe then you should display this output in files.! 

Hope this is helpful. Please let me know incase of any queries.

Helpful resources

Announcements
Expanding the Synapse Forums

New forum boards available in Synapse

Ask questions in Data Engineering, Data Science, Data Warehouse and General Discussion.

LearnSurvey

Fabric certifications survey

Certification feedback opportunity for the community.

April Fabric Update Carousel

Fabric Monthly Update - April 2024

Check out the April 2024 Fabric update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors