Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, get a free DP-600 exam voucher to use by the end of 2024. Register now

Reply
Josh_Analysis
Frequent Visitor

Fabric Notebook too big

I have a Notebook that runs fine when I run it manually. But once I try to automate the notbook to run I get this error Failed to run notebook due to invalid request. [Error: Size of notebook content and exit value must be less than 8MB. Please try to write large outputs to files (Parameter 'resultSnapshot') Actual value was Microsoft.Analytics.SynapseNotebookExecService.Models.RunNotebookSnapshot.]

I've tried reducing the size of the notebook to remove the visuals and outputs, but the notebook is retrieving data from an API, loading it into a csv, I save the csv to a dataframe, transforming it, then loading the dataframe into the Lakehouse.

Any tips or advice to get the notebook to automate?

Thanks in advance. 

1 ACCEPTED SOLUTION
Anonymous
Not applicable

Hi @Josh_Analysis ,

Thanks for using Fabric Community and reporting this issue.

While going through your scenario team is in plan to extend size to 32M, same as the notebook size limit. This will increase output storage size and help users to work productively.
Unfortunately 32M may be not enough in your case, since notebook content will have a limit size and team can't extend this size to infinite.

In your case loading large data is fine, the problem is to display large dataframe in notebook output. 
Note: Display in notebook = storage a large notebook content in our service system storage

You should try to display large output in files and not directly displaying in console.

We can observe display dataframe code cell in your notebook.

When trying to run through pipeline or schedule, Notebook service will store all code cell outputs. But in manually run, it will show directly without saving outputs. That's why you are only facing this error in pipeline or schedule. The correct thing to do is check all code cells and aviod display or print large dataframe. Incase if requires to display the dataframe then you should display this output in files.! 

Hope this is helpful. Please let me know incase of any queries.

View solution in original post

5 REPLIES 5
ramonsuarez
Advocate II
Advocate II

Have you tried breaking up your notebook and then run those smaller notebooks from a master notebook using `%%run notebook-name`. This way you can spread the memory usage and show the visuals without the overhead of your massive dataframe.

Anonymous
Not applicable

Hi @Josh_Analysis ,

Thanks for using Fabric Community.

As I understand you were facing an issue - [Error: Size of notebook content and exit value must be less than 8MB. Please try to write large outputs to files (Parameter 'resultSnapshot') Actual value was Microsoft.Analytics.SynapseNotebookExecService.Models.RunNotebookSnapshot.] while retrieving data from an API.

 

Can you please share your workspace id, artifact id and activity id? We'd like to check if it's our issue or hit the error by design. It will be great if you can also share the code snippet along with it. We would like to understand why notebook requires snapshot more than 8M?

You can send us this information through email to AzCommunity[at]Microsoft[dot]com with the below details,
 

Email subject: <Attn - v-gchenna-msft  : Fabric Notebook too big>

Thread URL: Fabric Notebook too big - Microsoft Fabric Community

Thanks for the quick response. I've sent everything over. 

Anonymous
Not applicable

Hi @Josh_Analysis ,

Thank you for sharing the necessary details.
Team is looking into this issue and we will get back to you as soon as we have an update.

Anonymous
Not applicable

Hi @Josh_Analysis ,

Thanks for using Fabric Community and reporting this issue.

While going through your scenario team is in plan to extend size to 32M, same as the notebook size limit. This will increase output storage size and help users to work productively.
Unfortunately 32M may be not enough in your case, since notebook content will have a limit size and team can't extend this size to infinite.

In your case loading large data is fine, the problem is to display large dataframe in notebook output. 
Note: Display in notebook = storage a large notebook content in our service system storage

You should try to display large output in files and not directly displaying in console.

We can observe display dataframe code cell in your notebook.

When trying to run through pipeline or schedule, Notebook service will store all code cell outputs. But in manually run, it will show directly without saving outputs. That's why you are only facing this error in pipeline or schedule. The correct thing to do is check all code cells and aviod display or print large dataframe. Incase if requires to display the dataframe then you should display this output in files.! 

Hope this is helpful. Please let me know incase of any queries.

Helpful resources

Announcements
November Carousel

Fabric Community Update - November 2024

Find out what's new and trending in the Fabric Community.

Live Sessions with Fabric DB

Be one of the first to start using Fabric Databases

Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.

November Update

Fabric Monthly Update - November 2024

Check out the November 2024 Fabric update to learn about new features.

Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Early Bird pricing ends December 9th.