Fabric Notebook too big

[Josh_Analysis]

I have a Notebook that runs fine when I run it manually. But once I try to automate the notbook to run I get this error Failed to run notebook due to invalid request. [Error: Size of notebook content and exit value must be less than 8MB. Please try to write large outputs to files (Parameter 'resultSnapshot') Actual value was Microsoft.Analytics.SynapseNotebookExecService.Models.RunNotebookSnapshot.]

I've tried reducing the size of the notebook to remove the visuals and outputs, but the notebook is retrieving data from an API, loading it into a csv, I save the csv to a dataframe, transforming it, then loading the dataframe into the Lakehouse.

Any tips or advice to get the notebook to automate?

Thanks in advance. 

[ramonsuarez]

Have you tried breaking up your notebook and then run those smaller notebooks from a master notebook using `%%run notebook-name`. This way you can spread the memory usage and show the visuals without the overhead of your massive dataframe.

[Anonymous]

Hi @Josh_Analysis ,

Thanks for using Fabric Community.

As I understand you were facing an issue - [Error: Size of notebook content and exit value must be less than 8MB. Please try to write large outputs to files (Parameter 'resultSnapshot') Actual value was Microsoft.Analytics.SynapseNotebookExecService.Models.RunNotebookSnapshot.] while retrieving data from an API.

Can you please share your workspace id, artifact id and activity id? We'd like to check if it's our issue or hit the error by design. It will be great if you can also share the code snippet along with it. We would like to understand why notebook requires snapshot more than 8M?

You can send us this information through email to AzCommunity[at]Microsoft[dot]com with the below details,
 

Email subject: <Attn - v-gchenna-msft  : Fabric Notebook too big>

Thread URL: Fabric Notebook too big - Microsoft Fabric Community

[Josh_Analysis]

Thanks for the quick response. I've sent everything over. 

[Anonymous]

Hi @Josh_Analysis ,

Thank you for sharing the necessary details.
Team is looking into this issue and we will get back to you as soon as we have an update.

[Anonymous · Accepted solution]

Hi @Josh_Analysis ,

Thanks for using Fabric Community and reporting this issue.

While going through your scenario team is in plan to extend size to 32M, same as the notebook size limit. This will increase output storage size and help users to work productively.
Unfortunately 32M may be not enough in your case, since notebook content will have a limit size and team can't extend this size to infinite.

In your case loading large data is fine, the problem is to display large dataframe in notebook output. 
Note: Display in notebook = storage a large notebook content in our service system storage

You should try to display large output in files and not directly displaying in console.

We can observe display dataframe code cell in your notebook.

When trying to run through pipeline or schedule, Notebook service will store all code cell outputs. But in manually run, it will show directly without saving outputs. That's why you are only facing this error in pipeline or schedule. The correct thing to do is check all code cells and aviod display or print large dataframe. Incase if requires to display the dataframe then you should display this output in files.! 

Hope this is helpful. Please let me know incase of any queries.