data integration
2 Topics
Fabric Notebook too big
I have a Notebook that runs fine when I run it manually. But once I try to automate the notbook to run I get this error Failed to run notebook due to invalid request. [Error: Size of notebook content and exit value must be less than 8MB. Please try to write large outputs to files (Parameter 'resultSnapshot') Actual value was Microsoft.Analytics.SynapseNotebookExecService.Models.RunNotebookSnapshot.] I've tried reducing the size of the notebook to remove the visuals and outputs, but the notebook is retrieving data from an API, loading it into a csv, I save the csv to a dataframe, transforming it, then loading the dataframe into the Lakehouse. Any tips or advice to get the notebook to automate? Thanks in advance.Solved5.8KViews0likes5Comments
What are the most effective Activities you use when transforming data via a pipeline?
I am relativey new to Azure Synapse and I'm trying to build some pipelines. There is a wide variety of activities to choose from. What are the most effective activities you use when building pipelines?. For example there is DataBricks Notebook and Synapse Notebook. I'm not sure which one to choose but I would imagine one of those notebooks will be enough to transform an entire dataset in a single script.Solved1.4KViews0likes2Comments