Data Pipeline variable is too large to pass to a notebook

[DeborahCarter]

Hello!  I am puling data from an API which returns a TON of data.  I set the data to a variable in a data pipeline which I tried to pass into a notebook to parse.  The variable exceeds the max size for notebooks.  I need a way to parse the data into a table.  I tried a dataflow gen2 but it doesn't see the variable.  I tried to use the copy data going directly from the API call but the data needs to be converted into a base64 string before it can be used and there didn't appear to be a way to convert it in the copy data activity.  Does anyone have a suggestion as to how I can process the variable?

Thank you for the suggestions.

[v-tsaipranay]

Hi @DeborahCarter ,

Thank you for reaching out to the Microsoft Fabric Community forum.

The issue you're encountering is due to a known limitation. Fabric pipeline variables have a size limit and aren't suitable for passing large API responses into notebooks.

To resolve this, we recommend moving the API call logic directly into the notebook itself. This avoids the variable size constraint and allows you to handle, parse, and transform large datasets more efficiently. You can then write the processed data directly to a Lakehouse table. Use the Notebook Activity within your pipeline to trigger this notebook, which maintains your orchestration while bypassing variable limitations.

For testing, you can simulate the API response with mock JSON data in the notebook, then later replace it with a real API call using Python’s requests library. This approach is scalable, aligns with Fabric best practices, and ensures smooth integration with your data flows.

Please refer to the following documentation for detailed information: Notebook activity - Microsoft Fabric | Microsoft Learn

Hope this helps. Please reach out for further assistance.

If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.

Thank you.