topic Re: Pandas in Fabric - Data size limit in Data Engineering

Pandas in Fabric - Data size limit

frithjof_v — Sun, 28 Jul 2024 21:31:02 GMT

Hi,

What is the data size limit for working with Pandas dataframes in a Fabric notebook?

Does it depend on the F SKU size?

(E.g. F4 vs F64)

What happens if we go above the limit?

Thanks in advance!

Re: Pandas in Fabric - Data size limit

Anonymous — Mon, 29 Jul 2024 07:22:56 GMT

Hi frithjof_v ,

As I understand it, the data size limit for using Pandas data frames in a Fabric notebook is largely dependent on the memory available in the environment. the F64 has more memory and can also handle larger data frames more efficiently.

If the memory limit of the environment is exceeded, the notebook may become very slow or unresponsive. It may even encounter memory errors that cause the operation to fail.

I don't see any official documentation mentioning or describing the limitations in detail, so maybe you could consider proposing an idea and waiting for users with similar needs to vote for your idea.

Best Regards,

Ada Wang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Pandas in Fabric - Data size limit

frithjof_v — Mon, 29 Jul 2024 08:58:46 GMT

Thanks Anonymous ,

However, this is a question, not an idea suggestion.

I will wait to see if some users can help to answer my question here ☺️

Re: Pandas in Fabric - Data size limit

frithjof_v — Tue, 30 Jul 2024 05:02:23 GMT

I got some answers here: https://www.reddit.com/r/MicrosoftFabric/s/90fY07XK3r