Forum Discussion
Solved
MS Fabric Notebook: Error when using dataframe.to_json() with Fabric Lakehouse storage
Hi all, I'm getting the error below when using pandas dataframe method to_json() with dataframe.size() "4263942". Dataframes with lower size can be saved without issue. The filepath and name are j...
Hi,
I switched from JSON to Parquet and had no issue with storing the data. So for the moment I circumvented the issue.
Hi Anonymous ,
Thanks for your help.
The computer in this case is the Fabric instance (Premium P1 node). There's not much to configure as it runs as Software as a Service.
The other JSON files that are sucessfully written to the Lakehouse storage are up to 119 MB in size. So the 4MB limit you mentioned seems not to be valid.
I'll check out the streaming library approach.
Hi trutz ,
I have reached the internal team for help regarding this issue. I will update you once I hear from them.