Forum Discussion

trutz's avatar
trutz
Advocate IV
2 years ago
Solved

MS Fabric Notebook: Error when using dataframe.to_json() with Fabric Lakehouse storage

Hi all, I'm getting the error below when using pandas dataframe method to_json() with dataframe.size() "4263942". Dataframes with lower size can be saved without issue. The filepath and name are j...
data engineering
Data Integration
one lake
  • trutz's avatar
    trutz
    2 years ago

    Hi,
    I switched from JSON to Parquet and had no issue with storing the data. So for the moment I circumvented the issue.

trutz's avatar
trutz
Advocate IV
2 years ago

In the mean time I switched the method to "dataframe.to_parquet()" to be able to continue with the data exploration. An update on the bug would be helpful nontheless. Thanks for your help so far.

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi   ,

    The internal team has updated me that there is no such issue at their side. They are able to repro your scenario successfully using 5M records. I have attached the screenshots for your reference.
    So, can you please confirm is this a consistent error or a one time glitch which throwed due to Internal Server Error.


     

    • trutz's avatar
      trutz
      Advocate IV
      2 years ago

      I received this error on consecutive days.
      I used the orient=records option. Maybe this got soemthing to do with the error.
      Maybe my JSON data contains a character that is not properly escaped by the pandas library and then leads to errors in the underlying libraries.
      Unfortunately the data is sensitive, so I can't provide an example.

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi ,


        Apologies for the issue you are facing. To eliminate the data issue, you can try validating the json to see if data is in proper json format. This might help us in identifying if the issue is from data or from ADLFS package.