JSON data is not loading into lakehouse table

[fabric_newbie]

Hi - I am running into an issue with JSON and the copydata functionality. I have JSON files coming into a lakehouse from Amazon S3. These files are then loaded into tables in the same lakehouse using copydata. Everything was going well until recently, where now the JSON data part of the JSON with all of the data elements is not being loaded. Any idea what might be happening?

Thanks

[fabric_newbie · Accepted solution]

Hi - the notebook with Spark failed with a size limitation message (although the files are not large) - but worked fine with Pandas. FYI.

[v-cboorla-msft]

Hi @fabric_newbie 

Glad that you were able to find some insights and thank you for sharing the same with the community as it can be helpful to others.

Please continue using Fabric Community on your further queries.

Thanks.

[v-cboorla-msft]

Hi @fabric_newbie 

Thanks for using Microsoft Fabric Community.

Apologies for the inconvenience that you are facing here.

There might be a couple of reasons for all the JSON data elements is not being loaded into Lakehouse. Here are some trobleshooting steps that might help you.

Data Format Mismatch: Check the schema of your Lakehouse table. Does it match the structure of your JSON data? Mismatched data types or missing columns in the table schema can lead to partial data load.

Incorrect Copy Data Settings: Verify the format settings in your Copy Data activity. Ensure "JSON" is selected as the format type. Explore options like "First row as header" if your JSON data has headers for each element.

Nested JSON Structure: If your JSON data has nested structures, Copy Data might not automatically flatten it. You might need to use a Data Factory script or a Data Engineering Notebook to transform the data before loading it.

Corrupted Data Source: There's a chance the source JSON file itself is corrupted or incomplete. Try validating the JSON file and ensure it's well-formed.

Please refer to this link that might help you : Transforming JSON to Lakehouse Tables with Microsoft Fabric Notebooks: A Step-by-Step Guide.

I hope this information helps. Please do let us know if you have any further questions.

Thanks.