Forum Discussion
Error ingesting json files in warehouse table
A final update for now at least: Microsoft has confirmed this is a bug, they hope to fix in the 1st quarter of 2026.
In the mean time they suggest to ingest it as a string and than use a notebook to parse the json array back into its original columns, or change the source, which for us wasn't an option.
Since it took ages for them to got to this stage I had already figured out I could use a notebook to read the json files in the lakehouse, read them into a dataframe, and when all files were read, write that dataframe to a table, in my case with an option "overwrite" since this is just a raw staging table.Final kicker: They want me to give permission to archive the case (no doubt because of their KPI's), but the bug is not registered on MS's known issue list. I want them to document it there, but they for some reason don't want to do that, they want to keep it internal, so now my Dutch stubberness is playing up .
The python code was quite simple in the end, even though we had multiple nested arrays. I used co-pilot to do the heavy lifting for me since i had to alias about 30 columns since i had to put the data in an existing table and the names needed to match
These are the libraries I needed:
Folder path to read json files
Exploding the main body of the json (just part of it, as example)
Thats it in a nutshell
Hi All,
Just a further update , that is not really an update. Due to the amount of time it is taking something that wasn't urgen, now is becomming urgent and I may have no option than to tackle one of our biggest and most datasets and add in some form or python notebook, that can take a json array, that now needs to be loaded in a single column and flatten it into its respective columns like it just to be.
MS assured me Friday, after i pointed this out and asked for an escalation, it is already escalated to the "Product Group Team" . However I wont have time to wait months for them to figure this out.
This change will be massive, since the dataset goes back over 10 years and data can not be retrieved again from the source if i get this wrong . So it means backup plans, change requests, risk etc.😥
Hi smeetsh,
Thank you for the update. I understand the urgency and impact this has on your workload. Since the issue has already been escalated to the Product Group, I’d recommend continuing to monitor the escalation progress.
