Forum Discussion
Error ingesting json files in warehouse table
A final update for now at least: Microsoft has confirmed this is a bug, they hope to fix in the 1st quarter of 2026.
In the mean time they suggest to ingest it as a string and than use a notebook to parse the json array back into its original columns, or change the source, which for us wasn't an option.
Since it took ages for them to got to this stage I had already figured out I could use a notebook to read the json files in the lakehouse, read them into a dataframe, and when all files were read, write that dataframe to a table, in my case with an option "overwrite" since this is just a raw staging table.Final kicker: They want me to give permission to archive the case (no doubt because of their KPI's), but the bug is not registered on MS's known issue list. I want them to document it there, but they for some reason don't want to do that, they want to keep it internal, so now my Dutch stubberness is playing up .
The python code was quite simple in the end, even though we had multiple nested arrays. I used co-pilot to do the heavy lifting for me since i had to alias about 30 columns since i had to put the data in an existing table and the names needed to match
These are the libraries I needed:
Folder path to read json files
Exploding the main body of the json (just part of it, as example)
Thats it in a nutshell
Hi smeetsh,
The error "Not able to get enumerator for non-array. Path: databody.daysQuality.testResults" usually means the mapping expects an array at that JSON path, but at least one incoming file has a null, a single object, or some other non-array value there. In other words, the source shape changed and the Copy activity can no longer iterate it (inference based on how Fabric parses hierarchical JSON).
I'd suggest trying to flatten the column and treat it as a string, that way it will load nicely regardless if it is null or not.
- Open the Copy activity > Mappings > expand Column flatten settings and, just for the problematic column, enable Treat array as string or Treat struct as string. This bypasses enumeration and lands the raw value as text that you can parse downstream. See the official mapping options for arrays/structs in Fabric’s Copy activity.
For the skip incompatible rows setting, this is for row-level incompatibilities between source rows and the sink schema (type mismatches, PK violations, etc.). It can keep a load going and log skipped rows, but it won’t fix a fundamental JSON shape mismatch that throws earlier during parsing. Use it only if you can safely ignore some bad rows and review the session log.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this as the solution.
I want to get back to this: " In other words, the source shape changed and the Copy activity can no longer iterate it (inference based on how Fabric parses hierarchical JSON)."
I have gone back and can say that
A: The source has not changed, there have always been arrays as null depening on the part of the month/week we were in and
B: We are now seeing this with two very different API's from different vendors
smeetsh
By two different APIs, do you mean that Fabric and another platform is experiencing this, or that you're eperiencing this from two different sources in Fabric?I do a lot of work with JSON data and I've not experienced this, but I've always flattened out my JSON as a string when loading it into staging, and then I re JSON-ify it in my notebooks while doing transformations to handle any schema shift.
Are you able to try this approach?
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this as the solution.Hi tayloramy With two different API's I mean using Fabric we address multiple different API's , 2 of those are now showing this issue. I have gone back to the start of this year since we had some json older files and they regulalry had the null in it one day and data the other. I will have to have a look what we can do with flattening the jso, but that could be a major re-write of the ingest and ETL of the affected pipelines.
PS I have created a case with MS, but I have little hope from past experience.