Forum Discussion

smeetsh's avatar
smeetsh
Continued Contributor
3 months ago
Solved

Fabric having issues importing or ingesting json files and responses

Good morning All, We ingest a lot of data using the API endpoint of our third parties, and I have build a lot of pipelines doing just that I am working with a new API since friday and I am having...
  • v-hashadapu's avatar
    v-hashadapu
    3 months ago

    Hi smeetsh , Yeah, I think the issue is with the JSON complexity rather than the payload size too. Fabric generally handles large JSON files fine, but deeply nested and irregular schemas with optional arrays/objects can make the automatic schema import struggle or hang. The fact that even the smaller endpoint failed during schema import points more toward a schema inference limitation than a pure performance problem.

     

    Your notebook -> Lakehouse -> ETL/Warehouse approach is probably the better fit here anyway. Spark gives you much more control over flattening arrays, handling nullable branches and managing evolving schemas compared to relying on automatic inference in the import UI.