Forum Discussion
Microsoft EmployeeFabric Eventstream: TypeConversionError when writing nested JSON to Lakehouse Delta table
Hi moboyal1,
Thanks for the clarification.
From the current behavior, Eventstream Lakehouse destinations expect a relatively consistent schema while writing to Delta tables. Because of that, fully dynamic nested JSON payloads (especially where fields can change between null, objects, or arrays) may result in TypeConversionError.
If the requirement is to preserve the payload exactly as received from Event Hubs, one possible approach would be to first store the raw JSON as-is and then handle parsing/schema evolution downstream using Spark notebooks or other processing options where schema handling is more flexible.
At this time, there does not seem to be an option to completely disable schema enforcement for Lakehouse destinations in Eventstream.
Sharing a few related documents which may help clarify the current behavior and supported patterns:
Lakehouse and Delta Tables - Microsoft Fabric | Microsoft Learn
Microsoft Fabric Eventstreams Overview - Microsoft Fabric | Microsoft Learn
Add a lakehouse destination to an eventstream - Microsoft Fabric | Microsoft Learn
Hope this helps if you have any queries we are happy to assist you further.
Regards,
Community Support Team.
Microsoft EmployeeHow it will handle ever-evolving schema?
I don't want to write data to Eventhouse. I want to read from EventHubs in the same format upstream pushes it and then load it into Lakehouse in same format
Hi moboyal1,
Thanks for the clarification.
From the current behavior, Eventstream Lakehouse destinations expect a relatively consistent schema while writing to Delta tables. Because of that, fully dynamic nested JSON payloads (especially where fields can change between null, objects, or arrays) may result in TypeConversionError.
If the requirement is to preserve the payload exactly as received from Event Hubs, one possible approach would be to first store the raw JSON as-is and then handle parsing/schema evolution downstream using Spark notebooks or other processing options where schema handling is more flexible.
At this time, there does not seem to be an option to completely disable schema enforcement for Lakehouse destinations in Eventstream.
Sharing a few related documents which may help clarify the current behavior and supported patterns:
Microsoft Fabric Eventstreams Overview - Microsoft Fabric | Microsoft Learn
Add a lakehouse destination to an eventstream - Microsoft Fabric | Microsoft Learn
Hope this helps if you have any queries we are happy to assist you further.
Regards,
Community Support Team.
Hi moboyal1,
I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We are always here to support you.
Regards,
Community Support Team.Hi moboyal1,
I hope the information provided above assists you in resolving the issue. If you have any additional questions or concerns, please do not hesitate to contact us. We are here to support you and will be happy to help with any further assistance you may need.
Regards,
Community Support Team.