Forum Discussion

moboyal1's avatar
moboyal1
Microsoft Employee
3 months ago
Solved

Fabric Eventstream: TypeConversionError when writing nested JSON to Lakehouse Delta table

Eventstream fails to write nested JSON events from Azure Event Hub to a Lakehouse Delta table, throwing `OutputDataConversionError.TypeConversionError` on the destination. This occurs even with `schemaMode: "None"` and regardless of whether the table is pre-created or auto-created by the Eventstream.
 

 

Option I've tried:
1. Let Eventstream auto-create the table
Result: Table created with mixed schema — some nested objects inferred as `struct`, others as `string` (depending on whether first event had null for that field)
- Subsequent events with populated nested objects fail** with TypeConversionError because they don't match the string column type

2. Pre-create Delta table with correct struct schema via OneLake DFS API
- Created `_delta_log/00000000000000000000.json` with all nested fields defined as struct types
- Result: Eventstream ignores the pre-existing schema and creates its own parquet files with its own inferred types. TypeConversionError persists.
 
3. Push a fully-populated "seed" event first
- Sent an event with ALL nested objects populated (no nulls) so Eventstream infers struct types for everything
- Result: TypeConversionError — possibly due to arrays within the nested structs (`device.components[]`, `claimOrder.events[]`)

4. Push simplified seed event with no arrays
- Removed all array fields (set to null), kept all nested objects populated
- Result: Still TypeConversionError

5. Change schemaMode to "None"
- Updated `eventstreamProperties.json` via API: `"schemaMode": "None"`
- Result: Still TypeConversionError. schemaMode appears to only affect schema inference display, not the actual data conversion at the destination.

6. Select different inferred schema
- The Eventstream had 5 auto-inferred schemas. 
- Result: Still TypeConversionError for few cases

Reproduction Steps
1. Create an Event Hub with nested JSON events (mix of null and populated nested objects)
2. Create an Eventstream with the Event Hub as source
3. Add a Lakehouse destination (Delta table, Json format) it was cross-workspace for my work
4. Send events where some have nested objects populated and others have them as null
5. Observe TypeConversionError in Runtime logs

Few things which I want to understand:
1. Is `schemaMode: "None"` supposed to disable type conversion at the destination, or only at the source?
2. How should we handle events where a field can be either `null` or a nested object across different messages?
3. Is there a way to disable schema enforcement on the Lakehouse destination entirely?
4. Is cross-workspace destination a factor in this issue? Eventstream is in Workspace 1 but Lakehouse is in Workspace 2.
5. What is the recommended pattern for ingesting deeply nested JSON (3-4 levels, with arrays) into a Lakehouse via Eventstream?
data engineering
lakehouse

5 Replies

  • deborshi_nag's avatar
    deborshi_nag
    Super User
    3 months ago

    Hello moboyal1 

     

    Your eventstream creates a delta table based on the schema of the first record. If the first record contains a null value, it likely treats the field as a String, and later records with objects or arrays can cause a TypeConversion error.

     

    1. Does `schemaMode: "None"` disable type conversion at the destination, or just at the source?

    Schema enforcement applies when writing data into a lakehouse destination table.

     

    Add a lakehouse destination to an eventstream - Microsoft Fabric | Microsoft Learn

     

    2. How should events be handled when a field varies between `null` and a nested object?

    Consider flattening fields using the Manage Fields transformation.

     

    Microsoft Fabric Eventstreams Overview - Microsoft Fabric | Microsoft Learn

     

    3. Is it possible to disable schema enforcement on the Lakehouse destination?

    Schema enforcement cannot be disabled.

     

    4. Does using a cross-workspace destination affect this issue? Eventstream is in Workspace 1 and Lakehouse in Workspace 2.

    Cross-workspace destination is not an issue in your case.

     

    5. What is the best approach for ingesting deeply nested JSON (3-4 levels, including arrays) into a Lakehouse through Eventstream?

    Consider using an Eventhouse as a destination, where you can configure schemas for the Eventhouse destination.

     

    Add an Eventhouse destination to an eventstream - Microsoft Fabric | Microsoft Learn

     

    • moboyal1's avatar
      moboyal1
      Microsoft Employee
      3 months ago

      How it will handle ever-evolving schema?
      I don't want to write data to Eventhouse. I want to read from EventHubs in the same format upstream pushes it and then load it into Lakehouse in same format

      • v-hjannapu's avatar
        v-hjannapu
        Community Support
        3 months ago

        Hi moboyal1,

        Thanks for the clarification.

        From the current behavior, Eventstream Lakehouse destinations expect a relatively consistent schema while writing to Delta tables. Because of that, fully dynamic nested JSON payloads (especially where fields can change between null, objects, or arrays) may result in TypeConversionError.

         

        If the requirement is to preserve the payload exactly as received from Event Hubs, one possible approach would be to first store the raw JSON as-is and then handle parsing/schema evolution downstream using Spark notebooks or other processing options where schema handling is more flexible.

        At this time, there does not seem to be an option to completely disable schema enforcement for Lakehouse destinations in Eventstream.

         

        Sharing a few related documents which may help clarify the current behavior and supported patterns:

        Lakehouse and Delta Tables - Microsoft Fabric | Microsoft Learn
        Microsoft Fabric Eventstreams Overview - Microsoft Fabric | Microsoft Learn
        Add a lakehouse destination to an eventstream - Microsoft Fabric | Microsoft Learn

        Hope this helps if you have any queries we are  happy to assist you further.
        Regards,
        Community Support Team.