Eventstream Reading from Event Hub with Missing Columns in Lakehouse Ingestion

[raniasabdullah]

I'm using Eventstream to read JSON messages from Azure Event Hub. The data in Eventstream Hub appears correctly after multiple refreshes. However, when I ingest the data into the lakehouse or Eventhouse, one of the columns is not ingested and always shows as null, even though it exists in the stream hub. I checked the data type of this column; sometimes it appears as a number or datetime only inside eventstream, when its type is supposed to be dynamic when I checked it in azure explorer in eventhub.

[Wenyang]

Hi @raniasabdullah ,

Thank you for your feedback.

This is a known issue, and our team is actively working on resolving it.

Thanks,

Wenyang

[raniasabdullah]

Thank you for your response regarding the Eventhouse KQL; it is working on my end. However, in the Lakehouse, one of the dynamic columns in the message is missing one of its direct child elements.

[raniasabdullah]

The tool is highly efficient, with the only remaining task being the capability to read and ingest dynamic data. I’m looking forward to utilizing and accessing these features.

[Anonymous]

Hi @raniasabdullah ,

Please check for violations of the following restrictions:

• You can't change an existing column type. If you try to map to a column having a different format, you may end up with empty columns.

• Some data format mappings (Parquet, JSON, and Avro) support simple ingest-time transformations. To apply mapping transformations, create or update a column in the Edit columns window.

  Mapping transformations can be performed on a column of type string or datetime, with the source having data type int or long. Supported mapping transformations are:

  • DateTimeFromUnixSeconds
  • DateTimeFromUnixMilliseconds
  • DateTimeFromUnixMicroseconds
  • DateTimeFromUnixNanoseconds

Ensure that the content-type and headers are correctly set when sending JSON messages to the Event Hub.

For more details, please refer:

Get data from Azure Event Hubs - Microsoft Fabric | Microsoft Learn

Reading-data-from-event-hubs-into-eventstream-and-a-lakehouse-table

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.