Forum Discussion
Eventstream Reading from Event Hub with Missing Columns in Lakehouse Ingestion
I'm using Eventstream to read JSON messages from Azure Event Hub. The data in Eventstream Hub appears correctly after multiple refreshes. However, when I ingest the data into the lakehouse or Eventhouse, one of the columns is not ingested and always shows as null, even though it exists in the stream hub. I checked the data type of this column; sometimes it appears as a number or datetime only inside eventstream, when its type is supposed to be dynamic when I checked it in azure explorer in eventhub.
4 Replies
Hi raniasabdullah ,
Please check for violations of the following restrictions:
- You can't change an existing column type. If you try to map to a column having a different format, you may end up with empty columns.
-
Some data format mappings (Parquet, JSON, and Avro) support simple ingest-time transformations. To apply mapping transformations, create or update a column in the Edit columns window.
Mapping transformations can be performed on a column of type string or datetime, with the source having data type int or long. Supported mapping transformations are:
- DateTimeFromUnixSeconds
- DateTimeFromUnixMilliseconds
- DateTimeFromUnixMicroseconds
- DateTimeFromUnixNanoseconds
Ensure that the content-type and headers are correctly set when sending JSON messages to the Event Hub.
For more details, please refer:
Get data from Azure Event Hubs - Microsoft Fabric | Microsoft Learn
Reading-data-from-event-hubs-into-eventstream-and-a-lakehouse-table
Best Regards,
Neeko Tang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hi raniasabdullah ,
Thank you for your feedback.
This is a known issue, and our team is actively working on resolving it.
Thanks,
Wenyang
The tool is highly efficient, with the only remaining task being the capability to read and ingest dynamic data. I’m looking forward to utilizing and accessing these features.
Thank you for your response regarding the Eventhouse KQL; it is working on my end. However, in the Lakehouse, one of the dynamic columns in the message is missing one of its direct child elements.
