topic Eventstream Reading from Event Hub with Missing Columns in Lakehouse Ingestion in Eventstream

Eventstream Reading from Event Hub with Missing Columns in Lakehouse Ingestion

raniasabdullah — Thu, 26 Sep 2024 09:42:17 GMT

I'm using Eventstream to read JSON messages from Azure Event Hub. The data in Eventstream Hub appears correctly after multiple refreshes. However, when I ingest the data into the lakehouse or Eventhouse, one of the columns is not ingested and always shows as null, even though it exists in the stream hub. I checked the data type of this column; sometimes it appears as a number or datetime only inside eventstream, when its type is supposed to be dynamic when I checked it in azure explorer in eventhub.

Re: Eventstream Reading from Event Hub with Missing Columns in Lakehouse Ingestion

Anonymous — Fri, 27 Sep 2024 02:26:00 GMT

Hi raniasabdullah ,

Please check for violations of the following restrictions:

You can't change an existing column type. If you try to map to a column having a different format, you may end up with empty columns.
Some data format mappings (Parquet, JSON, and Avro) support simple ingest-time transformations. To apply mapping transformations, create or update a column in the Edit columns window.

Mapping transformations can be performed on a column of type string or datetime, with the source having data type int or long. Supported mapping transformations are:
- DateTimeFromUnixSeconds
- DateTimeFromUnixMilliseconds
- DateTimeFromUnixMicroseconds
- DateTimeFromUnixNanoseconds

Ensure that the content-type and headers are correctly set when sending JSON messages to the Event Hub.

For more details, please refer:

Get data from Azure Event Hubs - Microsoft Fabric | Microsoft Learn

Reading-data-from-event-hubs-into-eventstream-and-a-lakehouse-table

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.

Re: Eventstream Reading from Event Hub with Missing Columns in Lakehouse Ingestion

Anonymous — Thu, 10 Oct 2024 06:21:28 GMT

Hi raniasabdullah ,

Thank you for your feedback.

This is a known issue, and our team is actively working on resolving it.

Thanks,

Wenyang

Re: Eventstream Reading from Event Hub with Missing Columns in Lakehouse Ingestion

raniasabdullah — Sun, 13 Oct 2024 07:26:48 GMT

The tool is highly efficient, with the only remaining task being the capability to read and ingest dynamic data. I’m looking forward to utilizing and accessing these features.

Re: Eventstream Reading from Event Hub with Missing Columns in Lakehouse Ingestion

raniasabdullah — Sun, 13 Oct 2024 07:27:48 GMT

Thank you for your response regarding the Eventhouse KQL; it is working on my end. However, in the Lakehouse, one of the dynamic columns in the message is missing one of its direct child elements.