Eventhouse data transformation

[svk666]

Hi, I'm new to Fabric and I was learning Eventhouse.

I just wanted to know if I ingest data into Eventhouse and I was to make some transformation on that data, how do I do it?

Should I create a shortcut of that eh kql db into a lakehouse, attach it on a notebook, make some transformations and then dump it into a ware house?

And in a scenario like this where data is continuously ingesting into the EH, should I be using spark structured streaming?

Please let me know the correct procedure and best one as market standard.

[suparnababu8 · Accepted solution]

Hello @svk666 

If you're ingetsing the live data in to event house, you no need to create EH shortcut and no neeed to go for Lakehouse for tarsformation. 

In evenhous eit self we have KQL, by using this you can transform the data, Once the data is transfomed you can build the dashboard from eventhouse itself. Pls read this thread https://community.fabric.microsoft.com/t5/Real-Time-Intelligence-Community/Eventhouse-and-its-best-p... it might helps you

Thank you!

Did I answer your question? Mark my post as a solution!

Proud to be a Super User!

[v-menakakota]

Hi  @svk666  ,
Thanks for reaching out to the Microsoft fabric community forum. 

I would also take a moment to thank @Srisakthi  and @suparnababu8   , for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference. 
I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you.

Best Regards, 
Community Support Team

[v-menakakota]

Hi @svk666 ,

I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you.

Best Regards, 
Community Support Team

[Olufemi7]

Hello @svk666,

In Eventhouse you normally just transform the data there instead of moving it into Lakehouse or using Spark. Since it runs on the Kusto engine, it already handles ingestion and querying in real time.
You can use materialized views if you need pre-aggregated or continuously updated data, or update policies when you want transformation to happen as data is coming in. KQL functions are useful when you want to reuse logic across queries.
Spark or Lakehouse isn’t really needed here unless you’re doing more batch-style processing or ML work that Eventhouse isn’t meant for.
Docs
Eventhouse overview 
Materialized views 
Update policy overview 
Key point
Eventhouse already supports real-time transformation, so most of the time you don’t need to move the data out unless there’s a specific downstream need.

[Srisakthi · Accepted solution]

Hi @svk666 ,

There are multiple ways in which you can do transformation either you can use event stream or directly ingesting to event house as per @suparnababu8 . If there is any minimal transformation you can leverage predefined options in event stream, 

1. You can leverage SQL based transformation if you are more inclined with sql

Refer - https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/process-events-using-s...

2.Ingest the data into Eventhouse and use notebook-based transformations directly within the Event Stream—there’s no need to create shortcuts or add separate logic.

Refer - https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/add-destination-spark-...

Regards,

Srisakthi

[suparnababu8 · Accepted solution]

Hello @svk666 

If you're ingetsing the live data in to event house, you no need to create EH shortcut and no neeed to go for Lakehouse for tarsformation. 

In evenhous eit self we have KQL, by using this you can transform the data, Once the data is transfomed you can build the dashboard from eventhouse itself. Pls read this thread https://community.fabric.microsoft.com/t5/Real-Time-Intelligence-Community/Eventhouse-and-its-best-p... it might helps you

Thank you!

Did I answer your question? Mark my post as a solution!

Proud to be a Super User!