Forum Discussion
Regarding Adding reference data to my stream in Microsoft fabric event stream.
Hi Everyone,
I have an event stream which is consuming the data from eventhub.. before reaching the data to destination. i want to enrich the stream with reference data which is present in my lakehouse..I cannot see an option to add lakehouse as source.
According to the below video at 22:00 Min they added fabricqldatabase as reference data..i cannot see eventhat..
Building real solutions with Real-Time Intelligence in Fabric | BRK209 - YouTube
Thank you,
Venu
Makes sense. In this case, the pattern is established. Eventstream loads the initial stream into Eventhouse, where the stream is contextualized with the reference data and written to a new table. Then create a new eventstream that uses an Azure Data Explorer source (with the Eventhouse uri information). Even though the source says ADX Eventhouse is the same engine.
Then add a custom endpoint destination where your customer can subscribe. THe other thing you could consider is rather streaming back into the eventstream simply make the data available in eventhouse via the rest api
11 Replies
Hi VENUMADDIPATI52 ,
Thanks for posting in Microsoft Fabric Community.
Microsoft Fabric does support Lakehouse as a destination for Eventstream, but not yet as a reference data source for enrichment operations. This is likely why you're not seeing the Lakehouse option in the reference data dropdown.
The video you referred to may be showing functionality that is part of an internal preview and not generally available yet.
This setup is not yet generally available as a reference data source in the Eventstream UI. The currently supported sources are listed here: Microsoft Fabric event streams overview - Microsoft Fabric | Microsoft Learn.
These cover Event Hubs, IoT Hub, Service Bus, SQL Database CDC, PostgreSQL CDC, MySQL CDC, Cosmos DB CDC, Kafka, Kinesis, Pub/Sub, and other streaming services, but not Lakehouse or Fabric SQL Database as reference data.
When I try to add a source in Eventstream, the options available are shown in the screenshot below:
Hope this helps. Please reach out for further assistance.
Thank you.
Hi Venu,
You’re right — as of now, Fabric Event Streams doesn’t directly allow adding a Lakehouse table as reference data in the same way you can with a Fabric SQL database. The demo you mentioned uses a Fabric SQL DB because reference data sources currently need to be queryable in a structured format.
A couple of workarounds you can try:
Materialize your Lakehouse table into a Fabric SQL Database (via shortcut or pipeline), then reference that SQL table in your Event Stream.
If the reference data is relatively static, you could also pre-load it into a small external store (like Azure SQL DB) and link it in as reference data.
Keep an eye on updates—support for direct Lakehouse integration as reference data has been a frequent ask and may get added in future releases.
For now, the most straightforward route is to use a Fabric SQL DB as the bridge.
Hi Venu,
You’re right — as of now, Fabric Event Streams doesn’t directly allow adding a Lakehouse table as reference data in the same way you can with a Fabric SQL database. The demo you mentioned uses a Fabric SQL DB because reference data sources currently need to be queryable in a structured format.
A couple of workarounds you can try:
Materialize your Lakehouse table into a Fabric SQL Database (via shortcut or pipeline), then reference that SQL table in your Event Stream.
If the reference data is relatively static, you could also pre-load it into a small external store (like Azure SQL DB) and link it in as reference data.
Keep an eye on updates—support for direct Lakehouse integration as reference data has been a frequent ask and may get added in future releases.
For now, the most straightforward route is to use a Fabric SQL DB as the bridge.
Hi PeterP1 , I even tried this approach of loading my static data into fabric sql database and adding as refernce data. but i was not able to see the option for connecting to my fabric sql database..i can see only CDC type connections.
Thank you,
Venu MHi VENUMADDIPATI52 ,
Thanks for the update.At present, Eventstream doesn’t yet provide a way to directly add a Fabric SQL Database as data source/ reference data, which is why you don’t see it listed in the dropdown.
The only options currently available are the ones visible in the data source list: Microsoft Fabric event streams overview - Microsoft Fabric | Microsoft Learn
Similar discussion: Solved: Re: How to Use a Fabric SQL Database as a Source i... - Microsoft Fabric Community
Hope this helps. Please reach out for further assistance.
Thank you.
There is no need to add lakehouse as a source to add it as a reference stream. Use the eventstream to load your data to an eventhoues, and shortcut the data from lakehouse. Then join the data direclty in KQL to reference the data together.
Hi kustortininja , Thank for the reply. but here in my case i want to establish a fastpath to my customer by creating a custom end point...with as much as possible minimum latency i want to enrich my data and push into an customendpoint destination..then my customer will subscribe to that end point.
Makes sense. In this case, the pattern is established. Eventstream loads the initial stream into Eventhouse, where the stream is contextualized with the reference data and written to a new table. Then create a new eventstream that uses an Azure Data Explorer source (with the Eventhouse uri information). Even though the source says ADX Eventhouse is the same engine.
Then add a custom endpoint destination where your customer can subscribe. THe other thing you could consider is rather streaming back into the eventstream simply make the data available in eventhouse via the rest api
Hi VENUMADDIPATI52 ,
Just wanted to check if the responses provided were helpful. If further assistance is needed, please reach out.
Thanks kustortininja for sharing your inputs.Thank you.
Hi VENUMADDIPATI52 ,
We wanted to kindly follow up regarding your query. Have you had a chance to review the alternate method suggested by kustortininja . If you need any further assistance, please reach out.
Thank you.
