Forum Discussion
Best practice for deciding between Eventstream transformations and Eventhouse update policies
Hi Fabric Community,
I am exploring a Real-Time Intelligence architecture and would appreciate some guidance on where transformation logic should ideally be placed.
The proposed flow is:
Azure Event Hubs → Fabric Eventstream → Eventhouse → Real-Time Dashboard / Power BI
Fabric Eventstream supports filtering, field management, aggregation and other processing before events are written to the destination. An Eventhouse can also ingest the raw events first and transform them into curated tables through KQL update policies.
I am trying to understand the recommended boundary between these two layers.
For example, assume the incoming event contains:
- Device or customer identifier
- Event timestamp
- Event type
- Location
- Numeric readings
- Additional JSON properties
The required processing includes:
- Removing events that fail basic validation
- Renaming and standardizing fields
- Converting timestamps and data types
- Flattening selected JSON properties
- Enriching the event with reference data
- Creating five-minute aggregates
- Preserving the original event for auditing and future reprocessing
My current thinking is:
- Use Eventstream for lightweight filtering, routing and simple schema normalization.
- Land the original event in a Bronze table whenever replay or auditing is required.
- Use Eventhouse update policies or KQL for enrichment, reusable business logic and curated Silver tables.
- Use materialized views for frequently queried aggregations rather than calculating them repeatedly in dashboards.
However, I am unsure where Microsoft recommends drawing the line.
A few questions:
- Are there transformation types that should generally remain in Eventstream rather than Eventhouse?
- Is it considered good practice to send both the raw stream and a transformed derived stream into separate Eventhouse tables?
- When using Eventstream’s Event processing before ingestion mode, what are the trade-offs compared with direct ingestion followed by an Eventhouse update policy?
- How do teams handle changes to transformation logic when historical events need to be reprocessed?
- For reference-data enrichment, would you normally perform the lookup in Eventstream or after ingestion with KQL?
- Are five-minute or hourly aggregations better implemented in Eventstream, through an update policy, or with an Eventhouse materialized view?
Microsoft’s Eventstream destination guidance documents both direct ingestion and event processing before ingestion, while the KQL update-policy documentation provides another way to transform ingested data.
I would be interested to hear how others divide responsibility between Eventstream and Eventhouse in production, particularly where auditability, reprocessing and maintainability are important.
Thanks in advance!
Hello ShivekMaharaj
welcome to this Microsoft Fabric community portal.
In short, your current thinking is correct.
Use the Eventstream solely for ingesting the data and do the heavy lifting in Eventhouse.
You want to have access to the original message; then a proper medallion architecture is the way to go.
You ingest all data in bronze layer Eventhouse tables using Eventstream transformation only for technical purposes like rejecting corrupted messages or adding fields (like ingestion timestamp).
From there, use table update policies to transform the data to silver tables where needed. If deduplication is needed, add silver materialized views. Why the silver tables? Because some data-consuming Fabric Items work only with tables at the moment, not materialized views. This way, you can serve all items.
The update policies can even enrich the message (in a limited way) if needed.
Because you have the same data in both the bronze and silver tables, at some point in time you need to think about the retention time of the bronze data (you probably want to keep the silver quality data).
Because of the excellent compression of Eventhouse, you can postpone this decision so you have plenty of time to check and double-check if the policies do their job correctly.
I have a sweet spot for Direct Ingest because I have full control over the original table mapping towards the Bronze table. I love a combination of a message identifier of the sender (line sensorID) taken from the original message, the timestamp taken from the original message, and a dynamic with the full original message.
This way, if the message format changes, I'm still flexible because at least the message arrived in the Bronze table. From there, I can fix things.
This is because I'm probably not in control of the original message format. It's coming from the outside world so if the message is rejected due to some failing transformation, it's lost. I hope the identifier and timestamp do not move within the message...
Historical data can be moved around from table to table in Eventhouse. This takes time (ingestion is fast, but billions of rows can still take time). Do it in batches, and track any issues.
Aggregations- that looks like a job for materialized views as a gold layer. You want them for a certain purpose, so keep them separated. Materialized views can be backfilled if needed.
Keep in mind there is a learning curve for many of the concepts seen here.
Take some time to explore how the Eventhouse works and how to use the different 'tools' in there...
You will end up with a versatile, cost-effective, and blazingly fast timeseries database.
These are my 50 cents 😉
---
If this answer helps you, marking it as the accepted answer is appreciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.
2 Replies
Hi ShivekMaharaj,
Thank you for reaching out to Microsoft Fabric Community.
Here your proposed approach looks good and is aligned with common best practices.
- In general, use Eventstream for simple processing like filtering out invalid events, basic field mapping and schema cleanup before the data is ingested.
- Use Eventhouse update policies for more complex and reusable transformations such as enrichment, JSON parsing, business logic and creating curated tables.
- If auditability or future reprocessing is important, keep the raw events in a bronze table and build your curated tables from there.
- For frequently queried aggregations, materialized views are generally the preferred option, they improve query performance without recalculating the same results for every query.
Thanks and regards,
Anjan Kumar Chippa
Hello ShivekMaharaj
welcome to this Microsoft Fabric community portal.
In short, your current thinking is correct.
Use the Eventstream solely for ingesting the data and do the heavy lifting in Eventhouse.
You want to have access to the original message; then a proper medallion architecture is the way to go.
You ingest all data in bronze layer Eventhouse tables using Eventstream transformation only for technical purposes like rejecting corrupted messages or adding fields (like ingestion timestamp).
From there, use table update policies to transform the data to silver tables where needed. If deduplication is needed, add silver materialized views. Why the silver tables? Because some data-consuming Fabric Items work only with tables at the moment, not materialized views. This way, you can serve all items.
The update policies can even enrich the message (in a limited way) if needed.
Because you have the same data in both the bronze and silver tables, at some point in time you need to think about the retention time of the bronze data (you probably want to keep the silver quality data).
Because of the excellent compression of Eventhouse, you can postpone this decision so you have plenty of time to check and double-check if the policies do their job correctly.
I have a sweet spot for Direct Ingest because I have full control over the original table mapping towards the Bronze table. I love a combination of a message identifier of the sender (line sensorID) taken from the original message, the timestamp taken from the original message, and a dynamic with the full original message.
This way, if the message format changes, I'm still flexible because at least the message arrived in the Bronze table. From there, I can fix things.
This is because I'm probably not in control of the original message format. It's coming from the outside world so if the message is rejected due to some failing transformation, it's lost. I hope the identifier and timestamp do not move within the message...
Historical data can be moved around from table to table in Eventhouse. This takes time (ingestion is fast, but billions of rows can still take time). Do it in batches, and track any issues.
Aggregations- that looks like a job for materialized views as a gold layer. You want them for a certain purpose, so keep them separated. Materialized views can be backfilled if needed.
Keep in mind there is a learning curve for many of the concepts seen here.
Take some time to explore how the Eventhouse works and how to use the different 'tools' in there...
You will end up with a versatile, cost-effective, and blazingly fast timeseries database.
These are my 50 cents 😉
---
If this answer helps you, marking it as the accepted answer is appreciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.
