need help

12 Topics

Best practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.
Solved
ShivekMaharaj
8 days ago Place Real-Time Intelligence
89Views
0likes
2Comments
Best practice for deciding between Eventstream transformations and Eventhouse update policies
Hi Fabric Community, I am exploring a Real-Time Intelligence architecture and would appreciate some guidance on where transformation logic should ideally be placed. The proposed flow is: Azure Event Hubs → Fabric Eventstream → Eventhouse → Real-Time Dashboard / Power BI Fabric Eventstream supports filtering, field management, aggregation and other processing before events are written to the destination. An Eventhouse can also ingest the raw events first and transform them into curated tables through KQL update policies. I am trying to understand the recommended boundary between these two layers. For example, assume the incoming event contains: Device or customer identifier Event timestamp Event type Location Numeric readings Additional JSON properties The required processing includes: Removing events that fail basic validation Renaming and standardizing fields Converting timestamps and data types Flattening selected JSON properties Enriching the event with reference data Creating five-minute aggregates Preserving the original event for auditing and future reprocessing My current thinking is: Use Eventstream for lightweight filtering, routing and simple schema normalization. Land the original event in a Bronze table whenever replay or auditing is required. Use Eventhouse update policies or KQL for enrichment, reusable business logic and curated Silver tables. Use materialized views for frequently queried aggregations rather than calculating them repeatedly in dashboards. However, I am unsure where Microsoft recommends drawing the line. A few questions: Are there transformation types that should generally remain in Eventstream rather than Eventhouse? Is it considered good practice to send both the raw stream and a transformed derived stream into separate Eventhouse tables? When using Eventstream’s Event processing before ingestion mode, what are the trade-offs compared with direct ingestion followed by an Eventhouse update policy? How do teams handle changes to transformation logic when historical events need to be reprocessed? For reference-data enrichment, would you normally perform the lookup in Eventstream or after ingestion with KQL? Are five-minute or hourly aggregations better implemented in Eventstream, through an update policy, or with an Eventhouse materialized view? Microsoft’s Eventstream destination guidance documents both direct ingestion and event processing before ingestion, while the KQL update-policy documentation provides another way to transform ingested data. I would be interested to hear how others divide responsibility between Eventstream and Eventhouse in production, particularly where auditability, reprocessing and maintainability are important. Thanks in advance!
Solved
ShivekMaharaj
15 days ago Place Real-Time Intelligence
95Views
0likes
2Comments
New to microsoft fabric- time intelligence.
Hello everyone! I'm new to microsoft fabric and Real -Time Intellligence. I am excited to learn how the real time data processing works and look forward to learning from this community. I am beginner to microsoft fabric . So as a beginner with basic programming knowledge, what are the best beginner resources to start learning Real-Time Intelligence? I hope you will favour me....
Anj_il_a-
18 days ago Place Real-Time Intelligence
251Views
2likes
5Comments
Real time dashboard
Since the latest Microsoft Fabric update, I've noticed a change in how Real-Time Dashboard refresh behaves. Previously, I could configure visuals with Continuous refresh, allowing them to update almost immediately whenever new data arrived (typically within a few seconds, with a maximum delay of around 10 seconds). After the recent update, it appears that continuous refresh is enabled by default, but visuals now refresh at a fixed minimum interval of 10 seconds. In addition, visuals that don't support continuous refresh—or when no new data is received—seem to refresh every 30 seconds. I'm trying to understand whether this is expected behavior or if there's a way to restore the previous experience. Specifically: Is it still possible to configure Real-Time Dashboard visuals to refresh more frequently than every 10 seconds? Has the previous event-driven continuous refresh option been removed, or is it available through another setting? Are there any recommended approaches to achieve near real-time updates when new data arrives instead of relying on fixed refresh intervals? Was this change introduced intentionally as part of the latest Microsoft Fabric release? Our dashboards are used to monitor live operational processes, so even a 10–30 second refresh delay has a noticeable impact compared to the earlier behavior. Any clarification or guidance from the community or Microsoft team would be greatly appreciated.
Solved
powerbidev123
19 days ago Place Real-Time Intelligence
133Views
1like
1Comment
Working with Microsoft Fabric Support - A Collaborative Approach
At Microsoft Fabric Support, our goal is simple: help you resolve issues as quickly and smoothly as possible. Just like the recommendations shared in the Microsoft Fabric Community’s guidance on getting questions answered effectively, providing clear and complete information upfront helps everyone move faster and avoid unnecessary back-and-forth. Support is most effective when it’s a collaborative process. You bring the knowledge of your environment and workloads, and we bring deep platform expertise and diagnostics to help investigate the issue together. Help Us Help You When opening a support case, please include as much of the following information as possible: Issue Details Clear description of the problem Expected behavior vs. actual behavior Is the issue intermittent or consistently reproducible? Approximate timeframe of when the issue occurred (UTC preferred) Error Information Please include: Full error message text Screenshots (highly recommended) Activity IDs / Correlation IDs if available Even small details can significantly speed up investigation. Environment Information That Helps Investigation Depending on the Fabric workload being used, please provide the relevant item IDs and environment details. Workspace Information Workspace ID / Workspace URL To find this: Open the Fabric workspace Copy the URL from your browser Example: ".. https://app.fabric.microsoft.com/groups/<WorkspaceID>/… …" The value after /groups/ is the Workspace ID. Screenshot example above showcases the WorkspaceID and EventhouseID. Fabric Item IDs Please provide the item ID related to the affected workload, such as: EventhouseID ActivatorID LakehouseID WarehouseID SemanticModelID EventstreamID PipelineID These IDs help us locate telemetry and backend diagnostics more efficiently. You can typically find these: In the browser URL while inside the Fabric item Within item settings/details pages From the Fabric portal navigation pane Additional Environment Details Please also include: "… WorkspaceID: Capacity Name: Region …" Example: "… WorkspaceID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Capacity Name: Fabric-Prod-EastUS Region: East US …" This information helps us identify: Capacity-level issues Regional service impact Configuration-specific behavior Why This Matters Microsoft Fabric is a distributed cloud platform, and troubleshooting often depends on correlating logs, telemetry, timestamps, and resource identifiers together. Providing complete details upfront helps: Reduce delays Avoid repeated clarification requests Accelerate root cause analysis Create a smoother support experience for everyone involved We’re here to partner with you throughout the process, and we truly appreciate your collaboration in helping us investigate issues effectively. We look forward to working with you!
krlitt
1 month ago Place Real-Time Intelligence
1.1KViews
5likes
3Comments
Digital Twin Builder (preview): Bounding storage growth for an incremental time-series
Is there any supported way to apply a retention horizon to a Digital Twin Builder time-series mapping that uses incremental processing, so the underlying dtdm table doesn't grow without bound? My scenario: high-frequency live telemetry landing in an Eventhouse, surfaced to DTB through a lakehouse table (I've used both shortcuts and pipeline copy activities in my tests). The static layer and relationships are working well. The question is purely about the time-series layer. With incremental mapping enabled, the series appends continuously and the dtdm table grows linearly with no eviction. I've confirmed a couple of things that don't solve it, so I'll pre-empt them: Retention on the source table doesn't bound the twin copy. Incremental processing is forward-only, so source rows aging out are never removed from data the mapping already ingested. A mapping filter governs ingestion, not eviction, and only limits initial backfill (no relative/dynamic date filter). I assume full reload (incremental disabled) over a windowed source table does keep things bounded, since each run replaces the prior load, and that's my current fallback. But I'd like to confirm whether there's any retention/TTL control I've missed for the incremental path specifically, at the mapping, entity, or item level, or any sanctioned way to age out rows from the dtdm time-series table. If the answer is "not currently supported," that's useful to know too, and I'll file it as an Idea. Running on a Fabric capacity, DTB still in preview. Thanks in advance
Solved
smorenodev
1 month ago Place Real-Time Intelligence
1KViews
0likes
5Comments
Error in Microsoft Applied Skills: Implement a Real-Time Intelligence Assessment Lab : Need help
Hello All, since a week I was stuck on one error in Microsoft Applied Skills: Implement a Real-Time Intelligence solution with Microsoft Fabric Lab. Did anyone face the same issue if so, can anyone help me out how to resolve it. Attaching the error and related images below. Error: It says " unable to load container blobs. Verify that workspace identity is enabled and has storage blob data reader access to the storage account" . I am familiar with this permission and have enabled it in my own azure environment for my own blob that I have created but since these are pre created by Microsoft, I don't have access to these storage blobs and hence I am not sure how to proceed here. I have attempted this twice but I was stuck at the same point. I really appreciate your help and taking time in looking into my post. Thank you so much.
Solved
AshikaAlvakonda
3 months ago Place Real-Time Intelligence
1.8KViews
0likes
7Comments
Activator Not Showing in My Workspace
According to the Fabric Capacity Metrics, an Activator item is consuming a high amount of capacity units. However, when I check the workspace manually, no Activator item is visible. I wanted to delete or remove this item, as it appears to have been activated by mistake.
Solved
kasun_rumesh
4 months ago Place Real-Time Intelligence
1.6KViews
0likes
6Comments
Realtime - Business Events - cannot create business event shcema
Hi Has anyone succeeded in creating business event shcema? I keep getting the error "An unexpected error occourred". I have tried waiting a week but it seems we simply cannot make a "business event schema" (se screendump in appendix). Our tennant is in West Europe. Appendix - screendumps
Solved
KimTutein
4 months ago Place Real-Time Intelligence
1.1KViews
0likes
1Comment
How to reset Eventhouse Ingestion from Batching to Streaming ?
Hi everyone, I am experiencing a persistent latency issue in our Microsoft Fabric Eventhouse. A table that was previously performing perfectly has suddenly "downgraded" its ingestion path from Streaming to Batching and refuses to recover. The Situation: The Problem: In our PROD Eventhouse, ingestion latency is stuck at 12–15 seconds. The table is generating 45+ shards (extents) every 10 minutes, confirming it is in Batching Mode. The Discrepancy: Our DEV Eventhouse (identical schema and higher data volume) is still Streaming perfectly with ~5-second latency and 0 shards created per 10 minutes. The History: PROD was working fine (5s latency) and then switched to this slow Batching state on its own without any schema or policy changes. What we have verified: Streaming Policy: Both DEV and PROD have streamingingestion enabled. Batching Policy: We tested various ingestionbatching settings in a separate environment. We confirmed that the 12s latency in PROD is a result of the Batching Path overhead Unable to restore streaming: Running .alter table ... streamingingestion enable in PROD does not trigger a return to the Streaming. Our Questions: Why would an Eventhouse suddenly "blacklist" a table from the Streaming path and move it to permanent Batching if the Capacity (CU) is healthy (~50%)? How do we reverse this process? Once a table is stuck in this "Permanent Batching" state, what is the specific command or workflow to force the Eventhouse to re-evaluate it for the Streaming (Fast) Lane? We need to restore the 5-second latency to meet our real-time requirements. Any insights into the internal health-check logic of the Eventhouse would be incredibly helpful. Thank you !
Solved
lavginqo
4 months ago Place Real-Time Intelligence
32KViews
0likes
15Comments