general question

427 Topics

Best practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.
Solved
ShivekMaharaj
3 days ago Place Real-Time Intelligence
80Views
0likes
2Comments
Infrastructure and licensing costs for adopting Fabric Planning in production (business case)
Hi everyone, We're evaluating Fabric Planning for a client who wants to adopt it in production, and I need to put together a business case with the associated costs. I've already reviewed the role-based session model (Viewer/Stakeholder/Planner, ~37/168/847 CU-hours per 30-day session) and the F64+ capacity requirement for the XMLA endpoint, but I have a few specific questions on infrastructure and licensing that the documentation didn't fully answer: Per-user licensing: besides the capacity (F64+), is any individual license required (Power BI Pro or Premium Per User) for each Viewer/Stakeholder/Planner user, or does the session's CU consumption already cover full access? Production sizing: beyond the technical F64 minimum, what capacity size would you recommend as a practical floor for a real production scenario with several concurrent users, factoring in the ~30% additional infrastructure buffer (OneLake, XMLA API, etc.)? Billing status: is session/role-based billing already in effect (general availability), or is preview usage still free right now? Regional pricing differences: is there any regional price variation we should account for with a client outside the US (Latin America)? Any real-world sizing experience or business-case examples you can share would be much appreciated. Thanks in advance!
OscarNG
7 days ago Place IQ
39Views
0likes
1Comment
Best practice for deciding between Eventstream transformations and Eventhouse update policies
Hi Fabric Community, I am exploring a Real-Time Intelligence architecture and would appreciate some guidance on where transformation logic should ideally be placed. The proposed flow is: Azure Event Hubs → Fabric Eventstream → Eventhouse → Real-Time Dashboard / Power BI Fabric Eventstream supports filtering, field management, aggregation and other processing before events are written to the destination. An Eventhouse can also ingest the raw events first and transform them into curated tables through KQL update policies. I am trying to understand the recommended boundary between these two layers. For example, assume the incoming event contains: Device or customer identifier Event timestamp Event type Location Numeric readings Additional JSON properties The required processing includes: Removing events that fail basic validation Renaming and standardizing fields Converting timestamps and data types Flattening selected JSON properties Enriching the event with reference data Creating five-minute aggregates Preserving the original event for auditing and future reprocessing My current thinking is: Use Eventstream for lightweight filtering, routing and simple schema normalization. Land the original event in a Bronze table whenever replay or auditing is required. Use Eventhouse update policies or KQL for enrichment, reusable business logic and curated Silver tables. Use materialized views for frequently queried aggregations rather than calculating them repeatedly in dashboards. However, I am unsure where Microsoft recommends drawing the line. A few questions: Are there transformation types that should generally remain in Eventstream rather than Eventhouse? Is it considered good practice to send both the raw stream and a transformed derived stream into separate Eventhouse tables? When using Eventstream’s Event processing before ingestion mode, what are the trade-offs compared with direct ingestion followed by an Eventhouse update policy? How do teams handle changes to transformation logic when historical events need to be reprocessed? For reference-data enrichment, would you normally perform the lookup in Eventstream or after ingestion with KQL? Are five-minute or hourly aggregations better implemented in Eventstream, through an update policy, or with an Eventhouse materialized view? Microsoft’s Eventstream destination guidance documents both direct ingestion and event processing before ingestion, while the KQL update-policy documentation provides another way to transform ingested data. I would be interested to hear how others divide responsibility between Eventstream and Eventhouse in production, particularly where auditability, reprocessing and maintainability are important. Thanks in advance!
Solved
ShivekMaharaj
10 days ago Place Real-Time Intelligence
90Views
0likes
2Comments
Best patterns for ingesting dynamic tax & compliance API data into Fabric Lakehouse?
Hi everyone, We are currently migrating our firm's analytical workloads to Microsoft Fabric. A huge part of our business involves tracking dynamic regional tax rates, company registration statuses, and financial compliance data for startups. Currently, our analysts manually pull this data from external regulatory sources and calculators, like this FBR Sales Tax Calculator, and we import it via Excel/CSV into Power BI. We want to fully automate this in Fabric. What is the recommended pattern for hitting external REST APIs on a scheduled basis (daily) to pull dynamic financial data? Should we use Data Factory pipelines with a Web activity to land the JSON directly into a Lakehouse, or is it better to use a Notebook (PySpark) to handle the API pagination and JSON flattening before writing to delta tables? Would appreciate any architecture advice for handling external financial APIs in Fabric! Thanks.
secpporta
21 days ago Place IQ
108Views
2likes
1Comment
High volume Dataverse dataflow times out avoid NESTED FROMs and/or JOINs best approach to resolve?
Background We have a Gen1 Dataverse dataflow that calculates aggregate totals per record (yearly, prior-year, lifetime, and household rollups) and writes them to two output entities, Merge Account and Merge Contact. Each output entity is built from a chain of ~11 Table.NestedJoin (JoinKind.FullOuter) joins over ~12 upstream aggregate queries. The problem The dataflow refreshes without issue on smaller environments but times out on high-volume ones with: Microsoft SQL: "The updated two minutes (2) timeout period elapsed prior to completion of the operation. Please avoid using SELECT *, or NESTED FROMs and/or JOINs to execute the query with default five minutes (5) timeout." ErrorCode = 10478; Number = 40000 (Some environments show error 40197 instead.) It stops on the Merge step. What we've implemented so far We've built a monitoring solution: a cloud flow that detects dataflow refresh failures and logs the details (dataflow name, ID, and error message) into a custom Refresh Audit table, surfaced in a model-driven app so the team has visibility into failures. We've analyzed the failure across multiple environments and confirmed it correlates with data volume — the same dataflow completes on smaller datasets and times out on larger ones, with the filtered row volume and per-record density appearing to be the main drivers. We've identified that the full-outer joins don't appear to fold, so a large nested query is sent to the Dataverse TDS endpoint and hits the query timeout — i.e. it's a timeout, not a data/row error. Approaches we're evaluating — would these be effective? Increasing the query/command timeout for a Gen1 Dataverse dataflow — is that configurable, and where? Using Table.Buffer on the join inputs to force local evaluation and avoid pushing the nested query to the TDS endpoint — would this reliably help, or just shift the bottleneck? Staging the upstream queries into computed/staged entities (or a separate staging dataflow) — is this the recommended pattern, and does it need enhanced compute / premium? For very large datasets (1M+ rows), pushing the aggregation source-side (FetchXML aggregate / SQL view) — is this the more reliable long-term approach? Is there a supported way to get per-step timing / query diagnostics for a dataflow refresh, to pinpoint the heaviest step and enrich our monitoring? Any guidance on which of these works best for high-volume Dataverse dataflows with heavy full-outer join chains would be much appreciated. Thanks!
Solved
Tejakm123
25 days ago Place Power Query
328Views
0likes
5Comments
Best choice for writing data from Fabric to Dynamics 365 Business Central
Hi, in order to write data from Fabric to Dynamics 365 Business Central, which could be the best choice to follow, considering your real experiences? Many thanks
Solved
pmscorca
28 days ago Place Pipelines
245Views
0likes
5Comments
Still haven't received Exam Voucher
I guess it's been 10 days since i completed all Data Fabrics learning module and applied for the exam voucher and since then i haven't heard anything. Pls does anyone know why this is so?
Benzema_
1 month ago Place Apache Airflow Job
2.1KViews
1like
2Comments
ICS (Industial Controls Systems)/PLC (Programmable Logic Controller) Integration
Can Microsoft Fabric be integrated with ICS (Industrial Controls Systems) including PLC (Programmable Logic Controller) and HMI (Human-Machine Interface)? We're interested in exploring Data Activator with our ICS information. Thank you for your time!
Jeff_Baron
1 month ago Place Activator
2.8KViews
1like
3Comments
News about Refresh Materialized Lake View activity in Fabric Pipelines
Hi, this month the Refresh MLV activity in Fabric pipeline is available in Public Preview, as reported in this article Refresh Materialized Lake View activity in Fabric Pipelines (Preview), but when will this feature be in GA? It's a feature very useful and important! Many thanks
pmscorca
1 month ago Place Pipelines
618Views
1like
3Comments
Error - Using the MongoDB Om-Premise Connector
Hello everyone, I'm trying to establish a connection between Fabric and an on-premises MongoDB server. https://learn.microsoft.com/en-us/fabric/data-factory/connector-mongodb-overview In fact, the scenario involves writing data from Fabric to an on-premises MongoDB NoSQL database. The connection has been successfully established on the gateway side (with all necessary firewall rules configured). Then I see the connection in the “Copy Data” section of the “Data Pipeline.” So that means the mapping is working properly. However, the connection test isn't working... Of course, the error message doesn't help at all. For your information, here is the server pattern specified in the gateway connection mongodb01.example.com:27017,mongodb22.example.com:27017,mongodb03.example.com:27017/?authMechanism=DEFAULT&authSource=xxxxxxxxx Thank in advance for your help Have a nice day, Vivien
vivien57
1 month ago Place Pipelines
1.7KViews
0likes
11Comments