Changing Ingestions from JSON
I have an EventStream receiving IoT Data out of JSONs. In my KQL table the columns timestamp, deviceId and for example x and y exist. The data from one sensor gets ingested in the columns timestamp, deviceId and y. No x since there is no data in the JSON for that. I dont work with a Mapping because it only makes it more complicated and the names in the JSON and in the KQL table are identical. No I changed the named for data from y to x when creating the JSON. Since there is no mapping and the column already exists in the KQL table the data should be saved now in column x. My problem is, it doesn't the data is not in x nor in y. It seems to be gone. Is that a bug in fabric? Can I solve this?
Unable to create a Monitoring Eventhouse in a Microsoft Fabric workspace.
I am trying to enable Workspace Monitoring for a Microsoft Fabric workspace. The user has Fabric Administrator access and is also a Workspace Administrator. However, under: Workspace Settings → Monitoring the + Eventhouse option is disabled/not available. We have verified the tenant-level settings and confirmed that “Workspace admins can turn on monitoring for their workspaces” is enabled.
Fabric Eventstream MySQL CDC scans all databases and does not emit row-level events
Hello Fabric Team, We are testing the Microsoft Fabric Eventstream MySQL CDC connector with an AWS RDS MySQL database. The MySQL instance is accessible through a private IP, and Fabric connects through a VNet data gateway/private network connection. Connectivity is successful, and the connector remains Active without reporting processing errors. Our MySQL environment contains the same table structure across multiple databases, following a pattern similar to: application_<tenant>.sample_table We configured the source to capture only one fully qualified table: application_tenant1.sample_table The MySQL configuration is: log_bin = ON binlog_format = ROW binlog_row_image = FULL The connection user has SELECT, SHOW DATABASES, REPLICATION CLIENT, REPLICATION SLAVE/REPLICA, and the required snapshot permissions. Each Fabric source uses a unique Server ID. CDC from the same RDS instance has previously worked successfully through another CDC platform. However, we are seeing the following behavior: Although only one fully qualified table is selected, Fabric/Debezium scans schema metadata for every accessible database and table on the MySQL server. More than 45,000 schema events were generated across the tenant and history databases. The schema events contain DDL such as DROP TABLE IF EXISTS and CREATE TABLE. Snapshot mode is configured as Initial. The selected table contains approximately 16,000 records, but Fabric emitted zero snapshot r events. We committed a new insert into the selected table after schema discovery finished, but no c event was emitted. Eventhouse receives the schema events successfully, confirming that the source-to-Eventhouse path is operational. The Eventhouse destination stores the Debezium payload in a dynamic column using a mapper from payload to RawEvent. Could you please confirm the following? Is it expected for the Fabric-managed MySQL CDC connector to scan and publish schema events for every database when only one fully qualified table is selected? Does Fabric expose or support settings equivalent to: database.include.list table.include.list schema.history.internal.store.only.captured.databases.ddl schema.history.internal.store.only.captured.tables.ddl Why would the connector finish schema discovery but not emit existing rows from the selected table when Snapshot mode is Initial? Why are committed inserts not emitted after the schema phase appears to finish? Is there a known limitation involving AWS RDS MySQL accessed through a private IP and VNet data gateway? What is the recommended supported destination pattern for this CDC stream? Our main objective is reliable ongoing CDC capture. We are flexible about the landing destination and can use: Eventhouse ADLS Gen2 in JSON or Parquet Fabric Lakehouse/OneLake Snowflake as the final destination Please let us know which logs, connector identifiers, timestamps, or additional configuration details would help investigate this behavior. Thank you.Anomaly Detector is disabled
Hello, I have a big Table in which we have our TimeSeries data from various devices. Since not every device sends the same data we have a big schema of numeric columns that are not filled for every deviceId. Is that the reason why there is no Anomaly Detector in my EventHouse? In the schema I can see that columns for the data are int or real. Some columns are string, but that shoudn't be the reason, right? Has anyone faced the same issue?Fabric Business Events: what delivery guarantees and replay pattern should we design for?
Hi all, I am testing the newer Business Events capability in Fabric Real-Time Intelligence and trying to understand what reliability assumptions should be made for a production design. The pattern I am looking at is roughly: Eventstream → Business Event → Activator → downstream action / User Data Function with Eventhouse enabled so the published business events are also retained for historical analysis. The current documentation explains the publisher/consumer model and shows how Eventstream can publish a governed business event that Activator then consumes. What I have not been able to find clearly documented is the delivery contract between the published business event and its consumers. A few things I am trying to clarify: If an Activator consumer or downstream action is temporarily unavailable, does Fabric retry delivery of the business event? Should consumers assume at-least-once delivery and therefore be designed to handle duplicate events, or is a different delivery model used? Is event ordering guaranteed in any scope, for example for events from the same Eventstream publisher? Since published business events can also be retained automatically in Eventhouse, is that retained history intended to support replay/reprocessing after a consumer outage, or is it primarily an analytical record and replay would need to be implemented separately? Are there documented retry or delivery-retention windows that should be considered when designing an operational workflow? I am mainly trying to understand what a resilient production pattern should look like when the business event triggers something with side effects, where processing the same event twice or silently missing an event would matter. Would you generally make the downstream consumer idempotent and treat Eventhouse as an audit/recovery store, or is there a more Fabric-native pattern for this? Interested to hear how others are approaching this with Business Events and Activator.
Handling Multiple Schemas in Eventstream for Azure Event Hub and Azure IOT Hub data sources
One of the challenges with streaming data is that data comes in a variety of schemas, which can be dynamic and are not always predictable. As applications and data structures change, schema values can sometimes be wildly different across devices or event inputs. Recently a customer reached out with several questions on this common problem. With the capabilities in Fabric Real Time Intelligence we can flexibly ingest this data. Enough of the background let’s get to the problem!
25 November 2024 – Meetup Dutch Fabric User Group & Power BI Gebruikersgroep
We are pleased to announce details of our very special extended in-person user group meetup on November 25th. Which we are doing together with the Power BI Gebruikersgroep. It will be hosted by our friends KPMG in their office at Amstelveen. Together we have an incredible lineup in store, starting with Oskari Heikkinen will talk about "Best practices for Fabric Eventhouse". Afterwards, Marnix Jansen will talk about "The Science of Effective Business Reporting: Using IBCS Standards to Drive Action". To finish the evening, Benni De Jagere and Kasper de Jonge from Microsoft will do an Ignite 2024 recap. You can register to attend by following the link below. 25 November 2024 – Meetup Dutch Fabric User Group – Dutch Fabric User group Don't miss out on this opportunity to gain valuable insights in these amazing sessions.Zürich - 69th Fabric User Group [IN-PERSON]
Dear Data Wizards, We are looking forward to inviting all of you to our next meetup. This time in in-person mode. If you wish to participate, please reach out to me personally to register. Thanks! Topics What's New - Kristian E2E Scenario - from REST API to KQL Magic to Insights - Meinrad & Kristian The session will be recorded and made available on YouTube --> https://aka.ms/FabricUGYouTube E2E Scenario - from REST API to KQL Magic to Insights In this session, we’ll dive into the fascinating world of metadata-driven pipelines and KQL within Microsoft Fabric. Starting with REST APIs, we’ll explore how to extract stock data, including daily prices, and seamlessly store it in a Lakehouse. But that’s just the beginning! Real-time analytics will transform this data, making it readily available for Power BI reporting. Join us as we demystify the process, share best practices, and empower you to create robust end-to-end solutions. Good to know We want this group to be a safe environment that encourages open discussion, exchange of ideas and problems you may face. Therefore, we kindly ask that no members will leverage the information for unsolicited acquisitions of new customers or projects. This group builds on trust, and without it we cannot learn from each other and excel on this topic. Want to be a presenter? We are always looking for new speaker. If you are interested and would like to show something to the Power BI Meetup Group please feel free to contact us!
