kql database
69 TopicsChanging Ingestions from JSON
I have an EventStream receiving IoT Data out of JSONs. In my KQL table the columns timestamp, deviceId and for example x and y exist. The data from one sensor gets ingested in the columns timestamp, deviceId and y. No x since there is no data in the JSON for that. I dont work with a Mapping because it only makes it more complicated and the names in the JSON and in the KQL table are identical. No I changed the named for data from y to x when creating the JSON. Since there is no mapping and the column already exists in the KQL table the data should be saved now in column x. My problem is, it doesn't the data is not in x nor in y. It seems to be gone. Is that a bug in fabric? Can I solve this?128Views0likes11CommentsFabric Eventstream MySQL CDC scans all databases and does not emit row-level events
Hello Fabric Team, We are testing the Microsoft Fabric Eventstream MySQL CDC connector with an AWS RDS MySQL database. The MySQL instance is accessible through a private IP, and Fabric connects through a VNet data gateway/private network connection. Connectivity is successful, and the connector remains Active without reporting processing errors. Our MySQL environment contains the same table structure across multiple databases, following a pattern similar to: application_<tenant>.sample_table We configured the source to capture only one fully qualified table: application_tenant1.sample_table The MySQL configuration is: log_bin = ON binlog_format = ROW binlog_row_image = FULL The connection user has SELECT, SHOW DATABASES, REPLICATION CLIENT, REPLICATION SLAVE/REPLICA, and the required snapshot permissions. Each Fabric source uses a unique Server ID. CDC from the same RDS instance has previously worked successfully through another CDC platform. However, we are seeing the following behavior: Although only one fully qualified table is selected, Fabric/Debezium scans schema metadata for every accessible database and table on the MySQL server. More than 45,000 schema events were generated across the tenant and history databases. The schema events contain DDL such as DROP TABLE IF EXISTS and CREATE TABLE. Snapshot mode is configured as Initial. The selected table contains approximately 16,000 records, but Fabric emitted zero snapshot r events. We committed a new insert into the selected table after schema discovery finished, but no c event was emitted. Eventhouse receives the schema events successfully, confirming that the source-to-Eventhouse path is operational. The Eventhouse destination stores the Debezium payload in a dynamic column using a mapper from payload to RawEvent. Could you please confirm the following? Is it expected for the Fabric-managed MySQL CDC connector to scan and publish schema events for every database when only one fully qualified table is selected? Does Fabric expose or support settings equivalent to: database.include.list table.include.list schema.history.internal.store.only.captured.databases.ddl schema.history.internal.store.only.captured.tables.ddl Why would the connector finish schema discovery but not emit existing rows from the selected table when Snapshot mode is Initial? Why are committed inserts not emitted after the schema phase appears to finish? Is there a known limitation involving AWS RDS MySQL accessed through a private IP and VNet data gateway? What is the recommended supported destination pattern for this CDC stream? Our main objective is reliable ongoing CDC capture. We are flexible about the landing destination and can use: Eventhouse ADLS Gen2 in JSON or Parquet Fabric Lakehouse/OneLake Snowflake as the final destination Please let us know which logs, connector identifiers, timestamps, or additional configuration details would help investigate this behavior. Thank you.Solved62Views0likes4CommentsUnable to create a Monitoring Eventhouse in a Microsoft Fabric workspace.
I am trying to enable Workspace Monitoring for a Microsoft Fabric workspace. The user has Fabric Administrator access and is also a Workspace Administrator. However, under: Workspace Settings → Monitoring the + Eventhouse option is disabled/not available. We have verified the tenant-level settings and confirmed that “Workspace admins can turn on monitoring for their workspaces” is enabled.Solved131Views2likes6CommentsAnomaly Detector is disabled
Hello, I have a big Table in which we have our TimeSeries data from various devices. Since not every device sends the same data we have a big schema of numeric columns that are not filled for every deviceId. Is that the reason why there is no Anomaly Detector in my EventHouse? In the schema I can see that columns for the data are int or real. Some columns are string, but that shoudn't be the reason, right? Has anyone faced the same issue?93Views0likes5CommentsFabric Eventhouse: Capacity policy for .ingest inline is hard-capped at 1 concurrent operation
Problem When using the KQL Activity in Fabric Data Pipelines to execute .ingest inline commands against an Eventhouse KQL database, the effective ingestion capacity is always 1 concurrent operation — regardless of the Fabric capacity SKU (tested on F2 and F8) and regardless of the cluster capacity policy setting. Running .alter-merge cluster policy capacity with ClusterMaximumConcurrentOperations: 16 succeeds without error, but .show capacity still reports an effective ingestion capacity of 1. Any pipeline pattern that executes .ingest inline within a parallel ForEach immediately hits 429 TooManyRequests / ControlCommandThrottledException errors. Highlighting that the capacity is 1. This makes the KQL Activity (KustoQueryLanguage type) unusable for parallel ingestion scenarios — which is an important use case in data pipelines (logging, event emission, watermark updates during parallel table processing). Question / Request Do you recognize this issue? if so I have the below request: Increase the effective ingestion capacity for Fabric Eventhouse to match what the capacity policy allows (respect ClusterMaximumConcurrentOperations)Solved507Views0likes11CommentsHow to fix a Bad shortcut name
Hi, I have (had) a shortcut in may KQL Database called "terminals". The shortcut was pointing to the wrong lakehouse, so I had to drop it and create again. However, every time I create it again it adds a counter to the shortcut name: "_1", "_2" and so on. It doesn't matter if the previous shortcut was already dropped, it stills add a counter. How can I fix this problem? If I wait between dropping the original shortcut and creating the new one, woult it solve the problem ? I already tried waiting, but it didn't work. I imagine it may be a matter of how long I need to wait - how long? Is there any other way to solve this? Kind Regards, Dennes4KViews2likes6CommentsCopilot for EventHouse Queryset
Hi, RTI has introduced the preview feature of Copilot for the Eventhouse Queryset, but is there any way to automate the Copilot usage, via API endpoint or SDKs? The idea is to programmatically generate relevant and meaningful Kusto queries for business insights, from the tables and datasets present in the KQL DB in Eventhouse using Copilot. Thanks in advance!6.7KViews1like7CommentsDelay in Onelake Shortcut from Eventhouse to Lakehouse
I have a table in a KQL Database in an eventhouse that is hooked up to an Azure Eventhub. I have turned on One Lake storage for this table and added a shortcut to the lakehouse for this table. The table receives around 10-15k events per day and I have set the retention policy for the table to 30 days. The eventhouse table data was updating and getting reflected in the shortcut table within a few minutes when I first set it up. But now it takes almost an hour or more to see the latest event data in the shortcut table in my lakehouse. I turned the one lake availability off and then on again, and also tried updating the table policies but nothing seems to workSolved296Views1like4CommentsBCDR support for Fabric Eventhouse (KQL DB) – cross-region data replication and failover
Hi team, A few quick questions regarding BCDR options for Fabric Eventhouse (KQL): Does Fabric Eventhouse provide built-in cross-region redundancy/failover in case of regional outages? If yes, what is required to implement this, and what does the cost model look like? Does Fabric support native data replication/ copy capabilities? Specifically, is there support (or a roadmap) for: Azure Data Explorer (ADX/Kusto) → Fabric Eventhouse (KQL DB) Fabric Eventhouse (KQL DB) → Fabric Eventhouse (KQL DB) across regions (e.g., westcentralus ↔ westus) Could you please share any relevant documentation, known limitations, or guidance for these scenarios? Thanks in advance for your help!Solved1.4KViews2likes3CommentsDataFlow Gen 2 with KQL Engine Query URI as the destination
I'm trying to implement the Content Delivery Lifecycle using workspace Git integration- see https://learn.microsoft.com/en-us/fabric/cicd/git-integration/intro-to-git-integration I need the destination of my Data Flow to automatically pick up the right KQL Engine instance + connection, here: I can make it pick up the cluster URI from a library variable as show above. However, the problem that I have is that the connection name does not populate (also shown above). When I run the Data Flow (in this state) I get the error below:Solved1.4KViews0likes7Comments