kql queryset
34 TopicsFabric Eventstream MySQL CDC scans all databases and does not emit row-level events
Hello Fabric Team, We are testing the Microsoft Fabric Eventstream MySQL CDC connector with an AWS RDS MySQL database. The MySQL instance is accessible through a private IP, and Fabric connects through a VNet data gateway/private network connection. Connectivity is successful, and the connector remains Active without reporting processing errors. Our MySQL environment contains the same table structure across multiple databases, following a pattern similar to: application_<tenant>.sample_table We configured the source to capture only one fully qualified table: application_tenant1.sample_table The MySQL configuration is: log_bin = ON binlog_format = ROW binlog_row_image = FULL The connection user has SELECT, SHOW DATABASES, REPLICATION CLIENT, REPLICATION SLAVE/REPLICA, and the required snapshot permissions. Each Fabric source uses a unique Server ID. CDC from the same RDS instance has previously worked successfully through another CDC platform. However, we are seeing the following behavior: Although only one fully qualified table is selected, Fabric/Debezium scans schema metadata for every accessible database and table on the MySQL server. More than 45,000 schema events were generated across the tenant and history databases. The schema events contain DDL such as DROP TABLE IF EXISTS and CREATE TABLE. Snapshot mode is configured as Initial. The selected table contains approximately 16,000 records, but Fabric emitted zero snapshot r events. We committed a new insert into the selected table after schema discovery finished, but no c event was emitted. Eventhouse receives the schema events successfully, confirming that the source-to-Eventhouse path is operational. The Eventhouse destination stores the Debezium payload in a dynamic column using a mapper from payload to RawEvent. Could you please confirm the following? Is it expected for the Fabric-managed MySQL CDC connector to scan and publish schema events for every database when only one fully qualified table is selected? Does Fabric expose or support settings equivalent to: database.include.list table.include.list schema.history.internal.store.only.captured.databases.ddl schema.history.internal.store.only.captured.tables.ddl Why would the connector finish schema discovery but not emit existing rows from the selected table when Snapshot mode is Initial? Why are committed inserts not emitted after the schema phase appears to finish? Is there a known limitation involving AWS RDS MySQL accessed through a private IP and VNet data gateway? What is the recommended supported destination pattern for this CDC stream? Our main objective is reliable ongoing CDC capture. We are flexible about the landing destination and can use: Eventhouse ADLS Gen2 in JSON or Parquet Fabric Lakehouse/OneLake Snowflake as the final destination Please let us know which logs, connector identifiers, timestamps, or additional configuration details would help investigate this behavior. Thank you.Solved76Views0likes4CommentsUnable to create a Monitoring Eventhouse in a Microsoft Fabric workspace.
I am trying to enable Workspace Monitoring for a Microsoft Fabric workspace. The user has Fabric Administrator access and is also a Workspace Administrator. However, under: Workspace Settings → Monitoring the + Eventhouse option is disabled/not available. We have verified the tenant-level settings and confirmed that “Workspace admins can turn on monitoring for their workspaces” is enabled.Solved163Views2likes6CommentsCopilot for EventHouse Queryset
Hi, RTI has introduced the preview feature of Copilot for the Eventhouse Queryset, but is there any way to automate the Copilot usage, via API endpoint or SDKs? The idea is to programmatically generate relevant and meaningful Kusto queries for business insights, from the tables and datasets present in the KQL DB in Eventhouse using Copilot. Thanks in advance!6.7KViews1like7CommentsIs there a way to deactivate an Eventhouse/KQL DB?
Greetings, all. I have eventhouses and KQL DBs in Dev and Test workspaces that are part of a deployment pipeline. They generate capacity usage even though no data is flowing in and no one uses them. I want to deactivate or turn off the Dev/Test objects until we want to use them for code changes for deployments so Prod isn't broken. Is there a way to do this?Solved9.8KViews0likes4CommentsIssue creating a KQL materialized view
Hi, I'm testing an eventstream that reads source events from an Azure Event Hubs and write them into a KQL table. I've created successfully a KQL materialized view (using summarize and arg_max clauses) with the backfill=true option, without specifying the async clause, while the eventstream was active. After two days, for debug purposes, I've tried to create a parallel materialized view with a different name using the same KQL code and I've obtained this error: But firts it isn't required the async keyword. So, I've tried to create an async materialized view using .create async materialized-view with (backfill=true) mymaterview on table mytable an OperationId is shown but it doesn't appear any materialized views. Now, I don't understand because first it isn't necessary to specify the async keyword and then I need to write it. Then, I don't understand because an async materialized view doesn't appear. I don't understand because I cannot create a parallel clone materialized view with a different name. I hope that a materialized view, created with or without the back fill option, keep updating with new source event data without executing any alter action on it. Any helps to me, please? Many thanksSolved10KViews0likes2CommentsT-SQL executing directly in Eventhouse KQL Queryset without comment (--) prefix
Hey everyone, I was just exploring Eventhouse and noticed something weird. The docs say you need to add "--" on its own line before T-SQL queries in KQL querysets, but I'm running T-SQL commands just fine without it. I never added the "--" prefix and all T-SQL queries executed normally. Is this a recent change? The documentation still mentions needing the comment prefix to switch to T-SQL mode, but it seems like the query editor is just auto-detecting it now? Reference: https://learn.microsoft.com/en-us/kusto/query/t-sql?view=microsoft-fabricSolved3.9KViews0likes7Commentskusto error- partial query failure
hi all, error - Details: "Query execution has exceeded the allowed limits (80DA0001): Partial query failure: Runaway query (E_RUNAWAY_QUERY). (message: The Join output block has exceeded the memory budget during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits).: ). [0]Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException: Query execution has exceeded the allowed limits (80DA0001): Partial query failure: Runaway I am trying to join 5 tables in kusto to fetch all required columns , 3 tables from 1 cluster ans 2 from another cluster. But query is failing with memory error Dummy query : let onboarded = Pln | project STId; let regAss = external_table('PR') | where Region == 'France' | where Status !in ("Error", "Rejected") | project STId; let exeAppr = external_table('PR') | where Region == "France" and Status in ("Ready", "Approved") | project STId, Region, Status; let QReferenced= external_table('PQR') | where Region == 'France' and Status == "Completed" | project STId, Region, Status,RequestId; let Regionpsllls = cluster("cp.ws.kusto.windows.net").database("pslll").pslll | where State == "Active" and pslllType == "Region" | project pslllId, pslllName = Name, pslll_Reg = RegionName; let pslllServices = cluster("cp.ws.kusto.windows.net").database("pslll").PFL | where ProductType == "Service" | project pslllId, SOId; let pslll_Reg_Serv = Regionpsllls | join kind=inner pslllServices on pslllId | project pslll_Reg, pslllName, SOId; // Join onboarded with pslll let Onboardedpslll = onboarded | join kind=inner pslll_Reg_Serv on $left.STId == $right.SOId; // Join region assigned with pslll let regAsspslll = regAss | join kind=inner pslll_Reg_Serv on $left.STId == $right.SOId; // Join execution approved with pslll let exeApprpslll = exeAppr | join kind=inner pslll_Reg_Serv on ($left.STId == $right.SOId and $left.Region == $right.pslll_Reg); //join quota refernced with pslll let QReferencedpslll = QReferenced | join kind=inner pslll_Reg_Serv on ($left.STId == $right.SOId and $left.Region == $right.pslll_Reg); // Combine using inner joins (only psllls present in all three sets) Onboardedpslll | join kind=inner regAsspslll on pslll_Reg, pslllName | join kind=inner exeApprpslll on pslll_Reg, pslllName | join kind=inner QReferencedpslll on pslll_Reg, pslllName | project pslll_Reg, pslllName ( i want many more cols to project)Solved7.8KViews0likes4CommentsPartitioning policy is not copied to DatabaseSchema.kql
I have added a custom partitioning policy to some materialized views and tables like this: .alter materialized-view DimResponse policy partitioning "{\"PartitionKeys\":[{\"ColumnName\":\"OrgId\",\"Kind\":\"Hash\",\"Properties\":{\"Function\":\"XxHash64\",\"MaxPartitionCount\":128,\"Seed\":1,\"PartitionAssignmentMode\":\"Uniform\"}}],\"EffectiveDateTime\":\"2023-01-01T00:00:00.000000Z\",\"MinRowCountPerOperation\":0,\"MaxRowCountPerOperation\":0,\"MaxOriginalSizePerOperation\":0}" I have connected to source control, but in DatabaseSchema.kql, the policy is not in the schema, is this a limitation or a bug? My update policies are applied in the DatabaseSchema.kql, so something is working when altering materialized views and tables.Solved3.2KViews0likes1CommentKQL json mapping from Date Pipeline
Hi everyone. I’m running into an issue when ingesting data from MongoDB into my Kusto database using Azure Data Factory’s Copy Activity. Two of the fields in my source collection are Unix timestamps, but when they arrive in Kusto they end up stored as long instead of datetime. To fix this, I created a JSON ingestion mapping on my Kusto table that should convert those two timestamp columns into proper datetimes during ingestion. However, when I specify that mapping in the Copy Activity’s Sink settings, I get the following error: ErrorCode=KustoMappingReferenceHasWrongKind Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException Message=Mapping reference should be of kind 'Csv'. Mapping reference: 'UnixToDateTimeMapping'. Kind 'Json'. Source=Microsoft.DataTransfer.Runtime.KustoConnector I’ve tried passing the mapping name and adding various sink properties, but I can’t get ADF to apply my JSON mapping. Has anyone successfully configured a Copy Activity to use a Kusto JSON ingestion mapping? The code of my mapping as well .create-or-alter table info_connections_teste ingestion json mapping "UnixToDateTimeMapping" '[' ' { "column":"_id", "datatype":"string", "Properties":{"Path":"$._id"} },' ' { "column":"latency", "datatype":"real", "Properties":{"Path":"$.latency"} },' ' { "column":"packetloss_percentage","datatype":"int", "Properties":{"Path":"$.packetloss_percentage"} },' ' { "column":"download_occupancy","datatype":"real", "Properties":{"Path":"$.download_occupancy"} },' ' { "column":"upload_occupancy", "datatype":"real", "Properties":{"Path":"$.upload_occupancy"} },' ' { "column":"connection_id", "datatype":"string", "Properties":{"Path":"$.connection_id"} },' ' { "column":"createdAt", "datatype":"long", "Properties":{"Path":"$.createdAt"} },' ' { "column":"createdAtDate", "datatype":"datetime", "Properties":{"Path":"$.createdAt","Transform":"DateTimeFromUnixSeconds"} },' ' { "column":"updatedAt", "datatype":"long", "Properties":{"Path":"$.updatedAt"} },' ' { "column":"updatedAtDate", "datatype":"datetime", "Properties":{"Path":"$.updatedAt","Transform":"DateTimeFromUnixSeconds"} },' ' { "column":"__v", "datatype":"int", "Properties":{"Path":"$.__v"} }' ']'Solved9.5KViews0likes4CommentsHow can I split a variable-depth, slash-delimited path_column field into separate col in KQL?
Q : How can I split a variable-depth, slash-delimited path_column field into separate path_lvl_1, path_lvl_2, … columns in Kusto (Fabric) so that Power BI (in DirectQuery mode) can consume a fixed schema without requiring manual updates when the number of segments grows? Details: My raw table is named data_table and has this schema: id_column : string device_column : string event_name : string data_type : string unit_column : string path_column : string // e.g. "A/B/C/D" or "X/Y" or "Region/Zone/Section/Shelf/…" value_column : string timestamp_column : datetime static_flag : bool message_type : string path_column can have an arbitrary number of “/” segments—today might be 3 levels, tomorrow 10 or 50. We need Power BI (in DirectQuery) to see columns like: path_lvl_1, path_lvl_2, path_lvl_3, …, path_lvl_N where N grows automatically whenever any row’s path_column has more segments than before. Approaches tried and limitations: Split to a dynamic array (extend segments_array = split(path_column, "/")) then client-side “List → Record → Expand.” Works in Import mode, but breaks in DirectQuery because the M step (using List.Count, Record.FromList, Expand) cannot fold. Split into a fixed number of columns (e.g. assume max of 10 levels via path_lvl_1 = iff(array_length(parts) >= 1, parts[0], ""), …, path_lvl_10 = iff(array_length(parts) >= 10, parts[9], "")). Works in DirectQuery, but if a row ever has 11 segments, you lose data beyond level 10—and must manually alter the function/table to add path_lvl_11. mv-expand + evaluate pivot at query time for a truly dynamic set of columns. KQL function returns a result with as many columns as needed (0..max index), but the schema is not stable, so Power BI (DirectQuery) cannot bind to it. What I need: A way in Fabric/Kusto to produce a “wide” result that always has a fixed schema of path_lvl_1…path_lvl_N, automatically increasing N whenever deeper paths appear—without having to manually ALTER the table or function each time the maximum depth increases. Ideally, a pure KQL solution (since DirectQuery won’t let me do client-side M transformations), or some Fabric feature I’m missing (update policies, etc.) that can “auto-add” new columns. Why this matters: We want the Fabric → Power BI pipeline to be maintenance-free. If tomorrow a device reports path_column = "A/B/C/D/E/F/G/H" the next Power BI refresh should automatically see path_lvl_7 and path_lvl_8 without a manual “.alter table” or “re-publish function.” Actually I need these different columns to create hierarchy in one of power bi report. so that the dynamic schema can be handled properly. or if you can provide a solution to handle it at etl level, that will also work, like may be implementing any function or script so that if each time no of values getting inccreased in path_column, then automatically functions or script run and creates same table with new schema with updated policy. Has anyone in the Fabric community solved this? Is there a recommended pattern to handle truly variable-depth slash-delimited hierarchies in KQL so that downstream Power BI (DirectQuery) always sees a stable, “growing” set of path_lvl_X columns?Solved7.2KViews0likes9Comments