kql queryset
36 TopicsBest practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.Solved97Views0likes2CommentsCopilot for EventHouse Queryset
Hi, RTI has introduced the preview feature of Copilot for the Eventhouse Queryset, but is there any way to automate the Copilot usage, via API endpoint or SDKs? The idea is to programmatically generate relevant and meaningful Kusto queries for business insights, from the tables and datasets present in the KQL DB in Eventhouse using Copilot. Thanks in advance!6.7KViews1like7CommentsError in Microsoft Applied Skills: Implement a Real-Time Intelligence Assessment Lab : Need help
Hello All, since a week I was stuck on one error in Microsoft Applied Skills: Implement a Real-Time Intelligence solution with Microsoft Fabric Lab. Did anyone face the same issue if so, can anyone help me out how to resolve it. Attaching the error and related images below. Error: It says " unable to load container blobs. Verify that workspace identity is enabled and has storage blob data reader access to the storage account" . I am familiar with this permission and have enabled it in my own azure environment for my own blob that I have created but since these are pre created by Microsoft, I don't have access to these storage blobs and hence I am not sure how to proceed here. I have attempted this twice but I was stuck at the same point. I really appreciate your help and taking time in looking into my post. Thank you so much.Solved1.8KViews0likes7CommentsIssue Writing to OneLake-Enabled KQL Database Delta Tables
I enabled OneLake availability for my KQL database so the tables are exposed as Delta or Parquet files. I’m able to read the table using Parquet without any issues: df = spark.read.format("parquet").load(abfsspath) However, when I try to overwrite or append data to the table, it fails: append_df.write.format("delta").mode("append").save(abfsspath) I receive the following error: Operation failed: "Forbidden", 403, AuthorizationPermissionMismatch "This request is not authorized to perform this operation using this permission." I am the workspace admin, and I don’t want to ingest data using the conventional KQL ingestion methods because they are too slow for my use case. I specifically need to write records directly using Delta or Parquet. Could you help me understand how to enable this write operation for Delta tables?Solved4.3KViews0likes8CommentsArchitecture & GPS hardware
Hello Fabric community, I'm building a fleet monitoring platform for trucks/vehicles and need architectural guidance for using Microsoft Fabric Real-Time Intelligence, as well as GPS hardware recommendations. MY SCENARIO: ------------ - GPS trackers sending location data every 10-60 seconds via TCP - Data includes: GPS coordinates, speed, fuel level, engine RPM, temperature sensors, driver behavior (harsh braking, acceleration) - Need to support multiple tenants (each customer sees only their own fleet) - Planning to scale from 50 vehicles to 1000+ vehicles - Want real-time dashboards + historical analytics + automated alerts - Future: add video/dashcam capabilities MY PLANNED ARCHITECTURE: ------------------------ GPS Devices → TCP Server (Traccar/custom) → Azure Event Hub → Fabric Eventstream → Eventhouse → Real-Time Dashboard QUESTIONS: GPS Hardware: ------------- 1. What GPS tracker brands/models do you recommend that integrate well with custom backends and Microsoft Fabric? I'm considering Teltonika FMC920, Queclink, or Concox - any experience with these? 2. Which GPS devices have the best open protocols for parsing data and sending to Azure/Fabric? 3. Any recommendations for GPS trackers with OBD-II support to read vehicle diagnostics (fuel, RPM, error codes)? Microsoft Fabric Architecture: ------------------------------ 4. Is Eventstream + Eventhouse the right approach for high-frequency GPS telemetry data? 5. What's the recommended way to implement multi-tenancy (row-level security) for different fleet customers? 6. Should I use KQL Database or Lakehouse for storing historical GPS data (months/years)? 7. Any recommendations for handling geofencing alerts in real-time using Activator? 8. What Fabric SKU (F4, F8, F16?) would you recommend starting with for ~100 vehicles sending data every 30 seconds? I'm based in Mexico and planning to deploy this commercially for fleet owners. Any guidance from people who have built similar IoT/telematics solutions with Fabric would be greatly appreciated! Thanks in advance!Solved6.7KViews0likes4CommentsIs there a way to deactivate an Eventhouse/KQL DB?
Greetings, all. I have eventhouses and KQL DBs in Dev and Test workspaces that are part of a deployment pipeline. They generate capacity usage even though no data is flowing in and no one uses them. I want to deactivate or turn off the Dev/Test objects until we want to use them for code changes for deployments so Prod isn't broken. Is there a way to do this?Solved9.8KViews0likes4CommentsIssue creating a KQL materialized view
Hi, I'm testing an eventstream that reads source events from an Azure Event Hubs and write them into a KQL table. I've created successfully a KQL materialized view (using summarize and arg_max clauses) with the backfill=true option, without specifying the async clause, while the eventstream was active. After two days, for debug purposes, I've tried to create a parallel materialized view with a different name using the same KQL code and I've obtained this error: But firts it isn't required the async keyword. So, I've tried to create an async materialized view using .create async materialized-view with (backfill=true) mymaterview on table mytable an OperationId is shown but it doesn't appear any materialized views. Now, I don't understand because first it isn't necessary to specify the async keyword and then I need to write it. Then, I don't understand because an async materialized view doesn't appear. I don't understand because I cannot create a parallel clone materialized view with a different name. I hope that a materialized view, created with or without the back fill option, keep updating with new source event data without executing any alter action on it. Any helps to me, please? Many thanksSolved10KViews0likes2CommentsT-SQL executing directly in Eventhouse KQL Queryset without comment (--) prefix
Hey everyone, I was just exploring Eventhouse and noticed something weird. The docs say you need to add "--" on its own line before T-SQL queries in KQL querysets, but I'm running T-SQL commands just fine without it. I never added the "--" prefix and all T-SQL queries executed normally. Is this a recent change? The documentation still mentions needing the comment prefix to switch to T-SQL mode, but it seems like the query editor is just auto-detecting it now? Reference: https://learn.microsoft.com/en-us/kusto/query/t-sql?view=microsoft-fabricSolved3.9KViews0likes7Commentskusto error- partial query failure
hi all, error - Details: "Query execution has exceeded the allowed limits (80DA0001): Partial query failure: Runaway query (E_RUNAWAY_QUERY). (message: The Join output block has exceeded the memory budget during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits).: ). [0]Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException: Query execution has exceeded the allowed limits (80DA0001): Partial query failure: Runaway I am trying to join 5 tables in kusto to fetch all required columns , 3 tables from 1 cluster ans 2 from another cluster. But query is failing with memory error Dummy query : let onboarded = Pln | project STId; let regAss = external_table('PR') | where Region == 'France' | where Status !in ("Error", "Rejected") | project STId; let exeAppr = external_table('PR') | where Region == "France" and Status in ("Ready", "Approved") | project STId, Region, Status; let QReferenced= external_table('PQR') | where Region == 'France' and Status == "Completed" | project STId, Region, Status,RequestId; let Regionpsllls = cluster("cp.ws.kusto.windows.net").database("pslll").pslll | where State == "Active" and pslllType == "Region" | project pslllId, pslllName = Name, pslll_Reg = RegionName; let pslllServices = cluster("cp.ws.kusto.windows.net").database("pslll").PFL | where ProductType == "Service" | project pslllId, SOId; let pslll_Reg_Serv = Regionpsllls | join kind=inner pslllServices on pslllId | project pslll_Reg, pslllName, SOId; // Join onboarded with pslll let Onboardedpslll = onboarded | join kind=inner pslll_Reg_Serv on $left.STId == $right.SOId; // Join region assigned with pslll let regAsspslll = regAss | join kind=inner pslll_Reg_Serv on $left.STId == $right.SOId; // Join execution approved with pslll let exeApprpslll = exeAppr | join kind=inner pslll_Reg_Serv on ($left.STId == $right.SOId and $left.Region == $right.pslll_Reg); //join quota refernced with pslll let QReferencedpslll = QReferenced | join kind=inner pslll_Reg_Serv on ($left.STId == $right.SOId and $left.Region == $right.pslll_Reg); // Combine using inner joins (only psllls present in all three sets) Onboardedpslll | join kind=inner regAsspslll on pslll_Reg, pslllName | join kind=inner exeApprpslll on pslll_Reg, pslllName | join kind=inner QReferencedpslll on pslll_Reg, pslllName | project pslll_Reg, pslllName ( i want many more cols to project)Solved7.8KViews0likes4CommentsPartitioning policy is not copied to DatabaseSchema.kql
I have added a custom partitioning policy to some materialized views and tables like this: .alter materialized-view DimResponse policy partitioning "{\"PartitionKeys\":[{\"ColumnName\":\"OrgId\",\"Kind\":\"Hash\",\"Properties\":{\"Function\":\"XxHash64\",\"MaxPartitionCount\":128,\"Seed\":1,\"PartitionAssignmentMode\":\"Uniform\"}}],\"EffectiveDateTime\":\"2023-01-01T00:00:00.000000Z\",\"MinRowCountPerOperation\":0,\"MaxRowCountPerOperation\":0,\"MaxOriginalSizePerOperation\":0}" I have connected to source control, but in DatabaseSchema.kql, the policy is not in the schema, is this a limitation or a bug? My update policies are applied in the DatabaseSchema.kql, so something is working when altering materialized views and tables.Solved3.2KViews0likes1Comment