kql queryset

4 Topics

Best practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.
Solved
ShivekMaharaj
15 days ago Place Real-Time Intelligence
96Views
0likes
2Comments
Error in Microsoft Applied Skills: Implement a Real-Time Intelligence Assessment Lab : Need help
Hello All, since a week I was stuck on one error in Microsoft Applied Skills: Implement a Real-Time Intelligence solution with Microsoft Fabric Lab. Did anyone face the same issue if so, can anyone help me out how to resolve it. Attaching the error and related images below. Error: It says " unable to load container blobs. Verify that workspace identity is enabled and has storage blob data reader access to the storage account" . I am familiar with this permission and have enabled it in my own azure environment for my own blob that I have created but since these are pre created by Microsoft, I don't have access to these storage blobs and hence I am not sure how to proceed here. I have attempted this twice but I was stuck at the same point. I really appreciate your help and taking time in looking into my post. Thank you so much.
Solved
AshikaAlvakonda
3 months ago Place Real-Time Intelligence
1.8KViews
0likes
7Comments
Issue Writing to OneLake-Enabled KQL Database Delta Tables
I enabled OneLake availability for my KQL database so the tables are exposed as Delta or Parquet files. I’m able to read the table using Parquet without any issues: df = spark.read.format("parquet").load(abfsspath) However, when I try to overwrite or append data to the table, it fails: append_df.write.format("delta").mode("append").save(abfsspath) I receive the following error: Operation failed: "Forbidden", 403, AuthorizationPermissionMismatch "This request is not authorized to perform this operation using this permission." I am the workspace admin, and I don’t want to ingest data using the conventional KQL ingestion methods because they are too slow for my use case. I specifically need to write records directly using Delta or Parquet. Could you help me understand how to enable this write operation for Delta tables?
Solved
Dinzz
6 months ago Place Real-Time Intelligence
4.3KViews
0likes
8Comments
Architecture & GPS hardware
Hello Fabric community, I'm building a fleet monitoring platform for trucks/vehicles and need architectural guidance for using Microsoft Fabric Real-Time Intelligence, as well as GPS hardware recommendations. MY SCENARIO: ------------ - GPS trackers sending location data every 10-60 seconds via TCP - Data includes: GPS coordinates, speed, fuel level, engine RPM, temperature sensors, driver behavior (harsh braking, acceleration) - Need to support multiple tenants (each customer sees only their own fleet) - Planning to scale from 50 vehicles to 1000+ vehicles - Want real-time dashboards + historical analytics + automated alerts - Future: add video/dashcam capabilities MY PLANNED ARCHITECTURE: ------------------------ GPS Devices → TCP Server (Traccar/custom) → Azure Event Hub → Fabric Eventstream → Eventhouse → Real-Time Dashboard QUESTIONS: GPS Hardware: ------------- 1. What GPS tracker brands/models do you recommend that integrate well with custom backends and Microsoft Fabric? I'm considering Teltonika FMC920, Queclink, or Concox - any experience with these? 2. Which GPS devices have the best open protocols for parsing data and sending to Azure/Fabric? 3. Any recommendations for GPS trackers with OBD-II support to read vehicle diagnostics (fuel, RPM, error codes)? Microsoft Fabric Architecture: ------------------------------ 4. Is Eventstream + Eventhouse the right approach for high-frequency GPS telemetry data? 5. What's the recommended way to implement multi-tenancy (row-level security) for different fleet customers? 6. Should I use KQL Database or Lakehouse for storing historical GPS data (months/years)? 7. Any recommendations for handling geofencing alerts in real-time using Activator? 8. What Fabric SKU (F4, F8, F16?) would you recommend starting with for ~100 vehicles sending data every 30 seconds? I'm based in Mexico and planning to deploy this commercially for fleet owners. Any guidance from people who have built similar IoT/telematics solutions with Fabric would be greatly appreciated! Thanks in advance!
Solved
jhrjdata
6 months ago Place Real-Time Intelligence
6.7KViews
0likes
4Comments