Forum Discussion
How to Integrate Log Analytics Workspace to Eventhouse
Hi nandasatria ,
You can export the data from the Log Analytics workspace to an Azure Storage Account or Azure Event Hubs. This can be done using the Log Analytics workspace data export feature. You can then create an Eventhouse and then fetch the data from the event hub into the KQL database in the Eventhouse.
For more details, please refer:
Log Analytics workspace data export in Azure Monitor - Azure Monitor | Microsoft Learn
Create an eventhouse - Microsoft Fabric | Microsoft Learn
Get data from Azure Event Hubs - Microsoft Fabric | Microsoft Learn
If you prefer to continue using Blob Storage, you can set up a scheduled export from a log query using Azure Logic Apps. This method allows you to query data from a Log Analytics workspace and send it to Azure Storage.
Best Regards,
Neeko Tang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hi Anonymous
That was great! I successfully stored the data from Log Analytics using Data Exporter and Event Hub to the KQL database in the Eventhouse. However, the problem is with the data structure; it is in the project fields, not parsed data. When I try to add a transformation event to expand the data, it shows an error: "Please connect this node to the upstream node with a schema before configuring it."
Is this a common issue? Do we need to manually use KQL, like using mv-expand to expand the data in the array and project for every field in the records?
Or is there another best practice to make the data more ready to use?
Thanks
Hi nandasatria ,
The error message "Please connect this node to the upstream node with a schema before configuring it" typically indicates that the data schema is not properly defined or recognized in the transformation pipeline. This can happen when the data is not parsed correctly or when the schema is not propagated through the nodes.
It is recommended to use the Kusto Query Language (KQL) to parse and expand data. the “mv-expand” operator can be used to expand arrays and nested data structures. The following is an example:
```kusto let rawData = datatable (RawData: dynamic) [ dynamic({"field1": "value1", "field2": ["value2a", "value2b"]}), dynamic({"field1": "value3", "field2": ["value2c", "value2d"]}) ]; rawData | mv-expand field2 | project field1, field2 ```Best Regards,
Neeko Tang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hi Anonymous
Yes, currently I use KQL to parse the data manually, but the problem is that I need to project every field, which requires a lot of effort if the table is large. Moreover, when we need to aggregate incoming data for various analyses, we have to use mv-expand and project for each field repeatedly
This is my current Event Stream Flow:
Is it typical for data from Event Hub to come without a schema, or is the schema from the data in Event Hub not recognized. How to make this more seamless for many aggregation ?
Thanks
Hi nandasatria ,
When data is sent to Event Hub, it is typically in a raw format (e.g., JSON, Avro, or plain text). Event Hub itself does not enforce or recognize a schema; it simply acts as a message broker. As a result:
The data may arrive in a nested or semi-structured format (e.g., JSON arrays or objects).
Eventhouse (KQL database) does not automatically infer the schema unless explicitly defined during ingestion or transformation.
This is why you often need to manually parse and flatten the data using KQL functions like parse_json, mv-expand, and extend.
You can try to use Azure Data Factory to transform the data and write it to Eventhouse in a flattened format or use a notebook to preprocess the data and write it back to Eventhouse.
Best Regards,
Neeko Tang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.