Forum Discussion

nandasatria's avatar
nandasatria
Frequent Visitor
1 year ago
Solved

How to Integrate Log Analytics Workspace to Eventhouse

I need to analyze data from a Log Analytics workspace in Fabric.

Is it possible to integrate it with Eventhouse so I can run Kusto Queries from Eventhouse while the data remains in the Log Analytics workspace?

Or do we need to move the data from the Log Analytics workspace to Eventhouse? If yes, how can we do it?

Currently, I need to export the data to Blob Storage and use it in a notebook, but in that case, I cannot use KQL

 

Thanks

eventhouse
kql database
kql queryset

6 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi nandasatria ,

     

    You can export the data from the Log Analytics workspace to an Azure Storage Account or Azure Event Hubs. This can be done using the Log Analytics workspace data export feature. You can then create an Eventhouse and then fetch the data from the event hub into the KQL database in the Eventhouse.

    For more details, please refer:

    Log Analytics workspace data export in Azure Monitor - Azure Monitor | Microsoft Learn

    Create an eventhouse - Microsoft Fabric | Microsoft Learn

    Get data from Azure Event Hubs - Microsoft Fabric | Microsoft Learn

     

    If you prefer to continue using Blob Storage, you can set up a scheduled export from a log query using Azure Logic Apps. This method allows you to query data from a Log Analytics workspace and send it to Azure Storage.

    Export data from a Log Analytics workspace to a storage account by using Logic Apps - Azure Monitor | Microsoft Learn

     

    Best Regards,

    Neeko Tang

    If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

     

  • nandasatria's avatar
    nandasatria
    Frequent Visitor
    1 year ago

    Hi Anonymous 

    That was great! I successfully stored the data from Log Analytics using Data Exporter and Event Hub to the KQL database in the Eventhouse. However, the problem is with the data structure; it is in the project fields, not parsed data. When I try to add a transformation event to expand the data, it shows an error: "Please connect this node to the upstream node with a schema before configuring it."

    Is this a common issue? Do we need to manually use KQL, like using mv-expand to expand the data in the array and project for every field in the records?

    Or is there another best practice to make the data more ready to use?

     

    Thanks

    • Anonymous's avatar
      Anonymous
      Not applicable
      1 year ago

      Hi nandasatria ,

       

      The error message "Please connect this node to the upstream node with a schema before configuring it" typically indicates that the data schema is not properly defined or recognized in the transformation pipeline. This can happen when the data is not parsed correctly or when the schema is not propagated through the nodes.

       

      It is recommended to use the Kusto Query Language (KQL) to parse and expand data. the “mv-expand” operator can be used to expand arrays and nested data structures. The following is an example:

         ```kusto
   let rawData = datatable (RawData: dynamic)
   [
       dynamic({"field1": "value1", "field2": ["value2a", "value2b"]}),
       dynamic({"field1": "value3", "field2": ["value2c", "value2d"]})
   ];
   rawData
   | mv-expand field2
   | project field1, field2
   ```

       

      Best Regards,

      Neeko Tang

      If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

      • nandasatria's avatar
        nandasatria
        Frequent Visitor
        1 year ago

        Hi Anonymous 

        Yes, currently I use KQL to parse the data manually, but the problem is that I need to project every field, which requires a lot of effort if the table is large. Moreover, when we need to aggregate incoming data for various analyses, we have to use mv-expand and project for each field repeatedly

        This is my current Event Stream Flow:

         

        Is it typical for data from Event Hub to come without a schema, or is the schema from the data in Event Hub not recognized. How to make this more seamless for many aggregation ?

         

        Thanks