Forum Discussion
Azure Application Insights Continuous Export to Power BI
- 9 years ago
I took another crack at it and I think I figured it out. It first required tweaking some transform queries that PowerBI tries to do to the blob binary data which strips out some important characters in the json serialized strings. Then after that, I had to parse the strings as JSON and expanding the data from there.
I'm surprised considering Continuous Export from AAI is a Microsoft product/feature along with PowerBI, there isn't already some blog post about making this integration work seamlessly (or is this reserved for the BI Pros?) so if anybody needs it, here is the advanced query where you just replace the "<blob_storage_container_url>" with your own url path to your blob container with the blobs stored from AAI. From there it's just expanding the custom.dimensions column to grab all the properties you track and these are unique to whatever your custom event data contains.
let Source = AzureStorage.Blobs("<blob_storage_container_url>"), #"Invoke Custom Function1" = Table.AddColumn(Source, "JsonTransform", each Lines.FromBinary([Content],null,null,1252)), #"Renamed Columns1" = Table.RenameColumns(#"Invoke Custom Function1", {"Name", "Source.Name"}), #"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1", {"Source.Name", "JsonTransform"}), #"ExpandedJsonTransform" = Table.ExpandListColumn(#"Removed Other Columns1", "JsonTransform"), #"Parsed JSON" = Table.TransformColumns(#"ExpandedJsonTransform",{{"JsonTransform", Json.Document}}), #"Expanded Transform File from Query1" = Table.ExpandRecordColumn(#"Parsed JSON", "JsonTransform", {"event", "internal", "context"}, {"event", "internal", "context"}), #"Expanded event" = Table.ExpandListColumn(#"Expanded Transform File from Query1", "event"), #"Expanded event1" = Table.ExpandRecordColumn(#"Expanded event", "event", {"name", "count"}, {"event.name", "event.count"}), #"Expanded context" = Table.ExpandRecordColumn(#"Expanded event1", "context", {"application", "data", "device", "user", "session", "operation", "location", "custom"}, {"application", "data", "device", "user", "session", "operation", "location", "custom"}), #"Expanded custom" = Table.ExpandRecordColumn(#"Expanded context", "custom", {"dimensions"}, {"custom.dimensions"}) in #"Expanded custom"
I took another crack at it and I think I figured it out. It first required tweaking some transform queries that PowerBI tries to do to the blob binary data which strips out some important characters in the json serialized strings. Then after that, I had to parse the strings as JSON and expanding the data from there.
I'm surprised considering Continuous Export from AAI is a Microsoft product/feature along with PowerBI, there isn't already some blog post about making this integration work seamlessly (or is this reserved for the BI Pros?) so if anybody needs it, here is the advanced query where you just replace the "<blob_storage_container_url>" with your own url path to your blob container with the blobs stored from AAI. From there it's just expanding the custom.dimensions column to grab all the properties you track and these are unique to whatever your custom event data contains.
let
Source = AzureStorage.Blobs("<blob_storage_container_url>"),
#"Invoke Custom Function1" = Table.AddColumn(Source, "JsonTransform", each Lines.FromBinary([Content],null,null,1252)),
#"Renamed Columns1" = Table.RenameColumns(#"Invoke Custom Function1", {"Name", "Source.Name"}),
#"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1", {"Source.Name", "JsonTransform"}),
#"ExpandedJsonTransform" = Table.ExpandListColumn(#"Removed Other Columns1", "JsonTransform"),
#"Parsed JSON" = Table.TransformColumns(#"ExpandedJsonTransform",{{"JsonTransform", Json.Document}}),
#"Expanded Transform File from Query1" = Table.ExpandRecordColumn(#"Parsed JSON", "JsonTransform", {"event", "internal", "context"}, {"event", "internal", "context"}),
#"Expanded event" = Table.ExpandListColumn(#"Expanded Transform File from Query1", "event"),
#"Expanded event1" = Table.ExpandRecordColumn(#"Expanded event", "event", {"name", "count"}, {"event.name", "event.count"}),
#"Expanded context" = Table.ExpandRecordColumn(#"Expanded event1", "context", {"application", "data", "device", "user", "session", "operation", "location", "custom"}, {"application", "data", "device", "user", "session", "operation", "location", "custom"}),
#"Expanded custom" = Table.ExpandRecordColumn(#"Expanded context", "custom", {"dimensions"}, {"custom.dimensions"})
in
#"Expanded custom"
Hi
I am experiencing the same problem as listed above when i try to import Application Insights data that is continuously exported to Azure Storage Account in blobs.
When i import blob data into PowerBI, and used your query that is referred below, invoke Custom Function step is failing.
Below is my query
let
Source = AzureStorage.Blobs("XXX"),
XXX1= Source{[Name="XXX"]}[Data],
#"Invoke Custom Function1" = Table.AddColumn(Source, "JsonTransform", each Lines.FromBinary([Content],null,null,1252)),
#"Renamed Columns1" = Table.RenameColumns(#"Invoke Custom Function1", {"Name", "Source.Name"}),
#"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1", {"Source.Name", "JsonTransform"}),
#"ExpandedJsonTransform" = Table.ExpandListColumn(#"Removed Other Columns1", "JsonTransform"),
#"Parsed JSON" = Table.TransformColumns(#"ExpandedJsonTransform",{{"JsonTransform", Json.Document}}),
#"Expanded Transform File from Query1" = Table.ExpandRecordColumn(#"Parsed JSON", "JsonTransform", {"event", "internal", "context"}, {"event", "internal", "context"}),
#"Expanded event" = Table.ExpandListColumn(#"Expanded Transform File from Query1", "event"),
#"Expanded event1" = Table.ExpandRecordColumn(#"Expanded event", "event", {"name", "count"}, {"event.name", "event.count"}),
#"Expanded context" = Table.ExpandRecordColumn(#"Expanded event1", "context", {"application", "data", "device", "user", "session", "operation", "location", "custom"}, {"application", "data", "device", "user", "session", "operation", "location", "custom"}),
#"Expanded custom" = Table.ExpandRecordColumn(#"Expanded context", "custom", {"dimensions"}, {"custom.dimensions"})
in
#"Expanded custom"
Please help!
Best Regards,
sherin