grcanosaalstom's avatar
grcanosaalstom
New Member
2 days ago
Status:
New

PowerBI Delta Lake Connector without using Azure.DataLake

Current native connector to read delta lake tables in azure storage account using powerBI is to do this:

Source = AzureStorage.DataLake("https://STORAGE_ACCOUNT_NAME.dfs.core.windows.net/path/to/table/",[HierarchicalNavigation=true]),
Custom1 = DeltaLake.Table(Source),

This present a very large overcost in terms of "List" operations in the storageaccount since the AzureStorage.DataLake needs to list ALL parquet files even if a lot of them may be inactive.

A better approach would be to only list hte metadata files and then read only the active files directly.
This way, using the delta-lake version, you could also avoid future reads if the version has not changed since last read. 

I would like an improved PowerBI DeltaLake connector that actually uses PowerBI metadata. 

No CommentsBe the first to comment

Recent ideas