Forum Discussion
Performance and branch handling using azure blob storage
Hello,
I am wondering to switch azue blob storage into azure data lake storage in order to get folder management possibilities.
I have github files which i will push into azure blob storage (using java code or github actions) but i need to keep git branch structure.
After that I will need to connect to specific folders and files (csv ones) in power bi.
What power query should i write in order to maintain the best performance if i will use azure data lake storage?
i should connect to specific folder = branch name directly and list all files using dynamic parameter?
Or should I connect strightforward to file using connector like here and using parameters for branch name, folder name and files names?
https://www.sqlbi.com/blog/marco/2020/05/29/optimizing-access-to-azure-data-lake-storage-adls-gen-2-in-power-query/
in order to maitain great performance?
Best
Jacek
Hi jaryszek ,
Thank you for engaging with the Microsoft Fabric Community. Switching from Azure Blob Storage to Azure Data Lake Storage Gen2 is a solid decision, especially if you want better folder management and need to preserve Git branch structures for downstream use in Power BI.
If your scenario involves dynamically retrieving files from branch specific folders, you can use Power Query parameters along with the AzureStorage. DataLakeContents function. This allows you to list files within a specific folder dynamically without loading unnecessary data. This method is helpful when users need to explore or select different branches interactively.
Reference: Azure Data Lake Storage Gen2 - Power Query | Microsoft Learn
However, if you already know the exact file paths, the best practice for performance is to connect directly to those files using the ADLS Gen2 connector with parameters. This avoids scanning folder contents, significantly reduces metadata overhead, and improves data refresh speed, especially in large file systems.
— Yugandhar
Community Support Team.
3 Replies
Hi jaryszek ,
Thank you for engaging with the Microsoft Fabric Community. Switching from Azure Blob Storage to Azure Data Lake Storage Gen2 is a solid decision, especially if you want better folder management and need to preserve Git branch structures for downstream use in Power BI.
If your scenario involves dynamically retrieving files from branch specific folders, you can use Power Query parameters along with the AzureStorage. DataLakeContents function. This allows you to list files within a specific folder dynamically without loading unnecessary data. This method is helpful when users need to explore or select different branches interactively.
Reference: Azure Data Lake Storage Gen2 - Power Query | Microsoft Learn
However, if you already know the exact file paths, the best practice for performance is to connect directly to those files using the ADLS Gen2 connector with parameters. This avoids scanning folder contents, significantly reduces metadata overhead, and improves data refresh speed, especially in large file systems.
— Yugandhar
Community Support Team.thanks!
One more question: Can i make parameters and use one Base Path to Azure Data Lake Storage and add in the end & "table1.csv" as dynamic variable within each query?
Best,
JacekApologies for the delayed response. Yes, you can create a base path parameter in Power Query and then append the file or folder name, such as table1.csv, to access the required resource. This method improves the flexibility and organization of your queries, making them more dynamic, particularly when managing multiple folders.
Thank You.
