Forum Discussion
Performance and branch handling using azure blob storage
Hi jaryszek ,
Thank you for engaging with the Microsoft Fabric Community. Switching from Azure Blob Storage to Azure Data Lake Storage Gen2 is a solid decision, especially if you want better folder management and need to preserve Git branch structures for downstream use in Power BI.
If your scenario involves dynamically retrieving files from branch specific folders, you can use Power Query parameters along with the AzureStorage. DataLakeContents function. This allows you to list files within a specific folder dynamically without loading unnecessary data. This method is helpful when users need to explore or select different branches interactively.
Reference: Azure Data Lake Storage Gen2 - Power Query | Microsoft Learn
However, if you already know the exact file paths, the best practice for performance is to connect directly to those files using the ADLS Gen2 connector with parameters. This avoids scanning folder contents, significantly reduces metadata overhead, and improves data refresh speed, especially in large file systems.
— Yugandhar
Community Support Team.
Hi jaryszek ,
Thank you for engaging with the Microsoft Fabric Community. Switching from Azure Blob Storage to Azure Data Lake Storage Gen2 is a solid decision, especially if you want better folder management and need to preserve Git branch structures for downstream use in Power BI.
If your scenario involves dynamically retrieving files from branch specific folders, you can use Power Query parameters along with the AzureStorage. DataLakeContents function. This allows you to list files within a specific folder dynamically without loading unnecessary data. This method is helpful when users need to explore or select different branches interactively.
Reference: Azure Data Lake Storage Gen2 - Power Query | Microsoft Learn
However, if you already know the exact file paths, the best practice for performance is to connect directly to those files using the ADLS Gen2 connector with parameters. This avoids scanning folder contents, significantly reduces metadata overhead, and improves data refresh speed, especially in large file systems.
— Yugandhar
Community Support Team.
thanks!
One more question: Can i make parameters and use one Base Path to Azure Data Lake Storage and add in the end & "table1.csv" as dynamic variable within each query?
Best,
JacekApologies for the delayed response. Yes, you can create a base path parameter in Power Query and then append the file or folder name, such as table1.csv, to access the required resource. This method improves the flexibility and organization of your queries, making them more dynamic, particularly when managing multiple folders.
Thank You.
