Managing blob snapshots when using Azure Data Lake...

a-g-g · ‎12-19-2022

Hello All,

I'm trying out the option to use an Azure Data Lake Gen2 storage account for Power BI Dataflows.

I've connected a test workspace to my storage account, and can see the "powerbi" container with subfolders for the individual tables within the dataflow.

Now, every time I refresh a dataflow, a new blob snapshot is created within the storage account. This means that regular refreshes will soon give me a large pile of obsolete snapshots.

Does anyone know whether the Power BI Service will automatically clean up the obsolete snapshots? I assume it must do this when its own internal storage is used for dataflows.

Thanks.

a-g-g · ‎01-04-2023

The Lifecycle Management feature of Azure Data Lake Storage provides an option to "Limit blobs with filters", but the filter syntax does not allow paths in the format used within the "powerbi" container that hosts the dataflow blobs. The path within the "powerbi" container has a hierarchy that uses dataflow and table names, which may include characters (e.g. uppercase characters, spaces) that are not allowed within a blob filter.

Anonymous · ‎12-20-2022

Hi @a-g-g ,

There is a document for Configuring dataflow storage to use Azure Data Lake Gen2, hope it is helpfule for you.

Configuring dataflow storage to use Azure Data Lake Gen 2 - Power BI | Microsoft Learn

Thanks for your efforts & time in advance.

Best regards,
Community Support Team_Binbin Yu

a-g-g · ‎12-20-2022

Thanks @Anonymous for your reply. I had used that document to set up the configuration but didn't see anything about blob management. However, reading it again, I now see it says "We only write to this storage account and do not currently delete data." Therefore I will try setting up a Lifecycle management policy in my storage account to delete obsolete blob snapshots.