Azure Data Lake Gen2 CDM folder refresh times

[PowerBIU10]

Hey all,

We've recently configured Azure Synapse Link for Dataverse to export all selected tables to a Data Lake Gen2 storage (not using Azure Synapse Workspaces, data goes straight from Dataverse to Data Lake). 

There, one can get the data by connecting to the CDM folder of that Data Lake storage
(see: https://docs.microsoft.com/en-us/powerapps/maker/data-platform/export-to-data-lake-data-powerbi)

The problem concerns the refresh time of that CDM folder:

Let's say I add row in onw of the tables I'm exporting. 

The Synapse Link adds it to ADLS very fast, takes a few seconds. 

On the Power BI side, it takes at least 30 minutes to show.

Why is that?

Really any help would be much appreciated!

[stuartreid · Accepted solution]

Hello.

Don't know if you got an answer to this one yet... but I'll give it a shot.

When you're using the CDM folders to pull in data from the datalake, it's actually using the snapshot and not reading through all the individual files.

You're snapshot is probably being updated every 30 minutes.

Hope that helps.

[Anonymous]

Hi @PowerBIU10 ,

Please check this blog, it may be helpful to you. The conclusions inside introduce the reasons for the slowness.

Best Regards,

Stephen Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[PowerBIU10]

Thank you! 

Though I am looking at why the CDM folder itself takes time to reflect changes in the ADLS?

The CSVs have been added new data, but the CDM does not show these changes for 30 minutes to 1 hour.

Thanks again,