Inconsistent CSV Dataflow Refresh Times

[Anonymous]

Hi Everyone,

I have a simple dataflow (Premium workspace) that connects to multiple csv/excel files in a sharepoint folder on an hourly refresh schedule. I have noticed that the refresh times vary from 1 min to 7 hours! Does anyone know what could be causing this?

I notice the long refresh times tend to happen between 6AM to 12PM Australian Eastern Standard Time (AEST). I am not performing any complex transformations such as merging or grouping. Only pulling data from the csv file and changing column types.

1 of the CSV files is quite large (~1.2M rows, 400MB) , I am trying to implement an incremental refresh policy on this table

Any help on this would be much appreciated!

[v-cazheng-msft]

Hi @Anonymous ,

From the screenshot, we can see that refresh duration is not too long, but their wait time is a little bit long. So during the refresh, too much time is costed on waiting for the resource such as memory to handle the refresh.

May I know how many datasets in your Premium workspace? Is there any other scheduled refresh adjacent to the scheduled refresh time of current dataset? If you move this dataset to a new Premium capacity and refresh there, will you see the refresh duration be long?

Best Regards,

Community Support Team _ Caiyun

[Anonymous]

Hi Caiyun,

Thanks for responding.

There are only 6 other dataflows in the workspace (No datasets). None of these dataflows share a scheduled refresh during the problematic time period (6AM-12PM). Although there is 1 dataflow that refreshes roughly at 5AM. I have copied the dataflow to a seperate, empty premium workspace and there is no difference in refresh performace. 

Is there any way to narrow down what is costing the memory resource?