Forum Discussion
Syed_Aamiruddin
3 years agoRegular Visitor
Reduce refresh time in incremental refresh
I have setup an incremental refresh policy which archives past 24 months of data and refreshes past 1 day with an offset on 1 day. While doing the monthly partitions refreshes for the fact table which contains around 156 million rows for one month it takes over four hours. Is there anything that can be done to reduce the refresh time of the partition? Executing the same query on databricks for one months data takes less than 5 mins. Any help would be appreciated.
2 Replies
- AnonymousNot applicable
Curious, what are the reasons for refreshing a monthly partition ? Is the workspace using a P-SKU?
- Syed_AamiruddinRegular Visitor
Hi, yes it is a ppu workspace. I found one solution, the hierarchy part of the column was taking too much space so disabled it and it works now.