Forum Discussion
Temporarily disable incremental refresh for dev/testing purposes
Hi,
it's basically what the subject says. Is there any way to temporarily and globally disable incremental refresh on a specific dataset for development/testing purposes?
When you've got a golden dataset (that's still getting bigger) and you have maybe 10-15 fact tables where incremental refresh is configured and enabled, it isn't very efficient having to disable it, just for testing purposes, and then having to configure EVERYTHING again (and maintaining a list of the affected tables) because it doesn't even store the previous settings (archival timespan, refresh timespan, etc.).
I just want to be able to use the RangeStart/RangeEnd parameters set to maybe 1 or 2 months' worth of data to test the live reports that are connected to the golden dataset and check that everything works as expected without having to wait maybe 30 minutes for the dataset to go through a full refresh cycle; which is also a burden on the respective databases.
Hey Anonymous ,
I recommend reading this article about incremental refresh using Tabular Editor: https://docs.tabulareditor.com/te2/incremental-refresh.html
Pausing incremental refresh for one or more tables inside a dataset is an advanced task, that requires a deep understanding of refresh policies and how to tinker with the current settings, and restart the incremental refresh when you are done with development/testing.
My recommendation would be to use two different datasets at all, a small one (but still large enough to be representative in regards of model complexity and of course of size). Then you can use Tabular Editor to deploy the model to production, of course "new/changed" columns from tables affected by incremental refresh need to be reprocessed. You can also use the ALM toolkit to deploy changes w/o affecting partitioned tables: http://alm-toolkit.com/Hopefully, this helps to tackle your challenge
Regards,
Tom
2 Replies
- TomMartens
Super User
Hey Anonymous ,
I recommend reading this article about incremental refresh using Tabular Editor: https://docs.tabulareditor.com/te2/incremental-refresh.html
Pausing incremental refresh for one or more tables inside a dataset is an advanced task, that requires a deep understanding of refresh policies and how to tinker with the current settings, and restart the incremental refresh when you are done with development/testing.
My recommendation would be to use two different datasets at all, a small one (but still large enough to be representative in regards of model complexity and of course of size). Then you can use Tabular Editor to deploy the model to production, of course "new/changed" columns from tables affected by incremental refresh need to be reprocessed. You can also use the ALM toolkit to deploy changes w/o affecting partitioned tables: http://alm-toolkit.com/Hopefully, this helps to tackle your challenge
Regards,
Tom
Hi TomMartens ,
Thanks for pointing me in the right direction! I will look closely into Tabular Editor, ALM, and the article mentioned.
Schönes Wochenende 😉