Forum Discussion
Can't setup Incremental Refresh for Dataflow to SnowFlake
Super UserLet me rephrase that.
Unlike a Power BI dataset (where you need to define the RangeStart and RangeEnd parameters in Power Query on the desktop) for dataflows the Power BI service creates these parameters for you when you indicate that you want incremental refresh (and specify the datetime column for it). Power BI service will also automatically add the filter step to the end of the dataflow M code (which is stupid, in my opinion - it should be possible to specify that filter much earlier in the code).
If you had manually created the parameters and then try to enable incremental refresh for a data flow all hell will break loose. Most of the time you will lose access to the dataflow and will have to delete and redo it.
I understand the response you are giving (makes total sense), but what I can't figure out is how do I go about limiting the table size for design purposes if I can't add any parameters? When working with datasets that's not a problem because RangeStart and RangEnd are override by the incremental refresh ones, but in this case I have no other choice but load all the data every time I need to work on the dataflow design? (BTW, in my case I'm working on a Datamart, but it behaves pretty much like a dataflow in this sense).
- lbendlin
you can have parameters. What you cannot have is control over the RangeStart and RangeEnd parameters - these are entirely controlled by the dataflow incremental refresh setup process. As Anonymous mentioned your responsibility is to provide the datetime (or dateinteger) field that these parameters will be applied to. Not by you, by the dataflow checker.
Sure, I can have parameters, but if I use them to limit my query result to work on the datamart design, then I have to -at some point- remove all temp parameter limitations so the incremental refresh ones can take full control during refresh, which takes a long time to load. I was hoping there was an alternative way (like they did with datasets), but I guess I'll have to live with that.
Thanks for your response, dough.
- lbendlin
That's a much deeper question. As you noticed unlike with datasets there is no way to change parameter values for dataflows in the service. If this is important to you please consider voting for an existing idea or raising a new one at https://ideas.powerbi.com
In the meantime, set a ridiculously low threshold for incremental refresh (one month/one day) for testing and then once you confirmed it working you can set the true thresholds.