Forum Discussion
Responsive ResidentIncremental Refresh with Linked/Referenced Tables?
Hey arock-well ,
as always it depends. A dataflow is a data source from the perspective of dataset. From my experience loading data from a dataflow if often much more faster than importing data from the data source directly. This reason for this simple, all the heavy lifting is done inside the dataflow. Loading data from a dataflow into a table should always be like this "SELECT * FROM TABLE." There is no need for any transformation, no need for data shaping.
But then, if you add 10k rows incrementally to a dataflow that already holds 1BN rows, then, of course loading 1BN rows to a dataset takes more time than adding only the increment.
Your choice, you pay faster dataset refresh with solution complexity.
What I do, load dataflows incrementally, to get the cleaned and beautified data as fast as possible, then I give the full load to the dataset a try.
The reason for this: I always want to keep my solution as fast, but also as simple as possible.
Hopefuyll, this helps to find your way.
Regards,
Tom
Super UserHey arock-well ,
If there are chains of dataflows, and you want to experience of shorter data refresh duration in subsequent stages you have to configure incremental refresh for all the instances of that table.
I do not understand your 2nd question, as i do not know how the referenced table will impact the increment or will be impacted by the impact.
Hopefully, this provides some help to tackle your challenge.
Regards,
Tom
Responsive ResidentTomMartens Thank you for that. I'm assuming I would also need to set up an incremental refresh for any datasets that use that computed entity, too?