Forum Discussion
Incremental Refresh with Linked/Referenced Tables?
Hey arock-well ,
as always it depends. A dataflow is a data source from the perspective of dataset. From my experience loading data from a dataflow if often much more faster than importing data from the data source directly. This reason for this simple, all the heavy lifting is done inside the dataflow. Loading data from a dataflow into a table should always be like this "SELECT * FROM TABLE." There is no need for any transformation, no need for data shaping.
But then, if you add 10k rows incrementally to a dataflow that already holds 1BN rows, then, of course loading 1BN rows to a dataset takes more time than adding only the increment.
Your choice, you pay faster dataset refresh with solution complexity.
What I do, load dataflows incrementally, to get the cleaned and beautified data as fast as possible, then I give the full load to the dataset a try.
The reason for this: I always want to keep my solution as fast, but also as simple as possible.
Hopefuyll, this helps to find your way.
Regards,
Tom
TomMartens Do I also need to set up a separate incremental refresh on any datasets that connect to an already incrementally refreshed dataflow? This post says no need to do that, but interested in your take on it: Solved: Re: Help with incremental refresh dataflow and dat... - Microsoft Fabric Community
Hey arock-well ,
as always it depends. A dataflow is a data source from the perspective of dataset. From my experience loading data from a dataflow if often much more faster than importing data from the data source directly. This reason for this simple, all the heavy lifting is done inside the dataflow. Loading data from a dataflow into a table should always be like this "SELECT * FROM TABLE." There is no need for any transformation, no need for data shaping.
But then, if you add 10k rows incrementally to a dataflow that already holds 1BN rows, then, of course loading 1BN rows to a dataset takes more time than adding only the increment.
Your choice, you pay faster dataset refresh with solution complexity.
What I do, load dataflows incrementally, to get the cleaned and beautified data as fast as possible, then I give the full load to the dataset a try.
The reason for this: I always want to keep my solution as fast, but also as simple as possible.
Hopefuyll, this helps to find your way.
Regards,
Tom