Forum Discussion
Dataflow Gen2 Incremental Refresh and Data Refresh
Hi!
My name is Jeroen Luitwieler, and I’m a Senior Product Manager on the Dataflow team. While this guide shows how to use an OData source, it’s just an example. You can use this method with your on-premises SQL Server without any issues. If you want to perform a full refresh of your data every time you refresh, you can change the current data destination setting from “append” to “replace”.
Hope this resolves your issue! 😁
Hi BeyzaKzlky
Thanks for using Fabric Community.
As I understand the above question, you are trying to achieve incremental load using Dataflow Gen2.
Incrementally amassing data in a data destination requires a technique to load only new or updated data into your data destination. This technique can be done by using a query to filter the data based on the data destination.
Please refer to this link for more information: https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2
Hope this helps. Please let us know if you have any further queries.
Hi Anonymous ,
Thanks for your response. I have seen the learning page which you have shared with me, but it's OData source. Unfortunately not proper for my case.
I'm getting data from on-prem sql server.
And also there is a data duplication problem with dataflow like I have mentioned above.
I think you can use that link from Anonymous also with on-prem sql server.
That procedure is only to append rows to the Lakehouse table.
Not update (change) existing data in the Lakehouse table.
So it's called Incrementally amass data.
I think the main thinking behind that procedure is that you connect your Dataflow Gen2 to your Lakehouse table, to find out what is the newest row in your Lakehouse table. Then you use that information to filter your on-prem sql server data, so you ask the sql server to only return rows which are newer than your newest row in the Lakehouse. Then you append this new data from sql server into the Lakehouse.
When using that procedure, it shouldn't become duplicate rows, because your query to the SQL server is only asking for new rows.
There is an idea for Incremental refresh (which includes both append and updates to existing data), please vote:
Microsoft IdeaHi!
My name is Jeroen Luitwieler, and I’m a Senior Product Manager on the Dataflow team. While this guide shows how to use an OData source, it’s just an example. You can use this method with your on-premises SQL Server without any issues. If you want to perform a full refresh of your data every time you refresh, you can change the current data destination setting from “append” to “replace”.
Hope this resolves your issue! 😁
Hi,
Thank you very much for your response.
Yes I know the option which you have mentioned, but we cannot see that option since yesterday, also data destination selection is greyed out like below:
Also, we set scheduled refresh from the settings of dataflow:
and also I want to mention any other problem scheduled refresh is not triggered randomly.
Is there any limit to show data on data transformation in dataflow? It cuts the operation while merging (inner join) multiple tables after 2 million rows.
I will try the mentioned example.
Thanks for your help.
Add data destination is greyed out if the query already has a destination.
It is possible to edit the Destination settings by clicking on the settings icon (cog wheel) in the bottom right corner of the screen:
I don't have information to answer about the other questions, someone else answer please 🙂
I can confirm that it works - with that example i made incremental refreshes from on prem sql server: query warehouse for max date (or entry no, if Your tables have them) and filter source sql server, check that it works correctly and query folding works. I have also implemented pipelines and stored procedures to delete some data from warehouse and refresh a week or a month old of data.
Hi,
I double checked your metioned setting this morning. I'm already selecting replace for tables.
Have a nice day.
Hi BeyzaKzlky
Apologies for the issue you have been facing. Are you still facing this issue?
If yes we will respond back with more details and try to help.
Thanks.
