Forum Discussion
Dataflow Gen2 Incremental Refresh and Data Refresh
Hi!
My name is Jeroen Luitwieler, and I’m a Senior Product Manager on the Dataflow team. While this guide shows how to use an OData source, it’s just an example. You can use this method with your on-premises SQL Server without any issues. If you want to perform a full refresh of your data every time you refresh, you can change the current data destination setting from “append” to “replace”.
Hope this resolves your issue! 😁
Hi Anonymous ,
Thanks for your response. I have seen the learning page which you have shared with me, but it's OData source. Unfortunately not proper for my case.
I'm getting data from on-prem sql server.
And also there is a data duplication problem with dataflow like I have mentioned above.
Hi!
My name is Jeroen Luitwieler, and I’m a Senior Product Manager on the Dataflow team. While this guide shows how to use an OData source, it’s just an example. You can use this method with your on-premises SQL Server without any issues. If you want to perform a full refresh of your data every time you refresh, you can change the current data destination setting from “append” to “replace”.
Hope this resolves your issue! 😁
Hi,
Thank you very much for your response.
Yes I know the option which you have mentioned, but we cannot see that option since yesterday, also data destination selection is greyed out like below:
Also, we set scheduled refresh from the settings of dataflow:
and also I want to mention any other problem scheduled refresh is not triggered randomly.
Is there any limit to show data on data transformation in dataflow? It cuts the operation while merging (inner join) multiple tables after 2 million rows.
I will try the mentioned example.
Thanks for your help.
Add data destination is greyed out if the query already has a destination.
It is possible to edit the Destination settings by clicking on the settings icon (cog wheel) in the bottom right corner of the screen:
I don't have information to answer about the other questions, someone else answer please 🙂Hi BeyzaKzlky !
It is correct what fabricator1 mentioned and if a data destination is set you can edit it in the left bottom corner. However, the issue you are facing with not triggering a refresh if it was scheduled is quite interesting. Can you create a support ticket when you face this issue again so that our support team can look into it? There should also be no row limit for the operations that you are perfroming. You may want to consider to split the ingestion of the data and the merge in two separate queries so that the lakehouse/warehouse compute can engage and help you perform the inner join faster.
I can confirm that it works - with that example i made incremental refreshes from on prem sql server: query warehouse for max date (or entry no, if Your tables have them) and filter source sql server, check that it works correctly and query folding works. I have also implemented pipelines and stored procedures to delete some data from warehouse and refresh a week or a month old of data.
Hi,
I double checked your metioned setting this morning. I'm already selecting replace for tables.
Have a nice day.
Hi BeyzaKzlky
Apologies for the issue you have been facing. Are you still facing this issue?
If yes we will respond back with more details and try to help.
Thanks.
