Forum Discussion
Dataflow Gen2 Incremental Refresh and Data Refresh
Hi!
My name is Jeroen Luitwieler, and I’m a Senior Product Manager on the Dataflow team. While this guide shows how to use an OData source, it’s just an example. You can use this method with your on-premises SQL Server without any issues. If you want to perform a full refresh of your data every time you refresh, you can change the current data destination setting from “append” to “replace”.
Hope this resolves your issue! 😁
Hi,
Thank you very much for your response.
Yes I know the option which you have mentioned, but we cannot see that option since yesterday, also data destination selection is greyed out like below:
Also, we set scheduled refresh from the settings of dataflow:
and also I want to mention any other problem scheduled refresh is not triggered randomly.
Is there any limit to show data on data transformation in dataflow? It cuts the operation while merging (inner join) multiple tables after 2 million rows.
I will try the mentioned example.
Thanks for your help.
Hi BeyzaKzlky !
It is correct what fabricator1 mentioned and if a data destination is set you can edit it in the left bottom corner. However, the issue you are facing with not triggering a refresh if it was scheduled is quite interesting. Can you create a support ticket when you face this issue again so that our support team can look into it? There should also be no row limit for the operations that you are perfroming. You may want to consider to split the ingestion of the data and the merge in two separate queries so that the lakehouse/warehouse compute can engage and help you perform the inner join faster.