Forum Discussion
DataFlow Incremental Refresh
The setup seems good to me.
"refreshing the table using Replace method works fine."
Are you saying this exactly same dataflow runs successfully if you choose Replace method, it only fails if you choose Append method?
So your M code is exactly the same in both cases. The only change is the destination settings?
So the issue seems to be related to the destination settings?
However, using Replace method will erase all the old data so you don't want to use it in your case. You want to Append to keep the old data and add new data.
Btw, I see there are a couple of known issues with Data Factory <> Oracle.
https://learn.microsoft.com/en-us/fabric/get-started/fabric-known-issues
Could the issue be relates to the number format? Could you try to force the number format to whole number (Int64)?
(Please do the testing in a test workspace in your Fabric, not production workspace.)
Tbh I'm confused why you are getting an error.
Just to make sure, the IncrementalEventID query has disabled staging, and this query does not have a destination, right?
Perhaps you could create a support ticket to have the support team look at the case.
Yes, my IncrementalOrderID (acuatlly in my model it is Event ID not Order ID) is the query which returns the current Max ID from my destination table. I believe it rerurns scalar as I did it excatly as in the tutorial. See the screen:
M code of my filter statement (filter step): You can see on the screen that the filter works as there a currently 25 rows with event_id greater than the IncrementalEventID (maimum Event ID). During the refresh those 25 rows should be added to my table. The destination for this query is the same existing table and the Update method is Append.
The source of my table is on-premise Orcale database but I guess surce type should not matter here - refreshing the table using Replace method works fine.
Any ideas what is wrong with this incremental refresh pattern? Thanks.
The setup seems good to me.
"refreshing the table using Replace method works fine."
Are you saying this exactly same dataflow runs successfully if you choose Replace method, it only fails if you choose Append method?
So your M code is exactly the same in both cases. The only change is the destination settings?
So the issue seems to be related to the destination settings?
However, using Replace method will erase all the old data so you don't want to use it in your case. You want to Append to keep the old data and add new data.
Btw, I see there are a couple of known issues with Data Factory <> Oracle.
https://learn.microsoft.com/en-us/fabric/get-started/fabric-known-issues
Could the issue be relates to the number format? Could you try to force the number format to whole number (Int64)?
(Please do the testing in a test workspace in your Fabric, not production workspace.)
Tbh I'm confused why you are getting an error.
Just to make sure, the IncrementalEventID query has disabled staging, and this query does not have a destination, right?
Perhaps you could create a support ticket to have the support team look at the case.
“Just to make sure, the IncrementalEventID query has disabled staging, and this query does not have a destination, right?”
That was it! The IncrementalEventID query had a destination set - when doing everything as in the tutorial it says to copy the original query and change it to get the max ID. Apparently it copies the destination somehow – that is what was set:
When I removed the destination by clicking “x” it all now works fine. They mentioned in the tutorial to disable staging but it’s a shame they don’t mention to remove the destination from this query.
Thank you very much for being such an investigator! That helped a lot.