Forum Discussion
DataFlow Incremental Refresh
The setup seems good to me.
"refreshing the table using Replace method works fine."
Are you saying this exactly same dataflow runs successfully if you choose Replace method, it only fails if you choose Append method?
So your M code is exactly the same in both cases. The only change is the destination settings?
So the issue seems to be related to the destination settings?
However, using Replace method will erase all the old data so you don't want to use it in your case. You want to Append to keep the old data and add new data.
Btw, I see there are a couple of known issues with Data Factory <> Oracle.
https://learn.microsoft.com/en-us/fabric/get-started/fabric-known-issues
Could the issue be relates to the number format? Could you try to force the number format to whole number (Int64)?
(Please do the testing in a test workspace in your Fabric, not production workspace.)
Tbh I'm confused why you are getting an error.
Just to make sure, the IncrementalEventID query has disabled staging, and this query does not have a destination, right?
Perhaps you could create a support ticket to have the support team look at the case.
Which column do you mean? I am using Order ID to filter rows for incremental refresh. It is numeric column. Thank you
Incremental refresh has to be on a datetime column
According to this tutorial (https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2) in Datafow gen 2 you may achieve something similar to incremental refreshl not on date column and this tutorial shows how. I am trying to do the same thing
Check that your IncrementalOrderID query doesn't have staging enabled. It more than likely has and thats why its transforming that query into a table instead of leaving the value as a scalar. Right click the query and make sure that it doesn't have staging enabled.
My IncrementalOrderID query doesn't have staging enabled. I doublechecked. Do you think it matters if Orders query has staging enabled? I think that when the tutorial was written the default for query was to have staging enabled. Now when you create new query it has staging disabled by default. So when I created and loaded whole table using Order uery the staging on this query was disabled. Now any combination of staging disabled/enabled on those two queries produces the same error ("We cannot use operator < for Table and Number types") but the details a different.
When both queries have staging disabled:
- Acitivity "ORDERS_WriteToDataDestination" failed
When OORDERS query has staging enabled and IncrementalOrderID query has staging disabled.:
- Table ORDERS failed
- Acitivity "ORDERS_WriteToDataDestination" idle
Any idea how to fix this?
Thank you!
You are correct, I did not know that. In semantic models we use DataTimes.
