Forum Discussion
Resolver IDebuging MySql partitions in a dataflow with an incremental refresh
So that's my case.
We have a on-premises MySql connected to a dataflow with a Data Gateway and using incremental refresh.
The incremental refresh is set to archive last 10 years and refresh the last 2 months.
The MySql table has a DateTime Column and the data is updated once a week using a Python script that update or create the records in the table for the last week.
I've reviewed the database and I could see the new info in the database but after run the dataflow some records from the last week are missing.
So there's any way to look at this?
2 Replies
- lbendlin
Super User
a Python script that update or create the records in the table for the last week.That's not what incremental refresh is about. To emulate differential refresh you need to widen your hot update window (say, last 4 weeks instead of last week) and you need to plan for periodic full refreshes, also of older partitions.
- debiagui
a Python script that update or create the records in the table for the last week.
This is only trying to explain the general process, but the problem is in the incremental refresh, the script in python is just the way to feed the database
