Import table incrementally from ODBC on prem

[lench]

Hi, i'm trying to import tables from on prem ODBC gateway. I use Dataflow gen2. I have a table with 32 million records.

I've imported it once (took 4 hours).

Now i would like to schedule an hourly incremental update. So what i do is import the table as a query and then i apply a filter on the 'updateDate' col (holding 'Date' format data). 

So i've applied a test filter to return only records from the last 2 dates. The problem is that when i run the update, it takes a couple of hours. 

Is there a smarter / faster way to do it? 

Thank you!!

[v-saisrao-msft]

Hi @lench,

We haven’t heard back from you in a while. If your issue has been resolved, Mark the relevant response as the solution to help fellow community members.
If you still need support, let us know if you still need assistance.

Thank you.

[v-saisrao-msft]

Hi @lench,

checking in to see if your issue has been resolved. If the response provided was helpful, kindly mark it as the solution so that others with the same issue can benefit.
Please let us know if you still need assistance.

Thank you.

[v-saisrao-msft]

Hi @lench,

I hope you had a chance to review the solution shared by @lbendlin. If it addressed your question, please consider accepting it as the solution — it helps others find answers more quickly.
If you're still facing the issue, feel free to reply, and we’ll be happy to assist further.

Thank you.

[lbendlin · Accepted solution]

The minimum granularity for incremental refresh is "day", as in "daily partitions".

And since "Today"  means something different to the Power BI service than what it means to you (Remember the Power BI service runs on UTC)  it is very common that two partitions will be refreshed rather than one.

Consider using advanced incremental refresh where the "Today" partition is kept in Direct Query mode.

Incremental refresh for semantic models in Power BI - Power BI | Microsoft Learn