Forum Discussion

lench's avatar
lench
Regular Visitor
1 year ago
Solved

Import table incrementally from ODBC on prem

Hi, i'm trying to import tables from on prem ODBC gateway. I use Dataflow gen2. I have a table with 32 million records.

I've imported it once (took 4 hours).

Now i would like to schedule an hourly incremental update. So what i do is import the table as a query and then i apply a filter on the 'updateDate' col (holding 'Date' format data). 

So i've applied a test filter to return only records from the last 2 dates. The problem is that when i run the update, it takes a couple of hours. 

Is there a smarter / faster way to do it? 

 

Thank you!!

Dataflow
  • lbendlin's avatar
    lbendlin
    1 year ago

    The minimum granularity for incremental refresh is "day", as in "daily partitions".

    And since "Today"  means something different to the Power BI service than what it means to you (Remember the Power BI service runs on UTC)  it is very common that two partitions will be refreshed rather than one.

     

    Consider using advanced incremental refresh where the "Today" partition is kept in Direct Query mode.

     

    Incremental refresh for semantic models in Power BI - Power BI | Microsoft Learn

4 Replies

  • lbendlin's avatar
    lbendlin
    Super User
    1 year ago

    The minimum granularity for incremental refresh is "day", as in "daily partitions".

    And since "Today"  means something different to the Power BI service than what it means to you (Remember the Power BI service runs on UTC)  it is very common that two partitions will be refreshed rather than one.

     

    Consider using advanced incremental refresh where the "Today" partition is kept in Direct Query mode.

     

    Incremental refresh for semantic models in Power BI - Power BI | Microsoft Learn

  • v-saisrao-msft's avatar
    v-saisrao-msft
    Community Support
    1 year ago

    Hi lench,

    I hope you had a chance to review the solution shared by lbendlin. If it addressed your question, please consider accepting it as the solution — it helps others find answers more quickly.
    If you're still facing the issue, feel free to reply, and we’ll be happy to assist further.

     

    Thank you.

  • v-saisrao-msft's avatar
    v-saisrao-msft
    Community Support
    1 year ago

    Hi lench,

    checking in to see if your issue has been resolved. If the response provided was helpful, kindly mark it as the solution so that others with the same issue can benefit.
    Please let us know if you still need assistance.

     

    Thank you.

  • v-saisrao-msft's avatar
    v-saisrao-msft
    Community Support
    1 year ago

    Hi lench,

    We haven’t heard back from you in a while. If your issue has been resolved, Mark the relevant response as the solution to help fellow community members.
    If you still need support, let us know if you still need assistance.

     

    Thank you.