Forum Discussion

Atul0215's avatar
Atul0215
Frequent Visitor
1 year ago
Solved

How to Fetch 11M+ Rows from OData Entity in Power BI with Incremental Refresh?

I have a task where I need to fetch data from an OData entity with over 11 million rows and multiple columns into Power BI (either through Power BI Desktop or Fabric). However, I do not have SQL acce...
General Comment
  • Atul0215's avatar
    Atul0215
    1 year ago

    Hi Anonymous ,

    Since our database is very large, we plan to initially load all historical data up to the last 6 months. Going forward, we will enable incremental refresh to automatically load only new and updated data.

Akash_Varuna's avatar
Akash_Varuna
Super User
1 year ago

Hi Atul0215 , You could use Dataflows and enable incrimental refresh could you please try these out

  • Enable Incremental Refresh: Use RangeStart and RangeEnd parameters in Power Query to filter data by date, then configure incremental refresh in Power BI Service.
  • Ensure Query Folding: Verify that filtering steps fold back to the OData source to optimize performance.
  • Optimize Query: Limit columns using Table.SelectColumns and apply pagination if supported with $top and $skip.
  • Use Dataflows: Set up incremental refresh in a Power BI Dataflow for reusability and efficiency.
    If this post helped please do give a kudos and accept this as a solution
    Thanks In Advance
  • Atul0215's avatar
    Atul0215
    Frequent Visitor
    1 year ago

    Hello Akash_Varuna ,

    Thank you for your suggestions! I have tried the solutions you mentioned, but I am still unable to refresh or load the data, even in Dataflows. The entity I am working with contains a very large amount of data, and I need the complete historical dataset rather than just incremental changes.

    To overcome the OData limitations, I am now exploring the BYOD (Bring Your Own Database) solution. Do you think this would be a better approach for handling large datasets in Power BI? Any insights or recommendations would be greatly appreciated!

    Thanks again for your help.