Forum Discussion

Soumeli's avatar
Soumeli
Regular Visitor
7 months ago
Solved

Large data volume

For a report, I have around 500 million data. User wants all historical data, so volume will keep on increasing. How can I handle this much of data in power bi  without filtering on date range? Pleas...
  • grazitti_sapna's avatar
    7 months ago

    Hi Soumeli , 

    Handling 500 million rows in Power BI Desktop, with full historical data and no date filtering, needs a careful architecture. Power BI can do this—but not with default Import mode.


    What you can do is:- 
    DirectQuery + Aggregations:- 

    This is exactly designed for your scenario.

    How it works

    • Detailed historical data (500M rows) → stays in source DB

    • Aggregated data → stored in Power BI Import

    • Power BI automatically switches between them

    or 

    Push Data to a Semantic Layer

    If you control backend:

    Best sources for this scale

    • Azure SQL / SQL Server

    • Snowflake

    • Databricks SQL Warehouse

    • Fabric Lakehouse / Warehouse

    Use:

    • Pre-aggregated views

    • Partitioned tables

    • Indexed columns

    Power BI becomes only a visualization layer.

     

    🌟 I hope this solution helps you unlock your Power BI potential! If you found it helpful, click 'Mark as Solution' to guide others toward the answers they need.

    💡 Love the effort? Drop the kudos! Your appreciation fuels community spirit and innovation.

    🎖 As a proud SuperUser and Microsoft Partner, we’re here to empower your data journey and the Power BI Community at large.

    🔗 Curious to explore more? [Discover here].

    Let’s keep building smarter solutions together!



  • danextian's avatar
    7 months ago

    Hi Soumeli 

     

    Do you have the capacity to support data at this scale? I’m asking because one of my semantic models already exceeds 350 million rows and is well beyond the 1 GB dataset limit for a Pro workspace - even after extensive optimization.

     

    I’ve stripped the model down as much as possible: the datetime column was split into separate date and time fields, the time was rounded to 30-minute intervals, and all non-essential columns were removed. Despite these steps, the dataset remains over the limit and is expected to grow toward 500 million records and beyond. Given the volume and the need to keep the data in Import mode, incremental refresh is already enabled.

     

    I think this needs to go back to the stakeholders. It’s worth clarifying whether keeping all 500 million records is truly necessary, or if it’s simply something they want just in case. An aggregated dataset—by date and a few key grains—may be more than sufficient for the actual reporting needs. Note: I have several just in case semantic models but certainly not at this scale.

     

    It would also help to understand what they intend to do with the data. In practice, it’s unlikely anyone will be analyzing or viewing all 500 million rows at that level of detail, so retaining that granularity may not add much value compared to a well-designed aggregated model.