Forum Discussion

SBC's avatar
SBC
Icon for Helper III rankHelper III
1 year ago
Solved

How to efficiently load large dataset (>1GB) from source into Power BI

Hi Experts,

I have a situation with my Power BI report:

  • I can only select entire tables from the source (no option to write SQL queries or apply filters at the source level).After fetching the data into Power BI, I can make transformations.
  • I have a date column in two tables. If I filter only 3 days of data, it works fine. But if I try to pull all data, it’s huge (more than 1GB) and Power BI fails to load it.
  • I’ve already removed unused columns and duplicate data in Power Query.

My requirement:
I need to bring this large dataset into Power BI efficiently so that I can work with historical data without exceeding file size limits.

Question:

  • What is the best approach to handle this situation?
  • Can incremental refresh or dataflows help here when I cannot filter data at the source?
  • Are there other optimization techniques I should apply?

Thanks in advance for your help!

 

Regards,

SBC

Interesting Links
Need Help
Show and Tell
Tips and Tricks
  • Shahid12523's avatar
    Shahid12523
    1 year ago

    Don’t import full 1GB into Desktop — use Incremental Refresh with RangeStart/RangeEnd → Desktop loads only a slice, Service handles partitions.

    Use Dataflows as a staging layer for heavy tables.

    Keep recent data in detail, aggregate old data.

    Optimize model (drop unused columns, use proper types).

    If still too big → use Composite Model (Import recent + DirectQuery old).

     

    Best option: Incremental Refresh + model optimization.

5 Replies

  • audreygerred's avatar
    audreygerred
    Icon for Super User rankSuper User
    1 year ago

    Hello! Power BI typically handles large data great and the compression in import mode is amazing. What Fabric SKU do you have - that determines your size limitations. If you are just using pro, your model cannot exceed 1 GB. F64 is 25GB per model, F128 is 50GB per model, F256 is 100GB per model.

  • v-achippa's avatar
    v-achippa
    Icon for Community Support rankCommunity Support
    1 year ago

    Hi SBC,

     

    Thank you for reaching out to Microsoft Fabric Community.

     

    Thank you audreygerred for the prompt response.

     

    As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user for the issue worked? or let us know if you need any further assistance.

     

    Thanks and regards,

    Anjan Kumar Chippa

    • SBC's avatar
      SBC
      Icon for Helper III rankHelper III
      1 year ago

      Hi v-achippa ,

      Thank you for the follow-up.

      The previous response didn’t fully address my expectations. I’m reviewing alternative approaches and best practices within our Power BI Pro license constraints. I’m also monitoring the community for any strong suggestions that align with the requirement. 

       

      Thanks,
      SBC

  • Shahid12523's avatar
    Shahid12523
    Icon for Community Champion rankCommunity Champion
    1 year ago

    Don’t import full 1GB into Desktop — use Incremental Refresh with RangeStart/RangeEnd → Desktop loads only a slice, Service handles partitions.

    Use Dataflows as a staging layer for heavy tables.

    Keep recent data in detail, aggregate old data.

    Optimize model (drop unused columns, use proper types).

    If still too big → use Composite Model (Import recent + DirectQuery old).

     

    Best option: Incremental Refresh + model optimization.

    • v-achippa's avatar
      v-achippa
      Icon for Community Support rankCommunity Support
      1 year ago

      Hi @SBC,

       

      Thank you Shahid12523 for the response.

       

      As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by Shahid for the issue worked? Is your issue resolved? or let us know if you need any further assistance.

       

      Thanks and regards,

      Anjan Kumar Chippa