Forum Discussion

Koritala's avatar
Koritala
Post Patron
8 months ago
Solved

Redshift DB views Incremental Refresh Solution In Power BI

Hi All, I have redshift database views as source for my semantic model. many views are having huge volume of data (morethan 1000000 rows). when I tried to import views it is exceeding the maximum l...
Need Help
  • v-tejrama's avatar
    v-tejrama
    8 months ago

    Hi Koritala ,

     

    In Power BI, a Hybrid table is a single table that combines both Import mode and DirectQuery for different parts of its data. Usually, historical data is imported and stored in memory, while the most recent data is kept in DirectQuery mode, allowing Power BI to access the latest records directly from the source. This setup is automatically created when incremental refresh is enabled on a Premium or Fabric capacity, together with real-time data access.

     

    As a result, most of the table is imported, and only a small, recent section stays in DirectQuery, giving users the benefits of both Import performance and DirectQuery freshness in one table. This is referred to as a Hybrid table in Power BI.

     

    I'm also providing the Microsoft links for your reference--https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview?utm_source.

    Thank you.

Koritala's avatar
Koritala
Post Patron
8 months ago

Hi Tejrama,

After I read your commnets my understanding is if Redshift views definetions are having joins, incremental refresh will not work. To make it happen we need to change the views either into tables or Materialized views to enable the query folding option.

Please correct me if my understanding is correct or not?

Thanks,

Sri

v-tejrama's avatar
v-tejrama
Community Support
8 months ago

Hi Koritala ,

 

You’re very close, but there is an important detail to clarify so you have the full picture. Incremental refresh does not work in DirectQuery mode at all, regardless of whether a Redshift view has joins or folds correctly. As long as the model is in DirectQuery, incremental refresh will remain unavailable. The reason people often move from a view to a table or a materialized view in Redshift is that those objects typically fold cleanly, which then allows the model to be switched back to Import or Hybrid mode. Once the data is coming from a foldable table and the model is no longer DirectQuery, Power BI can push the date filters back to the source and incremental refresh can finally work.

 

So it isn’t just the joins in the view that cause the issue. The real requirement is having a foldable source and using Import or Hybrid mode. Converting the view into a table or materialized view is the right approach, and once you do that and reload the model in Import mode, incremental refresh will behave as expected.

Thank you.

  • Koritala's avatar
    Koritala
    Post Patron
    8 months ago

    Hi Tejrama,

    Thanks for your response.

    I just want to clarify onething that as you are saying instead of Redshift views, convert those to either to redshiaft tables or Materilized views. then connect those tables or materialized views connect with Import mode. then Foldinf will be enabled.

    Here my problem is we are using PPU license and most of the base tables/views pulling more than 10 MB data that is the maximum limit support from power bi. So no way I can use Import mode of tables or Materilized views.

    What are your suggestions in this scenario.

    Appriciate if you can provide any solution.

    Thanks,

    Sri

    • v-tejrama's avatar
      v-tejrama
      Community Support
      8 months ago

      Hello again Koritala ,

       

      The 10 MB warning you are seeing is specific to Power Query’s preview and Desktop query limits, and does not restrict your Import model. For production, model size is determined by your license: Pro allows up to 1 GB per dataset, Premium Per User (PPU) supports up to 100 GB, and Premium capacities can be increased further with Large semantic model storage.

       

      Converting complex Redshift views to tables or materialized views that support query folding enables loading via Import or Hybrid tables on PPU/Premium, and you can set up incremental refresh for efficient updates. Essentially, the preview warning does not prevent Import; ensure your workspace license and query folding capabilities align with your required model size and performance.

      Thank you.

      • Koritala's avatar
        Koritala
        Post Patron
        8 months ago

        Hi Tejrama,

        Thanks for your response.

        Thing is like, when I try to import the redshift view to power bi desktop using getdata option and selected the view and clickon load option, after few seconds or a minutes it is throwing a message saying that view has a maximum row limit in size and connect in direct query mode. selected view is not loading into power bi memory if i choose Import mode.

        Please sugest in this case how to takle the situation.

        Thanks,

        Sri