Forum Discussion

Koritala's avatar
Koritala
Post Patron
8 months ago
Solved

Redshift DB views Incremental Refresh Solution In Power BI

Hi All, I have redshift database views as source for my semantic model. many views are having huge volume of data (morethan 1000000 rows). when I tried to import views it is exceeding the maximum l...
Need Help
  • v-tejrama's avatar
    v-tejrama
    8 months ago

    Hi Koritala ,

     

    In Power BI, a Hybrid table is a single table that combines both Import mode and DirectQuery for different parts of its data. Usually, historical data is imported and stored in memory, while the most recent data is kept in DirectQuery mode, allowing Power BI to access the latest records directly from the source. This setup is automatically created when incremental refresh is enabled on a Premium or Fabric capacity, together with real-time data access.

     

    As a result, most of the table is imported, and only a small, recent section stays in DirectQuery, giving users the benefits of both Import performance and DirectQuery freshness in one table. This is referred to as a Hybrid table in Power BI.

     

    I'm also providing the Microsoft links for your reference--https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview?utm_source.

    Thank you.

v-tejrama's avatar
v-tejrama
Community Support
8 months ago

Hi  Koritala ,

 

It looks like everything you are running into is caused by how Power BI handles DirectQuery and query folding rather than anything specific to Redshift. Import mode is the only mode that supports incremental refresh, so once your dataset switched to DirectQuery to handle the size of your views, incremental refresh became unavailable. Even if the view allowed folding, incremental refresh still would not work because DirectQuery models simply do not support that feature.

 

Redshift views add another complication because they often cannot fold, especially if they include joins or aggregations, which is why you are seeing the native query option disabled in Power Query. To use incremental refresh with Redshift, the data has to be in Import or Hybrid mode, and the source has to support folding so Power BI can push the date filters back to the source. The typical solution is to replace the view with a table or a materialized view in Redshift so the model can be set back to Import mode. Once the data is coming from a foldable, date-filtered table and the model is using Import or Hybrid mode, incremental refresh will work as expected.

Thank you.

 
 

 

Koritala's avatar
Koritala
Post Patron
8 months ago

Hi Tejrama,

After I read your commnets my understanding is if Redshift views definetions are having joins, incremental refresh will not work. To make it happen we need to change the views either into tables or Materialized views to enable the query folding option.

Please correct me if my understanding is correct or not?

Thanks,

Sri

  • v-tejrama's avatar
    v-tejrama
    Community Support
    8 months ago

    Hi Koritala ,

     

    You’re very close, but there is an important detail to clarify so you have the full picture. Incremental refresh does not work in DirectQuery mode at all, regardless of whether a Redshift view has joins or folds correctly. As long as the model is in DirectQuery, incremental refresh will remain unavailable. The reason people often move from a view to a table or a materialized view in Redshift is that those objects typically fold cleanly, which then allows the model to be switched back to Import or Hybrid mode. Once the data is coming from a foldable table and the model is no longer DirectQuery, Power BI can push the date filters back to the source and incremental refresh can finally work.

     

    So it isn’t just the joins in the view that cause the issue. The real requirement is having a foldable source and using Import or Hybrid mode. Converting the view into a table or materialized view is the right approach, and once you do that and reload the model in Import mode, incremental refresh will behave as expected.

    Thank you.

    • Koritala's avatar
      Koritala
      Post Patron
      8 months ago

      Hi Tejrama,

      Thanks for your response.

      I just want to clarify onething that as you are saying instead of Redshift views, convert those to either to redshiaft tables or Materilized views. then connect those tables or materialized views connect with Import mode. then Foldinf will be enabled.

      Here my problem is we are using PPU license and most of the base tables/views pulling more than 10 MB data that is the maximum limit support from power bi. So no way I can use Import mode of tables or Materilized views.

      What are your suggestions in this scenario.

      Appriciate if you can provide any solution.

      Thanks,

      Sri

      • v-tejrama's avatar
        v-tejrama
        Community Support
        8 months ago

        Hello again Koritala ,

         

        The 10 MB warning you are seeing is specific to Power Query’s preview and Desktop query limits, and does not restrict your Import model. For production, model size is determined by your license: Pro allows up to 1 GB per dataset, Premium Per User (PPU) supports up to 100 GB, and Premium capacities can be increased further with Large semantic model storage.

         

        Converting complex Redshift views to tables or materialized views that support query folding enables loading via Import or Hybrid tables on PPU/Premium, and you can set up incremental refresh for efficient updates. Essentially, the preview warning does not prevent Import; ensure your workspace license and query folding capabilities align with your required model size and performance.

        Thank you.