Forum Discussion

Koritala's avatar
Koritala
Icon for Post Patron rankPost Patron
10 months ago
Solved

Redshift views incremental refresh possibility in power bi

Hi All,

I just want to check is this possible to implement the Incremental refresh on Redshift Views in power bi?

 

If anyone has worked can you please share the sample M code for anyone of the table/view to see the native sql option enable?

I was trying to implement, but I am not getting the query folding in applied steps. If Native Sql option is not enable my assumption is we can't set the Incremental load for a specific tabel /view as it will process the filters locally (In power BI) and in that case there is no use. Correct me if I am wrong. In this case, what would be your suggestion?

Appriciate if anyone can give the commetns ASAP as it is urgent and stoled the development due to query folding not happening.

 

Thanks,

Sri.

  • Anonymous's avatar
    Anonymous
    10 months ago

    Hi Koritala ,

    The reason View Native Query is still greyed out is because your Redshift view is a bit complex, and Power BI is not able to fold the query automatically, this is very common with views. To get incremental refresh working, you can follow a simple approach that is switch your table to Import mode (or Hybrid if you have Premium), then create a materialized view in Redshift with the same logic. After that, connect to this materialized view directly from the navigator without using any custom SQL. Once you apply the RangeStart/RangeEnd filter step in Power Query, the 'View Native Query' option will become active and incremental refresh will start working properly.

    Materialized views behave just like normal tables in Power BI and almost always fix this issue.

11 Replies

  • The #1 reason query folding breaks on Redshift views is writing a custom SQL statement in the connection dialog (e.g., SELECT * FROM my_view).

    Here is the solution to fix the folding and the M code.

     

    The Solution: Do Not Write SQL
    To ensure Query Folding works, you must connect to the server first and then navigate to the view using the Power Query interface. Do not put anything in the "SQL Statement" box when you first click "Get Data".

     

    The Correct M Code Structure
    Below is the standard pattern that preserves query folding. You can paste this into your Advanced Editor (replacing your specific details).

     

    let
    // Step 1: Connect to the Database level only (No SQL Query inside this function)
    Source = AmazonRedshift.Database("your-cluster-url.redshift.amazonaws.com", "your_database_name"),

    // Step 2: Navigate to your Schema (e.g., 'public')
    // Note: Power Query creates these navigation steps automatically if you click through the list
    public = Source{[Name="public"]}[Data],

    // Step 3: Select your View
    // Because we navigated to it, Power BI treats it like a table and can fold queries
    MyView_View = public{[Name="your_view_name"]}[Data],

    // Step 4: Apply the Incremental Refresh Filter
    // Because the previous steps were navigation, this step will fold into a WHERE clause
    #"Filtered Rows" = Table.SelectRows(MyView_View, each [YourDateColumn] >= RangeStart and [YourDateColumn] < RangeEnd)
    in
    #"Filtered Rows"

     

    How to Verify It Is Working
    Delete your current query if you used a custom SQL statement.

    Go to Get Data > Amazon Redshift.

    Enter the Server and Database, but leave the SQL Statement (Optional) box EMPTY.

    Click OK. A navigator window will appear.

    Expand your schema and select your View checkbox. Click Transform Data.

    Now, filter your date column using the RangeStart and RangeEnd parameters.

    Right-click the "Filtered Rows" step. The "View Native Query" option should now be clickable (black), not grayed out.

     

    What if it still doesn't fold?
    If you use the method above and it still doesn't fold, check these two things:

    Privacy Levels: Go to File > Options and settings > Options > Current File > Privacy. select "Ignore the Privacy Levels...". Sometimes merging data from different sources (or parameters) blocks folding if privacy levels aren't compatible.

    Value.NativeQuery (The Force Method): If the navigation method fails, you can force the SQL context using Value.NativeQuery.

     

    Force Folding M Code:

     

    let
    Source = AmazonRedshift.Database("your-url", "your-db"),
    // Force the query to be recognized as a foldable source
    RunQuery = Value.NativeQuery(
    Source,
    "SELECT * FROM public.your_view_name",
    null,
    [EnableFolding=true]
    ),
    #"Filtered Rows" = Table.SelectRows(RunQuery, each [Date] >= RangeStart and [Date] < RangeEnd)
    in
    #"Filtered Rows"

     

    If this explanation and solution resolve your issue, please like and accept the solution.

    • Koritala's avatar
      Koritala
      Icon for Post Patron rankPost Patron

      Hi Sandip,

      Is this possible to share sample pbxi file?

      After I read your comments, we can setup Incremental refresh only for the Import Redshift views only and can't setup Direct Quert mode views in my semantic model.

      Please help in this.

      Thanks,

      Sri

    • Koritala's avatar
      Koritala
      Icon for Post Patron rankPost Patron

      Hi Sandip,

      I have followed the same as you mentioned the steps. Unfortunately, View Native SQL is not enabling.

      All the above are working with SQL server tables only. For Redsdhift it is wired and strugling for solution like anything from last 2 months.

      If anyone wirked in realtiime projects with Redshift as db along with Power BI only can help think so.

      Thanks,

      Srini.