Forum Discussion
Redshift views incremental refresh possibility in power bi
Hi All,
I just want to check is this possible to implement the Incremental refresh on Redshift Views in power bi?
If anyone has worked can you please share the sample M code for anyone of the table/view to see the native sql option enable?
I was trying to implement, but I am not getting the query folding in applied steps. If Native Sql option is not enable my assumption is we can't set the Incremental load for a specific tabel /view as it will process the filters locally (In power BI) and in that case there is no use. Correct me if I am wrong. In this case, what would be your suggestion?
Appriciate if anyone can give the commetns ASAP as it is urgent and stoled the development due to query folding not happening.
Thanks,
Sri.
- Anonymous10 months ago
Hi Koritala ,
The reason View Native Query is still greyed out is because your Redshift view is a bit complex, and Power BI is not able to fold the query automatically, this is very common with views. To get incremental refresh working, you can follow a simple approach that is switch your table to Import mode (or Hybrid if you have Premium), then create a materialized view in Redshift with the same logic. After that, connect to this materialized view directly from the navigator without using any custom SQL. Once you apply the RangeStart/RangeEnd filter step in Power Query, the 'View Native Query' option will become active and incremental refresh will start working properly.
Materialized views behave just like normal tables in Power BI and almost always fix this issue.
11 Replies
- Sandip_Palit
Super User
The #1 reason query folding breaks on Redshift views is writing a custom SQL statement in the connection dialog (e.g., SELECT * FROM my_view).
Here is the solution to fix the folding and the M code.
The Solution: Do Not Write SQL
To ensure Query Folding works, you must connect to the server first and then navigate to the view using the Power Query interface. Do not put anything in the "SQL Statement" box when you first click "Get Data".The Correct M Code Structure
Below is the standard pattern that preserves query folding. You can paste this into your Advanced Editor (replacing your specific details).let
// Step 1: Connect to the Database level only (No SQL Query inside this function)
Source = AmazonRedshift.Database("your-cluster-url.redshift.amazonaws.com", "your_database_name"),// Step 2: Navigate to your Schema (e.g., 'public')
// Note: Power Query creates these navigation steps automatically if you click through the list
public = Source{[Name="public"]}[Data],// Step 3: Select your View
// Because we navigated to it, Power BI treats it like a table and can fold queries
MyView_View = public{[Name="your_view_name"]}[Data],// Step 4: Apply the Incremental Refresh Filter
// Because the previous steps were navigation, this step will fold into a WHERE clause
#"Filtered Rows" = Table.SelectRows(MyView_View, each [YourDateColumn] >= RangeStart and [YourDateColumn] < RangeEnd)
in
#"Filtered Rows"How to Verify It Is Working
Delete your current query if you used a custom SQL statement.Go to Get Data > Amazon Redshift.
Enter the Server and Database, but leave the SQL Statement (Optional) box EMPTY.
Click OK. A navigator window will appear.
Expand your schema and select your View checkbox. Click Transform Data.
Now, filter your date column using the RangeStart and RangeEnd parameters.
Right-click the "Filtered Rows" step. The "View Native Query" option should now be clickable (black), not grayed out.
What if it still doesn't fold?
If you use the method above and it still doesn't fold, check these two things:Privacy Levels: Go to File > Options and settings > Options > Current File > Privacy. select "Ignore the Privacy Levels...". Sometimes merging data from different sources (or parameters) blocks folding if privacy levels aren't compatible.
Value.NativeQuery (The Force Method): If the navigation method fails, you can force the SQL context using Value.NativeQuery.
Force Folding M Code:
let
Source = AmazonRedshift.Database("your-url", "your-db"),
// Force the query to be recognized as a foldable source
RunQuery = Value.NativeQuery(
Source,
"SELECT * FROM public.your_view_name",
null,
[EnableFolding=true]
),
#"Filtered Rows" = Table.SelectRows(RunQuery, each [Date] >= RangeStart and [Date] < RangeEnd)
in
#"Filtered Rows"If this explanation and solution resolve your issue, please like and accept the solution.
- Koritala
Post Patron
Hi Sandip,
Is this possible to share sample pbxi file?
After I read your comments, we can setup Incremental refresh only for the Import Redshift views only and can't setup Direct Quert mode views in my semantic model.
Please help in this.
Thanks,
Sri
- amitchandak
Super User
Koritala , If you are able to see the native query, it means the query is folding. But you do see that does not mean it is not folding, you have a check query at the source, refer video from Guyinacube -https://www.youtube.com/watch?v=8hjdOCni_ZY
You can use Value.NativeQuery as discussed in the above video and blog here to ensure query folding
https://learn.microsoft.com/en-us/power-query/native-query-folding
Another way is to create a String SQL before the native query and pass that in Native.Query step. The String SQL should take care of incremental data
- Koritala
Post Patron
Hi Sandip,
I have followed the same as you mentioned the steps. Unfortunately, View Native SQL is not enabling.
All the above are working with SQL server tables only. For Redsdhift it is wired and strugling for solution like anything from last 2 months.
If anyone wirked in realtiime projects with Redshift as db along with Power BI only can help think so.
Thanks,
Srini.