Forum Discussion
Redshift views incremental refresh possibility in power bi
- 8 months ago
Hi Koritala ,
The reason View Native Query is still greyed out is because your Redshift view is a bit complex, and Power BI is not able to fold the query automatically, this is very common with views. To get incremental refresh working, you can follow a simple approach that is switch your table to Import mode (or Hybrid if you have Premium), then create a materialized view in Redshift with the same logic. After that, connect to this materialized view directly from the navigator without using any custom SQL. Once you apply the RangeStart/RangeEnd filter step in Power Query, the 'View Native Query' option will become active and incremental refresh will start working properly.
Materialized views behave just like normal tables in Power BI and almost always fix this issue.
The #1 reason query folding breaks on Redshift views is writing a custom SQL statement in the connection dialog (e.g., SELECT * FROM my_view).
Here is the solution to fix the folding and the M code.
The Solution: Do Not Write SQL
To ensure Query Folding works, you must connect to the server first and then navigate to the view using the Power Query interface. Do not put anything in the "SQL Statement" box when you first click "Get Data".
The Correct M Code Structure
Below is the standard pattern that preserves query folding. You can paste this into your Advanced Editor (replacing your specific details).
let
// Step 1: Connect to the Database level only (No SQL Query inside this function)
Source = AmazonRedshift.Database("your-cluster-url.redshift.amazonaws.com", "your_database_name"),
// Step 2: Navigate to your Schema (e.g., 'public')
// Note: Power Query creates these navigation steps automatically if you click through the list
public = Source{[Name="public"]}[Data],
// Step 3: Select your View
// Because we navigated to it, Power BI treats it like a table and can fold queries
MyView_View = public{[Name="your_view_name"]}[Data],
// Step 4: Apply the Incremental Refresh Filter
// Because the previous steps were navigation, this step will fold into a WHERE clause
#"Filtered Rows" = Table.SelectRows(MyView_View, each [YourDateColumn] >= RangeStart and [YourDateColumn] < RangeEnd)
in
#"Filtered Rows"
How to Verify It Is Working
Delete your current query if you used a custom SQL statement.
Go to Get Data > Amazon Redshift.
Enter the Server and Database, but leave the SQL Statement (Optional) box EMPTY.
Click OK. A navigator window will appear.
Expand your schema and select your View checkbox. Click Transform Data.
Now, filter your date column using the RangeStart and RangeEnd parameters.
Right-click the "Filtered Rows" step. The "View Native Query" option should now be clickable (black), not grayed out.
What if it still doesn't fold?
If you use the method above and it still doesn't fold, check these two things:
Privacy Levels: Go to File > Options and settings > Options > Current File > Privacy. select "Ignore the Privacy Levels...". Sometimes merging data from different sources (or parameters) blocks folding if privacy levels aren't compatible.
Value.NativeQuery (The Force Method): If the navigation method fails, you can force the SQL context using Value.NativeQuery.
Force Folding M Code:
let
Source = AmazonRedshift.Database("your-url", "your-db"),
// Force the query to be recognized as a foldable source
RunQuery = Value.NativeQuery(
Source,
"SELECT * FROM public.your_view_name",
null,
[EnableFolding=true]
),
#"Filtered Rows" = Table.SelectRows(RunQuery, each [Date] >= RangeStart and [Date] < RangeEnd)
in
#"Filtered Rows"
If this explanation and solution resolve your issue, please like and accept the solution.
- Koritala8 months agoPost Patron
Hi Sandip,
Is this possible to share sample pbxi file?
After I read your comments, we can setup Incremental refresh only for the Import Redshift views only and can't setup Direct Quert mode views in my semantic model.
Please help in this.
Thanks,
Sri
- amitchandak8 months agoSuper User
Koritala , If you are able to see the native query, it means the query is folding. But you do see that does not mean it is not folding, you have a check query at the source, refer video from Guyinacube -https://www.youtube.com/watch?v=8hjdOCni_ZY
You can use Value.NativeQuery as discussed in the above video and blog here to ensure query folding
https://learn.microsoft.com/en-us/power-query/native-query-folding
Another way is to create a String SQL before the native query and pass that in Native.Query step. The String SQL should take care of incremental data - v-sshirivolu8 months agoCommunity Support
Hi Koritala ,
Power BI cannot apply Incremental Refresh when your Redshift view is connected in DirectQuery mode. Incremental Refresh only works when the data is loaded in Import or Hybrid mode, and it also needs query folding so Power BI can push the RangeStart/RangeEnd filters back to Redshift. Most Redshift views don’t support folding, and DirectQuery doesn’t use the Import engine at all, so the Incremental Refresh option simply won’t appear. Because of this, a sample PBIX can’t show Incremental Refresh on DirectQuery, any working example must use Import mode with a source that supports folding, such as a materialized view or a physical table. To move forward, try loading data from a materialized view, a proper table, or a simplified view that folds correctly. After that, Incremental Refresh will work normally.Microsoft’s own documentation and known limitations confirm all these points (see links below for reference):
https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview
https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-configure#query-folding- v-sshirivolu8 months agoCommunity Support
Hi Koritala ,
I hope the information provided above assists you in resolving the issue. If you have any additional questions or concerns, please do not hesitate to contact us. We are here to support you and will be happy to help with any further assistance you may need.
- Koritala8 months agoPost Patron
Hi Sandip,
I have followed the same as you mentioned the steps. Unfortunately, View Native SQL is not enabling.
All the above are working with SQL server tables only. For Redsdhift it is wired and strugling for solution like anything from last 2 months.
If anyone wirked in realtiime projects with Redshift as db along with Power BI only can help think so.
Thanks,
Srini.