cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Anonymous
Not applicable

Working with big datasets.

Hi

 

I need to create a new report and use incremental refresh in the future. For that I can not use custom SQL while pulling data from data warehouse because then incremental refresh will not work. Problem is I need data from tables that have data for 10+ years, I only need last 6 months worth of data. Can I filter the data in these tables without custom SQL before pulling the data? I can not overload this data warehouse pulling 10+ years of data and after that using filter. My only idea once again is creating a tabular model to another server but that's extra step for me for creating just 1 report.

4 REPLIES 4
judgejebber
Regular Visitor

Just create a sql view and select it like you would a table. Then incremental works just fine and can query fold succesfully.

Anonymous
Not applicable

If I create a view with a WHERE condition going back to almost a year doesn't that mean it still has to go through that same query that I am trying to avoid? This query takes too much time and resources on the data warehouse.

lbendlin
Super User
Super User

please provide more context for "I can not use custom SQL"

 

When you filter in Power Query it will attempt to push that filter upstream via "Query folding"  aka "native query".

 

please clarify " because then incremental refresh will not work " - that is not necessarily related.

Anonymous
Not applicable

With "Custom SQL" I meant when geting data. Get data-> Sql server. Under advanced options "SQL statement (optional, requires database)" I've learned that if I have anything written there then incremental refresh wouldn't work, so far this is the way I have been geting the data I need without downloading everything. Now I need another way since I have to get incremental refresh working.

Helpful resources

Announcements
May 2023 update

Power BI May 2023 Update

Find out more about the May 2023 update.

Submit your Data Story

Data Stories Gallery

Share your Data Story with the Community in the Data Stories Gallery.

Top Solution Authors