How to load more that 10 Millions rows in to power bi data set

[Saikumar_r7]

HI People,

Currently im accessing snowflake from power query using SQL to import data,

my use case is pulling out P&L Fixed cost data of past 2 years like 2022 & 2023 at Accounting document level,

1. Each month carries around 2 M rows which makes 24M in one year itself, i don't have an option to do agg tables also,

2. If i go for incremental refresh- in Power query im holding one month data and publishing it in service,

But im having error when it refreshes in service first in query im doing Unpivot & Pivot so its telling a column is missing,

secondly, incremental refresh need little understanding how to setup refresh for 2022 & 2023,

archive 1/1/2022 to 31/12/2022 & before refresh data 1/1/2023  to 31/12/2023 is what i have setup is this correct?,

Appreciate the community help on this?

[amustafa · Accepted solution]

Hi, I would pull this data into Fabric Lakehouse or Warehouse, perhaps in a datamart and do the aggragations there and build the Power BI model. 

[Saikumar_r7 · Accepted solution]

Hi @v-zhengdxu-msft 

I got the solution thru the above process and also got new solution by folding the query itself by unpivoting in SQL and its pretty faster for me.

thanks

[amustafa · Accepted solution]

Hi, I would pull this data into Fabric Lakehouse or Warehouse, perhaps in a datamart and do the aggragations there and build the Power BI model. 

[Saikumar_r7]

Ok amustafa, i will look in to this option.

[v-zhengdxu-msft]

Hi @Saikumar_r7 

Have you gotten the solution? If so, could you please mark the helpful post as Answered? It will help the others in the community find the solution easily if they face the same problem as yours. Thank you.

Best Regards    

Zhengdong Xu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.